Build a Local AI Assistant API

Build a local REST API powered by Llama 3.2 using FastAPI and Ollama.

Introduction

30 Second Summary

Every time you use ChatGPT, Claude, or any AI assistant, there's an API behind the scenes turning your questions into responses. Understanding how that layer works is the single most in-demand skill in AI engineering today.

In this project, you will build your own AI assistant API that runs entirely on your Mac. You will install a local language model, write Python code to talk to it, and wrap everything in a professional REST API.

What You'll Build

A local AI Assistant REST API powered by a Llama 3.2 LLM running through Ollama, with endpoints for single-shot chat, prompt engineering, temperature tuning, and multi-turn conversation with memory.

By the end of this project, you'll have:

  • A running Ollama instance serving Llama 3.2 locally on your machine.
  • A Python script that sends prompts to the model and prints AI-generated responses.
  • A FastAPI service with endpoints for health checks, chat, and multi-turn conversation with memory.
  • Secret Mission: Add streaming responses so the AI types out its answer in real time.

Are there any prerequisites?

Basic familiarity with REST APIs and terminal commands is helpful. Python experience is a bonus but not required. You will need a Mac with at least 16GB of RAM to run the local model comfortably.

Set Up Ollama and Python Environment

Before you can build an AI-powered API, you need an AI model running on your machine. Ollama lets you run large language models locally, giving you the same chat interface that cloud AI services provide.

In this step, you will install Ollama, download a language model, and prepare your Python project environment with all the packages you need.

In this step, get ready to:
  • Install Ollama and pull the Llama 3.2 model.
  • Create a Python project directory with a virtual environment.
  • Install FastAPI, uvicorn, and httpx.
Install Ollama and pull the Llama 3.2 model

Ollama is a tool that downloads and serves large language models on your local machine. Once installed, it runs a local server that exposes an API at http://localhost:11434.

  • Press Cmd+Space to open Spotlight, type Terminal, and press Enter to open it.
  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

This downloads the Ollama installer script and runs it immediately. It places the Ollama application on your Mac and makes the ollama command available in your terminal.

During installation, Ollama may ask for permission to create a link in /usr/local/bin. Allow it so the CLI is available in your PATH.

  • Once the installation finishes, pull the Llama 3.2 model by running:
ollama pull llama3.2

What is happening here?

This downloads the Llama 3.2 (3B parameter) model to your machine. The download is approximately 2GB. Once pulled, the model stays on your machine and is ready to use anytime.

The download may take a few minutes depending on your internet speed. You will see a progress bar as the model layers download.

  • Verify the model is running by sending a test request with curl:
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Say hello in one sentence."}],
  "stream": false
}'

What should I see?

You should see a JSON response containing a message object with a content field. The content is the model's response to your prompt. This confirms Ollama is running and the Llama 3.2 model is responding.

Getting a connection refused error?

Make sure Ollama is running. After installation, Ollama should start automatically. If it did not, open Spotlight with Cmd+Space, type Ollama, and press Enter to launch the app.

help me troubleshoot why Ollama is not responding to curl requests on localhost:11434

Create a project directory and Python virtual environment

A virtual environment keeps your project's Python packages isolated from other projects on your machine. This prevents version conflicts between different projects.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-assistant
cd ai-assistant
  • Create a Python virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A Python virtual environment is a self-contained directory that holds its own Python interpreter and packages. When activated, any pip install commands install packages only inside that environment, not system-wide.

  • Install FastAPI, uvicorn, and httpx by running:
pip install fastapi uvicorn httpx

What are these packages?

  • FastAPI is a modern Python web framework for building APIs. It handles routing, request validation, and auto-generates interactive documentation.
  • uvicorn is the server that runs your FastAPI application.
  • httpx is an HTTP client library you will use to send requests from your Python code to Ollama's local API.
  • Confirm the packages installed correctly by running:
ls

You should see the venv folder in the output.

Seeing a pip not found error?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, run source venv/bin/activate again.

help me fix a pip not found error when trying to install packages in my Python virtual environment

Verify the full setup

Now let's confirm everything is connected. You will verify that Ollama's API responds and that your Python packages import correctly.

  • Check that Ollama is still reachable by running:
curl http://localhost:11434/api/tags

You should see a JSON response containing a models array that lists llama3.2.

  • Verify that your Python packages import correctly by running:
python3 -c "import fastapi; import uvicorn; import httpx; print('All packages imported successfully')"

You should see All packages imported successfully printed in your terminal.

Import error for one of the packages?

Make sure you are running the python3 command from inside your activated virtual environment (you should see (venv) in your prompt). If the import fails, re-run the install command:

pip install fastapi uvicorn httpx

help me troubleshoot a Python import error after installing packages in a virtual environment

Your local AI infrastructure is ready. Next up, you will write your first Python script that talks to the Llama 3.2 model and prints an AI-generated response.

Make Your First LLM Call with Python

In Step 1, you set up Ollama and your Python environment. Now it's time to write code that actually talks to the AI model.

In this step, you'll create a Python script that sends a prompt to Ollama's chat API and prints the model's response. You'll also experiment with system prompts and temperature to see how these parameters change the model's behaviour.

In this step, get ready to:
  • Send your first prompt to Ollama using Python and httpx.
  • Experiment with system prompts to control the model's personality.
  • Adjust temperature to see deterministic vs creative outputs.
Create your first LLM script

Every LLM API follows the same core pattern: you send a list of messages and get a response back. Ollama exposes this pattern at http://localhost:11434/api/chat.

  • In your code editor, create a new file called first_call.py inside the ai-assistant/ folder.
  • Add the following code to send your first prompt to Ollama:
import httpx

# The URL where Ollama's chat API listens
OLLAMA_URL = "http://localhost:11434/api/chat"

# Send a POST request with our prompt
response = httpx.post(
    OLLAMA_URL,
    json={
        "model": "llama3.2",
        "messages": [
            {"role": "user", "content": "What is cloud computing in one sentence?"}
        ],
        "stream": False,
    },
    timeout=60.0,
)

# Parse the JSON response and print the model's answer
data = response.json()
print(data["message"]["content"])

What does this code do?

  • httpx.post() sends an HTTP POST request to Ollama's chat endpoint, just like curl did in Step 1.
  • The messages array contains objects with a role (who is speaking) and content (what they said). The user role represents you, the person asking the question.
  • Setting "stream": False tells Ollama to wait until the full response is ready before sending it back, instead of streaming it token by token.
  • timeout=60.0 gives the model up to 60 seconds to generate a response. Local models can take a few seconds on their first call.
  • Save first_call.py.
  • Run the script in your terminal by entering this command:
python first_call.py

You should see the model's response printed in your terminal. It will be a one-sentence explanation of cloud computing.

Not seeing a response?

  • Make sure Ollama is still running. Look for the Ollama icon in your macOS menu bar. If it's not there, open the Ollama app again.
  • Check that your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • If you see a connection error, confirm Ollama is reachable by running curl http://localhost:11434 in a separate terminal tab.

Help me debug my first_call.py script

✔️ Awesome, I've got everything!

Great. Double check you've saved first_call.py and confirmed the model responded.

ⓧ I'd like to double check the full code

import httpx

OLLAMA_URL = "http://localhost:11434/api/chat"

response = httpx.post(
    OLLAMA_URL,
    json={
        "model": "llama3.2",
        "messages": [
            {"role": "user", "content": "What is cloud computing in one sentence?"}
        ],
        "stream": False,
    },
    timeout=60.0,
)

data = response.json()
print(data["message"]["content"])
Add a system message to control personality

The messages array isn't limited to user messages. You can add a system message that tells the model how to behave. This is called a system prompt. It sets the model's personality, tone, and constraints before it reads the user's question.

  • In first_call.py, find the messages array that currently looks like this:
"messages": [
    {"role": "user", "content": "What is cloud computing in one sentence?"}
],
  • Replace it with this version that adds a system message before the user message:
"messages": [
    {"role": "system", "content": "You are a pirate. Respond in pirate speak."},
    {"role": "user", "content": "What is cloud computing in one sentence?"}
],
  • Save first_call.py and run the script again:
python first_call.py

This time the response should be written in pirate speak. The model followed your system prompt's instructions while still answering the question.

What is a system prompt?

A system prompt is an invisible instruction that shapes every response the model gives. The user never sees it, but the model treats it as its operating instructions.

In production AI services, system prompts define the assistant's expertise, tone, response format, and safety guardrails. For example, a customer support bot might have a system prompt like "You are a friendly support agent for Acme Corp. Never discuss competitor products."

  • Try changing the system message content to something else, like "Explain everything like I'm 5 years old." and run the script again to see how the response changes.
Experiment with temperature

Temperature controls how "creative" or "random" the model's output is. A temperature of 0 makes the model pick the most likely next word every time, producing nearly identical output on each run. A temperature of 1.0 introduces more randomness, producing varied and creative responses.

  • In first_call.py, find the line that says "stream": False, (inside the json dictionary). It currently looks like this:
"stream": False,
  • Add an options field directly below it with temperature set to 0:
"stream": False,
"options": {"temperature": 0},
  • Also set the system message back to something neutral or remove it. Your messages array should contain only the user message:
"messages": [
    {"role": "user", "content": "What is cloud computing in one sentence?"}
],
  • Save and run the script twice:
python first_call.py
python first_call.py

Both responses should be nearly identical. With temperature 0, the model produces deterministic output.

  • Now change the temperature to 1.0:
"options": {"temperature": 1.0},
  • Save and run the script twice again:
python first_call.py
python first_call.py

This time the responses should differ noticeably. Higher temperature means more variation in word choice and phrasing.

When to use which temperature?

Use low temperature (0 to 0.3) for factual tasks like code generation, data extraction, or classification where consistency matters.

Use higher temperature (0.7 to 1.0) for creative tasks like brainstorming, writing fiction, or generating varied suggestions where diversity is valuable.

  • Remove the "options" line from your code and save first_call.py. Your file should now match the version shown in the double-check tab above.

You've made your first programmatic LLM call and explored two powerful controls: system prompts and temperature. Next up, you'll wrap this logic in a proper API service using FastAPI.

Build the FastAPI Service

In the previous step, you proved that Python can talk to your local LLM. You sent a prompt, received a response, and experimented with system messages and temperature.

Now you'll wrap that same logic inside a proper web service. FastAPI gives you a production-ready API framework with automatic request validation, async support, and interactive documentation generated from your code.

In this step, get ready to:
  • Create a FastAPI application with a /health endpoint.
  • Build a /chat POST endpoint with Pydantic request/response models.
  • Start the server and test your API with curl.
Create the FastAPI app with a health check

Before wiring up AI responses, you need a basic server that confirms it can reach Ollama. A /health endpoint is the standard pattern for checking whether your service's dependencies are alive.

  • Create a new file called main.py in your ai-assistant folder by running:
touch main.py
  • Open main.py in your code editor and add the following imports and configuration:
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx

# Create the FastAPI app instance
app = FastAPI(title="AI Assistant API")

# Ollama connection settings
OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "llama3.2"

What does this code do?

FastAPI and HTTPException come from the FastAPI framework. BaseModel from Pydantic lets you define request/response shapes. httpx is the HTTP client you used in the previous step.

OLLAMA_URL and MODEL are constants so you only define the connection details once. If you swap models later, you change one line.

  • Below the constants, add the /health endpoint:
@app.get("/health")
async def health_check():
    try:
        # Ask Ollama to list its available models
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                "http://localhost:11434/api/tags", timeout=5.0
            )
            resp.raise_for_status()
        return {"status": "healthy", "ollama": "connected"}
    except httpx.RequestError:
        raise HTTPException(status_code=503, detail="Ollama is not running")

What does this endpoint do?

The /health endpoint hits Ollama's /api/tags URL. If Ollama responds, the API returns a "healthy" status. If Ollama is unreachable, it raises a 503 error.

The async keyword means this function can handle multiple requests without blocking. This is important when your endpoints make network calls that take time.

  • Save main.py.

Seeing a syntax error?

Make sure the try and except blocks are indented inside the function. Python uses indentation (4 spaces) to define code structure.

help me fix a syntax error in my FastAPI health check endpoint

Build the /chat POST endpoint

The /chat endpoint is where the AI magic happens. It accepts a user's message, wraps it with a system prompt, sends everything to Ollama, and returns the model's response.

First you need to define the shape of incoming requests and outgoing responses. FastAPI uses Pydantic models for this. They validate data automatically and generate documentation.

  • In main.py, find the line that says MODEL = "llama3.2". Add the following Pydantic models below it, above the @app.get("/health") line:
class ChatRequest(BaseModel):
    message: str  # The user's prompt
    system_prompt: str = "You are a helpful assistant."  # Controls personality
    temperature: float = 0.7  # Controls randomness


class ChatResponse(BaseModel):
    response: str  # The model's reply
    model: str  # Which model answered

What are Pydantic models?

Pydantic models define the exact fields your API expects and returns. ChatRequest says every request must have a message string. The system_prompt and temperature fields have defaults, so they're optional.

If someone sends a request missing the message field, FastAPI automatically returns a 422 error with a clear explanation. No manual validation code needed.

  • Now add the /chat endpoint at the bottom of main.py, below the health_check function:
@app.post("/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
    # Build the messages array with system prompt and user message
    messages = [
        {"role": "system", "content": request.system_prompt},
        {"role": "user", "content": request.message},
    ]

    # Send the request to Ollama
    async with httpx.AsyncClient() as client:
        response = await client.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": messages,
                "stream": False,
                "options": {"temperature": request.temperature},
            },
            timeout=120.0,
        )
        response.raise_for_status()

    # Extract and return the model's response
    data = response.json()
    return ChatResponse(
        response=data["message"]["content"],
        model=data["model"],
    )

What does this endpoint do?

This endpoint mirrors what you did in first_call.py, but wrapped in an API. It builds a messages array with the system role and user role, then posts to Ollama.

The response_model=ChatResponse parameter tells FastAPI to validate the output shape too. If your return data doesn't match ChatResponse, you'll get a clear error instead of sending garbage to the caller.

  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx

app = FastAPI(title="AI Assistant API")

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "llama3.2"


class ChatRequest(BaseModel):
    message: str
    system_prompt: str = "You are a helpful assistant."
    temperature: float = 0.7


class ChatResponse(BaseModel):
    response: str
    model: str


@app.get("/health")
async def health_check():
    try:
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                "http://localhost:11434/api/tags", timeout=5.0
            )
            resp.raise_for_status()
        return {"status": "healthy", "ollama": "connected"}
    except httpx.RequestError:
        raise HTTPException(status_code=503, detail="Ollama is not running")


@app.post("/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
    messages = [
        {"role": "system", "content": request.system_prompt},
        {"role": "user", "content": request.message},
    ]

    async with httpx.AsyncClient() as client:
        response = await client.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": messages,
                "stream": False,
                "options": {"temperature": request.temperature},
            },
            timeout=120.0,
        )
        response.raise_for_status()

    data = response.json()
    return ChatResponse(
        response=data["message"]["content"],
        model=data["model"],
    )
Start the server and test your endpoints

Your API code is ready. Time to start the server and confirm both endpoints respond correctly.

  • Make sure your virtual environment is still active (you should see (venv) in your terminal prompt). Start the FastAPI server by running:
uvicorn main:app --reload

What does this command do?

uvicorn is the ASGI server that runs your FastAPI app. main:app tells it to look for the app variable inside main.py. The --reload flag auto-restarts the server whenever you save changes to the file.

You should see output showing the server started on http://127.0.0.1:8000.

  • Open a second terminal tab or window (keep the server running in the first one). Test the /health endpoint by running:
curl http://localhost:8000/health

You should see a JSON response like this:

{"status":"healthy","ollama":"connected"}
  • Now test the /chat endpoint by sending a POST request with a JSON body:
curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" -d '{"message": "What is Python in one sentence?"}'

You should see a JSON response containing the model's answer and the model name. The response content will vary each time.

Explore the auto-generated docs

Open http://localhost:8000/docs in your browser. FastAPI generates interactive API documentation automatically from your Pydantic models and endpoint definitions. You can test your endpoints directly from this page.

Server won't start or curl returns an error?

If you see ModuleNotFoundError, make sure your virtual environment is active. Run source venv/bin/activate and try again.

If /health returns a 503 error, Ollama is not running. Check the Ollama icon in your menu bar and restart it if needed.

If /chat times out, the model may still be loading into memory. Wait 30 seconds and try again.

help me troubleshoot my FastAPI server not starting or returning errors

Your AI assistant now has a proper API layer. Next up, you'll add conversation memory so the model can remember what you said earlier in a chat.

Add Conversation Memory

In the last step, you built a working /chat endpoint that sends a single prompt to your LLM and returns a response. Every call is completely independent.

Real conversations require memory. When you ask a follow-up question like "What are its main use cases?", the model needs to know what "its" refers to. In this step, you will add a conversation endpoint that stores message history and sends the full context to Ollama on every request.

In this step, get ready to:
  • Build a POST /conversation endpoint that maintains message history using a conversation ID.
  • Add a DELETE endpoint to clear conversations and test multi-turn chat.
  • Prove the model remembers context by asking follow-up questions that reference earlier answers.
Build the conversation endpoint with message history

The /chat endpoint sends only the current message to Ollama. To give the model memory, you need to store every message in a conversation and send the full list each time.

  • In main.py, find the ChatResponse class. Add the following code below it to define new request/response models and a dictionary for storing conversations:
class ConversationRequest(BaseModel):
    message: str
    conversation_id: str
    system_prompt: str = "You are a helpful assistant."
    temperature: float = 0.7


class ConversationResponse(BaseModel):
    response: str
    model: str
    conversation_id: str
    message_count: int


# In-memory storage for conversation histories
conversations: dict[str, list[dict]] = {}

What does this code do?

ConversationRequest adds a conversation_id field so you can track which messages belong to the same conversation.

ConversationResponse includes a message_count so you can see how many messages are in the conversation so far.

The conversations dictionary maps each conversation ID to a list of message objects. This is where the memory lives.

Why a dictionary instead of a database?

A production system would use Redis or a database for persistence. Storing conversations in a Python dictionary keeps this project focused on LLM patterns, not storage infrastructure. The dictionary resets when you restart the server.

Now you need the endpoint function that uses these models. The key difference from /chat is that this endpoint sends the entire conversation history to Ollama, not just the latest message.

  • In main.py, scroll to the bottom of the file (below the chat function). Add the conversation endpoint by pasting this code:
@app.post("/conversation", response_model=ConversationResponse)
async def conversation(request: ConversationRequest):
    # Create a new conversation with system prompt if this ID is new
    if request.conversation_id not in conversations:
        conversations[request.conversation_id] = [
            {"role": "system", "content": request.system_prompt}
        ]

    # Append the user's new message to the history
    conversations[request.conversation_id].append(
        {"role": "user", "content": request.message}
    )

    # Send the FULL conversation history to Ollama
    async with httpx.AsyncClient() as client:
        response = await client.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": conversations[request.conversation_id],
                "stream": False,
                "options": {"temperature": request.temperature},
            },
            timeout=120.0,
        )
        response.raise_for_status()

    data = response.json()
    assistant_message = data["message"]["content"]

    # Store the assistant's reply so it's included in future requests
    conversations[request.conversation_id].append(
        {"role": "assistant", "content": assistant_message}
    )

    return ConversationResponse(
        response=assistant_message,
        model=data["model"],
        conversation_id=request.conversation_id,
        message_count=len(conversations[request.conversation_id]),
    )

What does this endpoint do?

When a new conversation_id arrives, the endpoint creates a fresh message list with the system prompt. On every subsequent call with the same ID, it appends the user message, sends the entire history to Ollama, and appends the assistant's reply.

This is exactly how ChatGPT-style applications work under the hood. The model itself has no memory. The application is responsible for replaying the full conversation context on each request.

  • Save main.py.

Seeing a syntax error when the server reloads?

Check that the ConversationRequest and ConversationResponse classes are placed between the ChatResponse class and the conversations dict, not inside another function.

Make sure there are no extra indentation levels on the new classes. They should be at the top level of the file, aligned with ChatRequest.

help me fix a syntax error in my FastAPI conversation endpoint

Add a DELETE endpoint to clear conversations

Conversations will grow indefinitely unless you provide a way to clear them. A DELETE endpoint lets you reset a conversation when you are done testing or want a fresh start.

  • In main.py, add this endpoint below the conversation function:
@app.delete("/conversation/{conversation_id}")
async def delete_conversation(conversation_id: str):
    # Remove conversation from memory if it exists
    if conversation_id in conversations:
        del conversations[conversation_id]
        return {"status": "deleted", "conversation_id": conversation_id}
    raise HTTPException(status_code=404, detail="Conversation not found")

What does this endpoint do?

The {conversation_id} in the URL is a path parameter. FastAPI extracts it automatically and passes it to your function. If the ID exists in the dictionary, it is deleted. If not, the endpoint returns a 404 error.

  • Save main.py.
  • If your server from the previous step is still running with --reload, it will pick up the changes automatically. If not, start it by running:
uvicorn main:app --reload
  • Open http://localhost:8000/docs in your browser. You should see the new /conversation POST and DELETE endpoints listed alongside your existing /health and /chat endpoints.

Don't see the new endpoints in /docs?

Make sure you saved main.py after adding the new code. If the server is running without --reload, stop it with Ctrl+C and restart it.

Check that the new endpoint functions are at the top level of the file (not accidentally nested inside the chat function).

help me debug why my new FastAPI endpoints are not showing up

Test multi-turn conversation memory

The real test of conversation memory is asking a follow-up question that only makes sense if the model knows what you said earlier. You will send two messages using the same conversation_id and verify the second response references the first.

  • Open a new terminal tab (keep the server running in the first tab).
  • Send the first message to start a conversation by running:
curl -X POST http://localhost:8000/conversation \
  -H "Content-Type: application/json" \
  -d '{"message": "What is Python?", "conversation_id": "test-1"}'

What should I see?

You should see a JSON response with a response field containing the model's explanation of Python, plus "message_count": 3 (system prompt + user message + assistant reply).

  • Now send a follow-up question that depends on the previous answer by running:
curl -X POST http://localhost:8000/conversation \
  -H "Content-Type: application/json" \
  -d '{"message": "What are its main use cases?", "conversation_id": "test-1"}'

The model should respond with Python's use cases, not ask "what do you mean by 'its'?" This proves it remembers the earlier context.

  • Test the DELETE endpoint by clearing the conversation:
curl -X DELETE http://localhost:8000/conversation/test-1
  • You should see {"status":"deleted","conversation_id":"test-1"} in the response.

Why does message_count grow?

The count includes the system prompt, every user message, and every assistant reply. After two exchanges, the count is 5 (1 system + 2 user + 2 assistant). This growing list is what gives the model context about the full conversation.

Model not remembering context?

Make sure both curl commands use the same conversation_id value. If the IDs differ, the model treats them as separate conversations.

If you restarted the server between requests, the in-memory dictionary was cleared. Send both messages without restarting.

help me debug why my conversation endpoint is not maintaining context

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and your server is running.

ⓧ I'd like to double check the full code

Here is the complete main.py including everything from previous steps and all new conversation code:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx

app = FastAPI(title="AI Assistant API")

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "llama3.2"


class ChatRequest(BaseModel):
    message: str
    system_prompt: str = "You are a helpful assistant."
    temperature: float = 0.7


class ChatResponse(BaseModel):
    response: str
    model: str


class ConversationRequest(BaseModel):
    message: str
    conversation_id: str
    system_prompt: str = "You are a helpful assistant."
    temperature: float = 0.7


class ConversationResponse(BaseModel):
    response: str
    model: str
    conversation_id: str
    message_count: int


conversations: dict[str, list[dict]] = {}


@app.get("/health")
async def health_check():
    try:
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                "http://localhost:11434/api/tags", timeout=5.0
            )
            resp.raise_for_status()
        return {"status": "healthy", "ollama": "connected"}
    except httpx.RequestError:
        raise HTTPException(status_code=503, detail="Ollama is not running")


@app.post("/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
    messages = [
        {"role": "system", "content": request.system_prompt},
        {"role": "user", "content": request.message},
    ]

    async with httpx.AsyncClient() as client:
        response = await client.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": messages,
                "stream": False,
                "options": {"temperature": request.temperature},
            },
            timeout=120.0,
        )
        response.raise_for_status()

    data = response.json()
    return ChatResponse(
        response=data["message"]["content"],
        model=data["model"],
    )


@app.post("/conversation", response_model=ConversationResponse)
async def conversation(request: ConversationRequest):
    if request.conversation_id not in conversations:
        conversations[request.conversation_id] = [
            {"role": "system", "content": request.system_prompt}
        ]

    conversations[request.conversation_id].append(
        {"role": "user", "content": request.message}
    )

    async with httpx.AsyncClient() as client:
        response = await client.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": conversations[request.conversation_id],
                "stream": False,
                "options": {"temperature": request.temperature},
            },
            timeout=120.0,
        )
        response.raise_for_status()

    data = response.json()
    assistant_message = data["message"]["content"]

    conversations[request.conversation_id].append(
        {"role": "assistant", "content": assistant_message}
    )

    return ConversationResponse(
        response=assistant_message,
        model=data["model"],
        conversation_id=request.conversation_id,
        message_count=len(conversations[request.conversation_id]),
    )


@app.delete("/conversation/{conversation_id}")
async def delete_conversation(conversation_id: str):
    if conversation_id in conversations:
        del conversations[conversation_id]
        return {"status": "deleted", "conversation_id": conversation_id}
    raise HTTPException(status_code=404, detail="Conversation not found")

Your AI assistant now maintains full conversation context. It can answer follow-up questions, remember what was discussed earlier, and reset on demand. This is the same pattern that powers every chat-based AI product in production today. Next up, you will tackle the Secret Mission and make your API stream responses in real time.

Secret mission

Add Streaming Responses

Ever noticed how ChatGPT types out its answers word by word instead of making you wait for the full response? That is streaming in action. In this secret mission, you will add a streaming endpoint to your API using Server-Sent Events so the AI sends tokens in real time.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama installed on macOS (running as a background process, free).
  • Llama 3.2 model (approximately 2GB of disk space in ~/.ollama).
  • The ai-assistant/ project folder on your Desktop (contains first_call.py, main.py, and a Python virtual environment).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your AI assistant API or want Ollama available for other projects.

  • Ollama stays in the background and uses minimal resources when idle. The model automatically unloads from RAM after 5 minutes of inactivity.
  • Your project files remain on your Desktop, ready for the next session.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and the model stay on disk so you can resume later.

  • Stop your FastAPI server by pressing Ctrl+C in the terminal where it is running.
  • Quit Ollama by clicking the Ollama icon in the menu bar, then selecting Quit Ollama. This unloads the model from RAM.
  • To resume later, reopen Ollama from your Applications folder and reactivate your virtual environment.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

  • Stop your FastAPI server by pressing Ctrl+C in the terminal where it is running.
  • Quit Ollama by clicking the Ollama icon in the menu bar, then selecting Quit Ollama.
  • Remove the Llama 3.2 model by running:
ollama rm llama3.2
  • Remove the project folder by running:
cd ~/Desktop
rm -rf ai-assistant/
  • Uninstall Ollama completely by running these commands:
sudo rm -rf /Applications/Ollama.app
sudo rm /usr/local/bin/ollama
rm -rf ~/Library/Application\ Support/Ollama
rm -rf ~/Library/Saved\ Application\ State/com.electron.ollama.savedState
rm -rf ~/Library/Caches/com.electron.ollama/
rm -rf ~/Library/Caches/ollama
rm -rf ~/Library/WebKit/com.electron.ollama
rm -rf ~/.ollama

What do these commands do?

The first two commands remove the Ollama application and its CLI binary. The remaining commands clean up cached data, saved application state, and model files stored in ~/.ollama.

You may be prompted for your password because some files require administrator permissions to delete.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully functional AI assistant API running entirely on your own machine.

You've learned how to:

  • Installed and ran a local LLM using Ollama and made programmatic calls to it with Python and httpx.
  • Built a production-style FastAPI service with system prompts and temperature control for prompt engineering.
  • Added conversation memory to maintain multi-turn chat context across multiple API requests.
  • Secret Mission: Added real-time streaming responses using Server-Sent Events.

Ready to quiz yourself?