Build an AI Chat API with Python

Build a local REST API that queries an AI model and streams responses.

Introduction

30 Second Summary

Every time you ask ChatGPT a question, your message travels to an API that sends it to an AI model and streams the answer back to you. That same architecture powers every AI product you use daily.

In this project, you will build your own AI chat API that runs entirely on your local machine. You will send questions via HTTP and get intelligent answers back from a real language model.

What You'll Build

A FastAPI-powered REST API that accepts questions via HTTP, queries a local Ollama language model, and returns AI-generated answers with both standard and streaming response modes.

By the end of this project, you'll have:

  • A working REST API with health check and chat endpoints validated by Pydantic models.
  • A local Ollama integration that queries an AI model without any API costs.
  • A streaming endpoint that sends tokens in real-time, just like ChatGPT's typing effect.
  • Secret Mission: Add conversation memory so the AI remembers previous messages in the same chat session.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, loops, if-statements). No prior experience with FastAPI, Pydantic, or async Python is required.

Install Python and Ollama

Your AI chat API needs two core tools to work.

Python is the programming language you will write your API in. Ollama is the tool that runs an AI model locally on your computer, so you can query it without paying for cloud API calls.

In this step, get ready to:
  • Install Python 3.13 on your machine.
  • Install Ollama and verify it is running.
  • Download a small AI model you can chat with locally.
Install Python 3.13

Python is the language your API code will run on. You need version 3.13 or newer installed on your system.

  • Open your browser and go to python.org/downloads.
  • Click the large Download Python 3.13.x button at the top of the page.
  • Once the installer downloads, double-click it to launch.
  • On the first screen of the installer, check the box that says Add python.exe to PATH. This is critical. Without it, your terminal will not recognize the python command.
  • Click Install Now.
  • Wait for the installation to complete, then click Close.

Now verify that Python installed correctly.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Check your Python version by running this command:
python --version

✔️ I see Python 3.13.x

You should see something like Python 3.13.13. The exact patch number does not matter as long as it starts with 3.13. Python is ready to go.

ⓧ Command not found

If PowerShell does not recognize the python command, the PATH setting was missed during installation.

  • Close PowerShell completely.
  • Re-run the Python installer (double-click the downloaded file again).
  • This time, make sure Add python.exe to PATH is checked before clicking Install Now.
  • Open a brand new PowerShell window and run python --version again.

Why a new PowerShell window?

PATH changes only take effect in terminals opened after the change. Any PowerShell window that was already open will not see the updated PATH.

Install Ollama

Ollama lets you run large language models (LLMs) directly on your computer. Instead of sending your data to a cloud service, Ollama hosts the AI model locally.

  • Open your browser and go to ollama.com/download/windows.
  • Download OllamaSetup.exe.
  • Double-click the downloaded file to install. No admin rights are needed.
  • Follow the prompts to complete the installation.

What is Ollama?

Ollama is an open-source tool that downloads, manages, and runs AI models locally. It exposes an API at http://localhost:11434 that your Python code will talk to.

Think of it as a local server for AI. Your FastAPI app will send questions to Ollama, and Ollama passes them to the AI model running on your machine.

After installation, Ollama starts running in the background automatically.

  • Go back to your PowerShell window (or open a new one).
  • Verify Ollama is installed by running this command:
ollama --version

✔️ I see a version number

You should see something like ollama version 0.9.5 (or newer). Ollama is installed and running.

ⓧ Command not found

If PowerShell does not recognize ollama, the installer may not have finished adding it to your PATH.

  • Close all PowerShell windows.
  • Check your system tray (bottom-right of the taskbar) for the Ollama icon to confirm it is running.
  • Open a new PowerShell window and try ollama --version again.
  • If it still does not work, restart your computer and try once more.
Pull the AI model

Ollama is installed, but it does not come with any AI models out of the box. You need to download one.

You will use qwen2.5:1.5b, a small but capable language model that is only about 1GB. It fits comfortably in memory alongside your other running applications.

  • Download the model by running this command in PowerShell:
ollama pull qwen2.5:1.5b

What does this command do?

This downloads the Qwen 2.5 model (1.5 billion parameters) from Ollama's model library to your machine. The download is about 1GB, so it may take a few minutes depending on your internet speed.

  • Once the download finishes, confirm the model is available by running:
ollama list

You should see qwen2.5:1.5b listed in the output along with its size.

Model not showing up?

Make sure the ollama pull command completed without errors. If you see a connection error, check that Ollama is still running in your system tray.

If the download was interrupted, run ollama pull qwen2.5:1.5b again. It will resume where it left off.

help me troubleshoot why ollama pull is not working

Your development environment is ready. Python can run your code, and Ollama can serve AI responses. Next up, you will create your project folder and build your first API endpoint.

Create Your Project and First Endpoint

Your tools are installed and ready. Python, Ollama, and the qwen2.5:1.5b model are all working on your machine.

Now you need a way to send questions to that AI model over HTTP and get answers back. In this step, you will create a FastAPI application with a health check endpoint and a chat endpoint that validates requests using Pydantic models.

In this step, get ready to:
  • Set up a Python virtual environment with FastAPI and Ollama packages installed.
  • Create a health check endpoint that confirms your API is running.
  • Define Pydantic models that validate chat requests and responses.
Set up your project folder and virtual environment

Before writing any code, you need a dedicated folder for your project and an isolated Python environment. A virtual environment keeps your project's packages separate from your system Python.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop.
  • Click New Folder, name it ai-chat-api, then select it and click Select Folder.
  • If VS Code asks whether you trust the authors of the files in this folder, click Yes, I trust the authors.

VS Code is now open with your empty ai-chat-api folder.

  • Open the integrated terminal by pressing Ctrl+` (backtick key, usually above Tab).

What is a virtual environment?

A virtual environment is an isolated copy of Python that lives inside your project folder. Packages you install here stay separate from your system Python, so different projects can use different package versions without conflict.

  • Create a virtual environment by running this command:
python -m venv venv
  • Activate the virtual environment by running:
venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

Seeing an execution policy error?

Windows PowerShell sometimes blocks scripts by default. Run this command to allow local scripts, then try the activate command again:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

help me fix the PowerShell execution policy error when activating my Python virtual environment

  • Install the project dependencies by running:
pip install "fastapi[standard]" ollama

This installs FastAPI (with its development server), Pydantic (included as a FastAPI dependency), and the official Ollama Python client.

What does this command install?

The fastapi[standard] package includes FastAPI itself, the Uvicorn web server, and the fastapi CLI tool. The ollama package is the official Python client for communicating with your local Ollama models.

  • Confirm the installation succeeded by running:
pip show fastapi

You should see output showing the FastAPI package name and version number.

Pip install failed?

Make sure you see (venv) in your terminal prompt. If you do not, run venv\Scripts\activate first, then try the install command again.

help me troubleshoot why pip install failed in my virtual environment

Create your first FastAPI app with a health check endpoint

Every API needs a way to confirm it is running. A health check endpoint is a simple GET route that returns a status message. It is the first thing you build to prove your server works.

  • In the VS Code Explorer sidebar (left panel), click the New File icon.
  • Name the file main.py and press Enter.
  • Paste the following code into main.py:
from fastapi import FastAPI


# Create the FastAPI application instance
app = FastAPI(title="AI Chat API", version="0.1.0")


# Health check endpoint - confirms the API is running
@app.get("/health")
async def health_check():
    return {"status": "healthy", "service": "ai-chat-api"}

What does this code do?

The FastAPI() call creates your application instance. The title and version parameters show up in the auto-generated documentation.

The @app.get("/health") decorator tells FastAPI to call the function below it whenever someone sends a GET request to /health. The function returns a Python dictionary, which FastAPI automatically converts to JSON.

  • Save main.py by pressing Ctrl+S.
  • Start the development server by running this command in your terminal:
fastapi dev main.py

You should see output that includes Server started at http://127.0.0.1:8000 and Uvicorn running on http://127.0.0.1:8000.

What is fastapi dev?

The fastapi dev command starts a local development server with auto-reload enabled. Every time you save a change to your code, the server restarts automatically. You do not need to stop and restart it manually.

  • Open your browser and go to http://127.0.0.1:8000/docs.

You should see the Swagger UI page with your API title "AI Chat API" and the /health endpoint listed. This documentation page is generated automatically by FastAPI.

Don't see the Swagger UI page?

Make sure the terminal still shows the server running. If you see an error, check that you saved main.py and that your virtual environment is active ((venv) in the prompt).

Also confirm the URL is exactly http://127.0.0.1:8000/docs with no extra characters.

help me debug why my FastAPI server is not starting or the docs page is not loading

Define Pydantic models for chat requests and responses

Your chat endpoint needs to know exactly what shape of data to expect from users and what shape of data to send back. Pydantic models define these shapes and automatically reject invalid requests before your code even runs.

  • In main.py, find the line from fastapi import FastAPI at the top of the file.
  • Add this import on the line directly below it:
from pydantic import BaseModel
  • Now find the line app = FastAPI(title="AI Chat API", version="0.1.0") in your file.
  • Add the following two model classes below it (leave a blank line between the app line and the first class):
class ChatRequest(BaseModel):
    # The question the user sends to the AI
    message: str


class ChatResponse(BaseModel):
    # The AI's answer and which model generated it
    message: str
    model: str

What are Pydantic models?

Pydantic models are Python classes that define the exact fields and types your data must have. When a request comes in, FastAPI uses these models to validate the data automatically. If someone sends a number where a string is expected, they get a clear error message without you writing any validation code.

ChatRequest requires a message string from the user. ChatResponse shapes the reply with both the AI's message and which model generated it.

  • Now add the chat endpoint at the bottom of the file, below the health_check function:
@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    # Return a hardcoded reply for now (we'll connect to Ollama in the next step)
    return ChatResponse(
        message="This is a hardcoded reply.",
        model="qwen2.5:1.5b",
    )

What does this endpoint do?

The @app.post("/chat") decorator creates a POST endpoint. The response_model=ChatResponse parameter tells FastAPI to validate the output shape and display it in the docs.

The request: ChatRequest parameter tells FastAPI to parse the incoming JSON body and validate it against the ChatRequest model. For now the reply is hardcoded. You will connect it to the real AI model in the next step.

  • Save main.py by pressing Ctrl+S.
  • Go back to your browser at http://127.0.0.1:8000/docs and refresh the page.

You should now see both the /health GET endpoint and the /chat POST endpoint listed in the Swagger UI.

  • Click the /chat endpoint to expand it.
  • Click Try it out.
  • Replace the example JSON body with:
{"message": "Hello AI"}
  • Click Execute.

You should see a response with "message": "This is a hardcoded reply." and "model": "qwen2.5:1.5b".

Seeing a validation error or 500 response?

If you see a 422 Unprocessable Entity error, make sure your JSON body has the exact field name message (lowercase) and the value is a string in quotes.

If you see a 500 Internal Server Error, check your terminal for a Python traceback. The most common cause is a typo in the model class names or field names.

help me debug the error I'm seeing when testing my /chat endpoint in Swagger UI

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and that both endpoints appear in the Swagger UI.

ⓧ I'd like to double check the full code

Here is the complete main.py at this point. Make sure your file matches exactly:

from fastapi import FastAPI
from pydantic import BaseModel


app = FastAPI(title="AI Chat API", version="0.1.0")


class ChatRequest(BaseModel):
    message: str


class ChatResponse(BaseModel):
    message: str
    model: str


@app.get("/health")
async def health_check():
    return {"status": "healthy", "service": "ai-chat-api"}


@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    return ChatResponse(
        message="This is a hardcoded reply.",
        model="qwen2.5:1.5b",
    )

Your API skeleton is working. You have validated requests, typed responses, and auto-generated documentation. Next up, you will replace that hardcoded reply with a real AI-generated answer from Ollama.

Connect to Ollama for AI Responses

Your API currently returns the same hardcoded reply no matter what question you ask. The /chat endpoint exists, but it is not connected to any AI model yet.

In this step, you will wire up the Ollama chat function so your endpoint sends the user's question to the local AI model and returns a real, generated answer. You will also add error handling so the API responds gracefully when something goes wrong.

In this step, get ready to:
  • Connect the /chat endpoint to the local Ollama model.
  • Test the AI integration through Swagger UI.
  • Add error handling for when Ollama is unavailable.
Replace the hardcoded reply with a real Ollama call

The ollama Python package provides a chat function that sends a message to your local model and returns the AI's response. You installed this package in Step 1, so it is ready to import.

  • In main.py, find the import line at the top of the file:
from fastapi import FastAPI
  • Add a new import line directly below it, so your first two lines look like this:
from fastapi import FastAPI
from ollama import chat
  • Now scroll down to the chat_endpoint function. Find these lines:
@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    return ChatResponse(
        message="This is a hardcoded reply.",
        model="qwen2.5:1.5b",
    )
  • Replace the entire function body (everything inside chat_endpoint) with the following code:
@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    # Send the user's message to the local Ollama model
    response = chat(
        model="qwen2.5:1.5b",
        messages=[{"role": "user", "content": request.message}],
    )
    # Extract the AI's reply and return it as a ChatResponse
    return ChatResponse(
        message=response.message.content,
        model="qwen2.5:1.5b",
    )

What does this code do?

The chat() function sends a message to your local Ollama model. The messages parameter takes a list of message dictionaries, each with a role and content field.

The response object has a message.content attribute containing the AI's generated text. You pass that into your ChatResponse model so it gets validated by Pydantic before being sent back to the caller.

  • Save main.py.
  • Check your terminal where fastapi dev is running. You should see a message indicating the server reloaded successfully.

Seeing an error in the terminal?

If you see ImportError: cannot import name 'chat' from 'ollama', make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.

If the server crashes with a SyntaxError, double-check that the square brackets and curly braces in the messages list match the code above exactly.

help me fix an import error or syntax error in my FastAPI app

Test the AI integration

Your endpoint is now connected to the AI model. Time to ask it a real question and see what comes back.

  • Open your browser and go to http://127.0.0.1:8000/docs.
  • Click the POST /chat endpoint to expand it.
  • Click Try it out.
  • Replace the request body with the following JSON:
{
  "message": "What is Python?"
}
  • Click Execute.

After a few seconds, you should see a response body containing an AI-generated explanation of Python. The response will have both a message field (the AI's answer) and a model field showing qwen2.5:1.5b.

  • Try a few more questions like "Explain REST APIs in one sentence" or "What is 2 + 2?" to confirm the model responds differently each time.

Why is the first response slow?

Ollama loads the model into memory on the first request. This can take a few seconds. Subsequent requests are much faster because the model stays loaded.

Getting a 500 Internal Server Error?

This usually means Ollama is not running. Check that the Ollama icon appears in your system tray (bottom-right of your taskbar). If it is missing, search for Ollama in the Start menu and open it.

You can also verify by opening a new PowerShell window and running ollama list. You should see qwen2.5:1.5b in the output.

help me debug why my Ollama call is failing

Add error handling

Your endpoint works when Ollama is running, but what happens if Ollama crashes or the model gets deleted? Without error handling, your API would return an unhelpful 500 error. A proper REST API returns a clear error message with a meaningful status code.

  • In main.py, find the first import line:
from fastapi import FastAPI
  • Add HTTPException to that import, so the line now looks like this:
from fastapi import FastAPI, HTTPException
  • Now scroll down to your chat_endpoint function and replace it with this version that wraps the Ollama call in a try/except block:
@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    try:
        # Send the user's message to the local Ollama model
        response = chat(
            model="qwen2.5:1.5b",
            messages=[{"role": "user", "content": request.message}],
        )
        # Extract the AI's reply and return it as a ChatResponse
        return ChatResponse(
            message=response.message.content,
            model="qwen2.5:1.5b",
        )
    except Exception as e:
        # Return a clear error if Ollama fails
        raise HTTPException(status_code=503, detail=f"Ollama error: {str(e)}")

What does this error handling do?

The try: block attempts the Ollama call. If anything goes wrong (Ollama is not running, the model is missing, or a network error occurs), Python jumps to the except block instead of crashing.

HTTPException is FastAPI's way of returning an error response with a specific status code. Status code 503 means "Service Unavailable", which tells the caller that a downstream service (Ollama) is not responding.

  • Save main.py.
  • Check your terminal to confirm the server reloaded without errors.
  • Go back to Swagger UI at http://127.0.0.1:8000/docs and send another question to POST /chat. You should still get a successful AI response.

Test the error path

To see your error handling in action, right-click the Ollama icon in your system tray and click Quit Ollama. Then send a request through Swagger UI. You should get a 503 response with a clear error message instead of a generic 500.

Restart Ollama before moving on to the next step.

Server not reloading?

Make sure you saved the file. If the terminal shows a traceback, check that HTTPException is imported on the same line as FastAPI, separated by a comma.

Also verify that the raise keyword appears before HTTPException in your except block. Without raise, the exception is created but never sent.

help me fix my error handling code

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and that your server is running without errors.

ⓧ I'd like to double check the full code

Here is the full main.py as it should look at the end of this step:

from fastapi import FastAPI, HTTPException
from ollama import chat
from pydantic import BaseModel


app = FastAPI(title="AI Chat API", version="0.1.0")


class ChatRequest(BaseModel):
    message: str


class ChatResponse(BaseModel):
    message: str
    model: str


@app.get("/health")
async def health_check():
    return {"status": "healthy", "service": "ai-chat-api"}


@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    try:
        response = chat(
            model="qwen2.5:1.5b",
            messages=[{"role": "user", "content": request.message}],
        )
        return ChatResponse(
            message=response.message.content,
            model="qwen2.5:1.5b",
        )
    except Exception as e:
        raise HTTPException(status_code=503, detail=f"Ollama error: {str(e)}")

Your API is now a real AI-powered endpoint. Next up, you will add streaming so responses arrive token by token, just like ChatGPT's typing effect.

Add Streaming Responses

Your API can now accept a question and return a complete AI-generated answer. That works, but there is a problem.

When you ask a complex question, the user has to wait in silence until the entire response is generated. FastAPI supports streaming responses, which send tokens to the user as they are generated. This is the same "typing" effect you see in ChatGPT. In this step, you will add a streaming endpoint to your API.

In this step, get ready to:
  • Add a streaming endpoint that yields tokens in real-time.
  • Learn how async/await makes streaming possible.
  • Test the streaming endpoint with a PowerShell command.
Create a streaming endpoint

Streaming means sending pieces of the AI response as they are generated, instead of waiting for the full answer. You need new imports to support this.

  • In main.py, find the import section at the top of the file:
from fastapi import FastAPI, HTTPException
from ollama import chat
from pydantic import BaseModel
  • Replace the entire import section with the following updated imports:
from collections.abc import AsyncIterable

from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from ollama import chat, AsyncClient
from pydantic import BaseModel

What changed in the imports?

  • AsyncIterable is a type hint that tells Python this function yields values asynchronously.
  • StreamingResponse is FastAPI's built-in class for sending data chunk by chunk instead of all at once.
  • AsyncClient is Ollama's async client, which lets you stream tokens from the model without blocking other requests.

Now you will add the streaming endpoint itself. This is the core of the step.

  • In main.py, scroll to the bottom of the file (below the chat_endpoint function).
  • Add the streaming endpoint by pasting this code at the end of the file:
@app.post("/chat/stream", response_class=StreamingResponse)
async def chat_stream_endpoint(request: ChatRequest) -> AsyncIterable[str]:
    # Inner async generator that yields tokens one at a time
    async def generate():
        try:
            # Create an async Ollama client for non-blocking calls
            client = AsyncClient()
            # Start a streaming chat request to the model
            stream = await client.chat(
                model="qwen2.5:1.5b",
                messages=[{"role": "user", "content": request.message}],
                stream=True,
            )
            # Yield each token as it arrives from the model
            async for chunk in stream:
                content = chunk["message"]["content"]
                if content:
                    yield content
        except Exception as e:
            yield f"\n\nError: {str(e)}"

    return StreamingResponse(generate(), media_type="text/plain")

What does this code do?

  • The @app.post("/chat/stream") decorator registers a new POST endpoint. The response_class=StreamingResponse tells FastAPI this endpoint streams its output.
  • generate() is an inner async generator function. Each time it calls yield, it sends a piece of text to the client immediately.
  • await client.chat(..., stream=True) asks Ollama to return tokens one at a time instead of waiting for the full response.
  • The async for chunk in stream loop processes each token as it arrives and yields it to the client.
  • Save main.py.
  • Your FastAPI dev server should auto-reload. Check the terminal for a message confirming the reload with no errors.

Seeing a syntax error in the terminal?

  • Make sure the generate() function is indented inside chat_stream_endpoint. It is a nested function.
  • Check that the return StreamingResponse(...) line is at the same indentation level as async def generate(). It belongs to the outer function, not the inner one.
  • Verify that from fastapi.responses import StreamingResponse appears in your imports. A missing import causes a NameError.

Help me fix a syntax error in my streaming endpoint

Understand how async streaming works

The streaming endpoint uses two Python keywords that are worth understanding before you move on.

What are async and await?

The async def keyword tells Python this function can pause and wait for slow operations (like the AI generating the next word) without blocking other requests. While one request waits for tokens, your server can handle other incoming requests.

The await keyword is where the pausing actually happens. When your code hits await client.chat(...), Python pauses this function and goes to do other work until Ollama responds. This is what makes real-time streaming possible.

Why not use the sync client for streaming?

Your /chat endpoint uses the synchronous chat() function because it waits for the full response anyway. The streaming endpoint needs AsyncClient because it must yield tokens as they arrive without holding up the entire server.

Test the streaming endpoint

Your streaming endpoint is ready. Time to confirm it actually works by sending a request.

  • Open a new PowerShell window (keep your dev server running in the VS Code terminal).
  • Test the streaming endpoint by running this command:
Invoke-RestMethod -Uri "http://127.0.0.1:8000/chat/stream" -Method POST -Body '{"message": "Tell me a short story"}' -ContentType "application/json"
  • You should see a complete AI-generated short story printed in your terminal. The endpoint streamed the tokens internally to build this response.

See streaming in action with Swagger UI

You can also test the /chat/stream endpoint in your browser at http://127.0.0.1:8000/docs. Click the Try it out button, enter a message, and click Execute. The response body shows the full streamed output.

Getting a connection error?

  • Make sure your FastAPI dev server is still running in the VS Code terminal. If it crashed, run fastapi dev main.py again.
  • Confirm Ollama is running in the background. Run ollama list in PowerShell. If it fails, restart Ollama from your Start menu.

Help me debug my streaming endpoint connection

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and your server reloaded without errors.

ⓧ I'd like to double check the full code

Here is the complete main.py file as it should look at the end of this step:

from collections.abc import AsyncIterable

from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from ollama import chat, AsyncClient
from pydantic import BaseModel


app = FastAPI(title="AI Chat API", version="0.1.0")


class ChatRequest(BaseModel):
    message: str


class ChatResponse(BaseModel):
    message: str
    model: str


@app.get("/health")
async def health_check():
    return {"status": "healthy", "service": "ai-chat-api"}


@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    try:
        response = chat(
            model="qwen2.5:1.5b",
            messages=[{"role": "user", "content": request.message}],
        )
        return ChatResponse(
            message=response.message.content,
            model="qwen2.5:1.5b",
        )
    except Exception as e:
        raise HTTPException(status_code=503, detail=f"Ollama error: {str(e)}")


@app.post("/chat/stream", response_class=StreamingResponse)
async def chat_stream_endpoint(request: ChatRequest) -> AsyncIterable[str]:
    async def generate():
        try:
            client = AsyncClient()
            stream = await client.chat(
                model="qwen2.5:1.5b",
                messages=[{"role": "user", "content": request.message}],
                stream=True,
            )
            async for chunk in stream:
                content = chunk["message"]["content"]
                if content:
                    yield content
        except Exception as e:
            yield f"\n\nError: {str(e)}"

    return StreamingResponse(generate(), media_type="text/plain")

Your AI Chat API is now fully functional with both standard and streaming response modes. You have built a production-style API from scratch, complete with data validation, error handling, and real-time streaming. Next up, the Secret Mission challenges you to add conversation memory.

Secret mission

Add Conversation Memory

Right now, every message you send starts a brand new conversation. The AI has no memory of what you asked before. Your mission is to add session-based conversation memory so the AI can reference earlier messages, just like ChatGPT does.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Python 3.13 installation.
  • Ollama installation and background service.
  • qwen2.5:1.5b model (~1GB on disk).
  • ai-chat-api project folder with virtual environment and dependencies.
  • main.py FastAPI application.

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue with Project 2 of the AI Knowledge Assistant series soon.

  • Ollama continues running in the background and uses minimal resources when idle.
  • Your virtual environment stays on disk ready for next time.
  • To restart your API later, open VS Code, activate your virtual environment, and run fastapi dev main.py.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and installations stay on disk for when you return.

  • Stop the FastAPI dev server by pressing Ctrl+C in the terminal where it is running.
  • Deactivate your virtual environment by running deactivate in the terminal.
  • Quit Ollama by right-clicking the Ollama icon in the system tray and selecting Quit.
  • To resume later, restart Ollama from the Start menu and reactivate your virtual environment with venv\Scripts\activate.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up disk space but means you will need to reinstall everything if you want to redo the project.

  • Delete the ai-chat-api folder from your Desktop. This removes your code, virtual environment, and all installed packages.
  • Remove the downloaded model by running ollama rm qwen2.5:1.5b in PowerShell. This frees up about 1GB of disk space.
  • Uninstall Ollama from Settings then Apps then Installed apps. Search for Ollama and click Uninstall.
  • Optionally uninstall Python 3.13 from the same Installed apps page. Only do this if you do not plan to use Python for other projects.

Nice Work!

Nice Work!

You just built your first AI-powered REST API from scratch. You can now send a question over HTTP and get an intelligent answer back from a model running entirely on your machine.

Here's what you accomplished:

  • Built a FastAPI application with Pydantic request and response validation.
  • Integrated a local LLM using the official Ollama Python library.
  • Implemented real-time token streaming using async/await.
  • Secret Mission: Added conversation memory with session tracking so the AI remembers earlier messages.

This is Project 1 of the AI Knowledge Assistant series. In Project 2, you will add RAG (Retrieval-Augmented Generation) so your API can answer questions from your own documents instead of just general knowledge.

Ready to quiz yourself?