Build an AI Backend with FastAPI

Build a local AI chat API using FastAPI and Ollama with Llama 3.1.

Introduction

30 Second Summary

Every time you ask ChatGPT or Claude a question, your message travels to a backend API that routes it to a language model and returns the response. That same architecture powers AI features in thousands of apps today.

In this project, you will build a local AI chat backend using FastAPI and Ollama. You'll create API endpoints that send messages to a Llama 3.1 model running on your own GPU and return AI-generated responses.

What You'll Build

A production-style AI chat backend that runs entirely on your machine, complete with a /chat endpoint, streaming support, and auto-generated interactive API documentation.

By the end of this project, you'll have:

  • Ollama installed and running with Llama 3.1 8B on your GPU for local AI inference.
  • A FastAPI application with /health, /chat, and /chat/stream endpoints.
  • Pydantic request/response models for type-safe API contracts and interactive Swagger documentation at /docs.
  • Secret Mission: Add conversation memory so your API maintains chat history across multiple requests.

Are there any prerequisites?

This project assumes you have moderate Python knowledge and experience using pip to install packages. You do not need prior experience with FastAPI or running local AI models.

Install Ollama and Pull Llama 3.1

Your AI backend needs a brain. Specifically, it needs a large language model running locally that can receive messages and generate responses.

In this step, you will install Ollama, a tool that runs LLMs on your machine with GPU acceleration. You will then download the Llama 3.1 8B model and confirm that the whole inference pipeline works.

In this step, get ready to:
  • Install Ollama on Windows and verify it is running.
  • Pull the Llama 3.1 8B model and confirm GPU acceleration.
  • Test the Ollama HTTP API directly from PowerShell.
Download and install Ollama

Ollama is a lightweight runtime that downloads, manages, and serves LLMs locally. It automatically detects your NVIDIA GPU and uses it for inference.

  • Open your web browser and navigate to the Ollama download page:
  • https://ollama.com/download/windows
  • Click the download button to get OllamaSetup.exe.
  • Run the installer and follow the prompts to complete installation.

What does the installer do?

The Ollama installer registers Ollama as a Windows service, adds it to your system PATH, and starts it automatically in the background. After installation, Ollama serves a local API at http://localhost:11434.

  • Once the installer finishes, open a brand new PowerShell window. Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Verify the installation by running this command:
ollama --version

✔️ I see ollama version is...

You should see output like ollama version is 0.x.x (the exact number will depend on the latest release). Ollama is installed and its background service is running.

  • Confirm Ollama's API is responding by opening your browser and navigating to http://localhost:11434.

You should see a page that says "Ollama is running". This confirms the background service is active and ready to serve models.

ⓧ 'ollama' is not recognized

This means your PowerShell session does not have Ollama on its PATH. The most common cause is that you opened this PowerShell window before the installer finished.

  • Close ALL open PowerShell and terminal windows completely.
  • Open a brand new PowerShell window using the Windows key search.
  • Run ollama --version again.

If it still does not work, the Ollama directory may not be in your PATH. Try running the installer again as Administrator by right-clicking the .exe and selecting Run as administrator.

Pull the Llama 3.1 8B model

Now that Ollama is running, you need to download a model. You will use Llama 3.1 8B, a capable open-source model that fits comfortably on your RTX 3050 GPU.

  • Download the model by running this command in PowerShell:
ollama pull llama3.1:8b

What does this command do?

This downloads the Llama 3.1 8B model files (approximately 4.9GB) to your local machine. Ollama stores models in a local registry so you only download once.

The 8B (8 billion parameter) version uses q4 quantization to fit within your GPU's VRAM. Your RTX 3050 will handle inference for this model size.

  • Wait for the download to complete. You will see a progress bar as the model files download.
  • Once complete, verify the model works by running a quick test:
ollama run llama3.1:8b "Hello, are you working?"

You should see the model generate a text response in your terminal. This confirms that Llama 3.1 is loaded and running inference on your machine.

  • Open Task Manager by pressing Ctrl+Shift+Esc.
  • Click the Performance tab, then select GPU in the left sidebar.
  • Confirm you see activity on your NVIDIA RTX 3050 while the model is generating.

Model download seems stuck or slow?

The download is approximately 4.9GB and depends on your internet speed. If it appears stuck, check that your network connection is stable.

If the download fails partway through, run ollama pull llama3.1:8b again. Ollama resumes from where it left off.

Help me troubleshoot my Ollama model download on Windows with an RTX 3050.

Test the Ollama HTTP API

Ollama exposes a REST API that your FastAPI application will talk to later. Before writing any Python code, you will confirm this API responds correctly.

  • Test the chat endpoint by running this command in PowerShell:
Invoke-WebRequest -Method POST -Body '{"model":"llama3.1:8b","messages":[{"role":"user","content":"Say hello"}],"stream":false}' -Uri http://localhost:11434/api/chat -ContentType "application/json"

What does this command do?

This sends an HTTP POST request to Ollama's chat API at http://localhost:11434/api/chat. The request body contains a JSON object specifying the model name and a user message.

Setting stream to false tells Ollama to return the full response at once instead of streaming individual tokens. This is the same API your Python backend will call.

You should see a response object with a StatusCode of 200 and Content containing a JSON payload with the model's reply.

Getting a connection refused error?

This means the Ollama service is not running. Look for the Ollama icon in your Windows system tray (bottom-right corner of the taskbar). If you do not see it, search for Ollama in the Start menu and launch it.

Help me fix Ollama not responding on localhost:11434 on Windows.

Your local LLM infrastructure is live. Ollama is serving Llama 3.1 8B on your GPU, and its HTTP API is responding to chat requests. Next up, you will set up a Python project that wraps this API in a production-style FastAPI backend.

Set Up Python Project and Virtual Environment

Ollama is running on your machine with Llama 3.1 8B loaded on your GPU. Now you need a Python application that can talk to it.

In this step, you will create a dedicated project folder with an isolated Python environment. This keeps your project's packages separate from anything else on your system.

In this step, get ready to:
  • Create the project directory and a Python 3.12 virtual environment.
  • Install FastAPI and the Ollama Python client.
  • Create the project file structure and open it in VS Code.
Create the project directory and virtual environment

A virtual environment is an isolated copy of Python that holds only the packages your project needs. Without one, installing packages globally can cause version conflicts between projects.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir agentic-ai-backend
cd agentic-ai-backend
  • Confirm the folder was created by running:
ls

You should see the agentic-ai-backend folder listed in the output.

  • Create a virtual environment inside your project folder by running:
python -m venv venv

What does this command do?

python -m venv venv tells Python to create a new virtual environment in a folder called venv.

The first venv is the Python module that creates virtual environments. The second venv is the folder name where it gets stored.

  • Activate the virtual environment by running:
.\venv\Scripts\activate
  • Verify that the virtual environment is using Python 3.12 by running:
python --version

You should see Python 3.12.x in the output. Your terminal prompt should also show (venv) at the beginning, confirming the virtual environment is active.

Don't see (venv) in your prompt?

If you get an error about execution policies, PowerShell might be blocking scripts. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned and then try the activate command again.

Make sure you are inside the agentic-ai-backend folder when you run the activate command.

Help me activate my Python virtual environment on Windows

Install project dependencies

Your project needs two main packages: FastAPI for building the API server, and the Ollama Python client for communicating with your local model.

  • Install both packages by running:
pip install "fastapi[standard]" ollama

What does this install?

fastapi[standard] installs FastAPI 0.139.0 along with Uvicorn (the server that runs your app), plus other useful extras like auto-generated documentation.

ollama installs the Ollama Python client 0.6.2 which gives you typed Python functions for chatting with your local model instead of writing raw HTTP requests.

You should see output ending with a success message listing the installed packages.

  • Lock your dependencies into a requirements.txt file by running:
pip freeze > requirements.txt

Why freeze dependencies?

requirements.txt records the exact versions of every package in your environment. Anyone who clones your project can run pip install -r requirements.txt to get the same setup.

Seeing a permission error or install failure?

Make sure your virtual environment is active (you should see (venv) in your prompt). Installing without the venv active will try to install globally and may require admin permissions.

If you see a network error, check your internet connection and try running the install command again.

Help me troubleshoot my pip install

Create the project file structure

Your project will have three Python files, each with a single responsibility. Separating concerns this way keeps the codebase maintainable as it grows.

  • Create the three project files by running:
New-Item -ItemType File -Path main.py, models.py, config.py

What is each file for?

  • main.py is the app entry point where your API endpoints live.
  • models.py holds your Pydantic schemas that define the shape of requests and responses.
  • config.py stores application settings like the model name and Ollama host URL.
  • Confirm the files exist by running:
ls

You should see config.py, main.py, models.py, requirements.txt, and venv in the output.

  • Open the project in VS Code by running:
code .

VS Code opens with your agentic-ai-backend folder in the Explorer sidebar. You should see all three .py files listed alongside requirements.txt and the venv folder.

VS Code didn't open?

If you see a message saying code is not recognized, VS Code's command line tools might not be in your system PATH. Open VS Code manually, click File, then Open Folder, and select the agentic-ai-backend folder on your Desktop.

Help me open my project in VS Code

Your project skeleton is ready with all dependencies installed. Next up, you will write the actual application code that turns these empty files into a working AI chat backend.

Build the FastAPI Application with Chat Endpoint

Your project is set up with all dependencies installed and an empty file structure ready to go. Now it's time to write the application code that connects FastAPI to your local Llama model through the Ollama Python client.

You'll define type-safe data models that describe exactly what your API accepts and returns. Then you'll build two endpoints: one to check that Ollama is reachable, and one to send chat messages to Llama 3.1.

In this step, get ready to:
  • Define configuration settings and Pydantic request/response models.
  • Build a /health endpoint and a /chat endpoint in FastAPI.
  • Run the server and test both endpoints using interactive Swagger docs.
Define the configuration and data models

Before writing any endpoint logic, you need to define how your app is configured and what shape data flows in and out of the API. Two small files handle this: config.py for settings, and models.py for Pydantic schemas.

  • In VS Code, open config.py and paste the following code:
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    # Connection URL for the local Ollama server
    ollama_host: str = "http://localhost:11434"
    # Which model to use for chat completions
    model_name: str = "llama3.1:8b"
    # App metadata used in the auto-generated docs
    app_name: str = "Agentic AI Backend"
    app_version: str = "0.1.0"


# Create a single settings instance the rest of the app imports
settings = Settings()

What does this code do?

The Settings class inherits from BaseSettings (provided by the pydantic-settings package, included with fastapi[standard]). Each field has a default value, so the app works out of the box without environment variables.

The settings = Settings() line at the bottom creates a single instance that every other file can import. This pattern keeps configuration centralized and easy to override later.

  • Save config.py.
  • Now open models.py and paste the following code:
from pydantic import BaseModel


# A single message in a conversation (user or assistant)
class Message(BaseModel):
    role: str
    content: str


# What the client sends to the /chat endpoint
class ChatRequest(BaseModel):
    messages: list[Message]
    stream: bool = False


# What the /chat endpoint returns
class ChatResponse(BaseModel):
    message: Message
    model: str
    total_duration: int | None = None

What does this code do?

These Pydantic models define the contract for your API. Message represents a single chat turn with a role (like user or assistant) and content (the actual text).

ChatRequest carries a list of messages (the conversation so far) and an optional stream flag. ChatResponse wraps the assistant's reply along with metadata like the model name and how long inference took.

  • Save models.py.
  • You won't see a visible change yet. The next file (main.py) imports both of these to power the endpoints.
Create the FastAPI application with endpoints

With settings and schemas defined, you can now build the actual REST API that accepts requests, talks to Ollama, and returns structured responses.

  • Open main.py and paste the following imports and app setup:
import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from ollama import chat

from config import settings
from models import ChatRequest, ChatResponse, Message

# Create the FastAPI app with metadata from settings
app = FastAPI(
    title=settings.app_name,
    version=settings.app_version,
)

What does this code do?

The imports pull in FastAPI itself, the Ollama chat function, and your local settings and models. The json and StreamingResponse imports are for the streaming endpoint you'll add in the next step.

The app = FastAPI(...) line creates the application instance. Passing title and version from your settings makes the auto-generated Swagger docs display your project name.

  • Below the app setup in main.py, add the /health endpoint:

@app.get("/health")
def health_check():
    try:
        from ollama import list as ollama_list

        # Query Ollama for available models
        models = ollama_list()
        return {
            "status": "healthy",
            "ollama": "connected",
            "models_available": len(models.models),
        }
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

What does this code do?

The /health endpoint tries to connect to Ollama and count the available models. If Ollama is running, it returns "status": "healthy" with a model count. If the connection fails, it catches the exception and returns "status": "unhealthy" with the error message.

The from ollama import list as ollama_list import is inside the function to avoid shadowing Python's built-in list at the module level.

  • Below the /health endpoint, add the /chat endpoint:

@app.post("/chat", response_model=ChatResponse)
def chat_endpoint(request: ChatRequest):
    # Convert Pydantic models to plain dicts for the Ollama client
    messages = [{"role": m.role, "content": m.content} for m in request.messages]

    # Call the local LLM via Ollama
    response = chat(
        model=settings.model_name,
        messages=messages,
    )

    # Wrap the Ollama response in our typed ChatResponse
    return ChatResponse(
        message=Message(
            role=response.message.role,
            content=response.message.content,
        ),
        model=response.model,
        total_duration=response.total_duration,
    )

What does this code do?

The /chat endpoint accepts a ChatRequest and converts the Pydantic Message objects into plain dictionaries that the Ollama client expects.

It then calls chat() with the model name from your settings and the conversation messages. The response from Ollama is wrapped in a ChatResponse so FastAPI can validate and serialize it automatically.

The total_duration field tells you how many nanoseconds the model spent generating the answer.

  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for each file at this point:

from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    ollama_host: str = "http://localhost:11434"
    model_name: str = "llama3.1:8b"
    app_name: str = "Agentic AI Backend"
    app_version: str = "0.1.0"


settings = Settings()
from pydantic import BaseModel


class Message(BaseModel):
    role: str
    content: str


class ChatRequest(BaseModel):
    messages: list[Message]
    stream: bool = False


class ChatResponse(BaseModel):
    message: Message
    model: str
    total_duration: int | None = None
import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from ollama import chat

from config import settings
from models import ChatRequest, ChatResponse, Message

app = FastAPI(
    title=settings.app_name,
    version=settings.app_version,
)


@app.get("/health")
def health_check():
    try:
        from ollama import list as ollama_list

        models = ollama_list()
        return {
            "status": "healthy",
            "ollama": "connected",
            "models_available": len(models.models),
        }
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}


@app.post("/chat", response_model=ChatResponse)
def chat_endpoint(request: ChatRequest):
    messages = [{"role": m.role, "content": m.content} for m in request.messages]

    response = chat(
        model=settings.model_name,
        messages=messages,
    )

    return ChatResponse(
        message=Message(
            role=response.message.role,
            content=response.message.content,
        ),
        model=response.model,
        total_duration=response.total_duration,
    )
Run the server and test the endpoints

Your code is complete. Time to start the development server and confirm everything works end to end.

  • Start the FastAPI development server by running this command in your terminal (make sure your virtual environment is still active):
fastapi dev main.py

What does this command do?

The fastapi dev command starts a development server with hot-reload enabled. Every time you save a file, the server restarts automatically so you see your changes immediately.

By default it serves on http://127.0.0.1:8000 and auto-detects main.py as the entry point.

  • Open your browser and navigate to http://127.0.0.1:8000/health.
  • You should see a JSON response showing "status": "healthy", "ollama": "connected", and a models_available count of at least 1.

Seeing "unhealthy" or a connection error?

  • Make sure Ollama is running. Open a new PowerShell window and run ollama list to confirm it responds.
  • Check that your virtual environment is active (you should see (venv) in your terminal prompt).
  • If you see an import error for pydantic_settings, confirm you installed fastapi[standard] (with the brackets) in the previous step.

Help me troubleshoot my FastAPI health endpoint returning unhealthy status

  • Now navigate to http://127.0.0.1:8000/docs in your browser to open the interactive Swagger documentation.
  • Click on the POST /chat endpoint to expand it.
  • Click Try it out.
  • Replace the request body with this JSON:
{
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France? Reply in one sentence."
    }
  ],
  "stream": false
}
  • Click Execute.
  • Wait a few seconds for the model to generate a response (the first request may take longer while the model loads into GPU memory).
  • You should see a JSON response containing a message object with role: "assistant" and a content field answering your question.

Request timing out or returning an error?

  • The first request can take 10-30 seconds while Llama 3.1 loads into your GPU memory. Wait for it to complete.
  • If you see a 500 error, check your terminal for the traceback. A common cause is a typo in model_name in config.py. Make sure it reads llama3.1:8b exactly.
  • If the terminal shows a connection refused error, Ollama may have stopped. Restart it by running ollama serve in a new PowerShell window.

Help me debug my /chat endpoint error

Your API is live and responding with real LLM output. Next up, you'll add a streaming endpoint so clients can receive tokens in real time as the model generates them.

Add Streaming Support

Your /chat endpoint works, but it has a limitation. The client waits for the entire response to generate before seeing a single token.

Streaming solves this by sending tokens as they are generated. This is how ChatGPT and other AI chat interfaces deliver that real-time "typing" effect.

In this step, get ready to:
  • Create a /chat/stream endpoint that delivers tokens incrementally.
  • Test streaming and compare response latency with the non-streaming endpoint.
Create the streaming endpoint

The /chat endpoint collects the full LLM response and returns it as one JSON object. A streaming endpoint instead opens a persistent connection and pushes each token to the client the moment it is generated.

You will use FastAPI's StreamingResponse combined with the Ollama client's stream=True parameter.

  • In VS Code, switch back to main.py.
  • Add the following endpoint below the existing chat_endpoint function:
@app.post("/chat/stream")
def chat_stream_endpoint(request: ChatRequest):
    messages = [{"role": m.role, "content": m.content} for m in request.messages]

    def generate():
        # Call Ollama with streaming enabled
        stream = chat(
            model=settings.model_name,
            messages=messages,
            stream=True,
        )
        # Yield each chunk as a server-sent event
        for chunk in stream:
            data = {
                "content": chunk.message.content,
                "done": chunk.done,
            }
            yield f"data: {json.dumps(data)}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

What does this code do?

This endpoint uses a Python generator function (generate()) to produce data incrementally instead of all at once.

  • chat(..., stream=True) tells the Ollama client to return an iterator of chunks instead of one complete response.
  • Each chunk contains a partial token in chunk.message.content and a chunk.done flag indicating whether generation is complete.
  • The yield keyword sends each chunk to the client immediately, formatted as a Server-Sent Event (data: {...}\n\n).
  • StreamingResponse wraps the generator and sets media_type="text/event-stream" so the client knows to expect a stream.
  • Save main.py.
  • Your dev server auto-reloads. Check the terminal for a successful reload message with no errors.

Seeing an import error?

Make sure the import json and from fastapi.responses import StreamingResponse lines are at the top of your file. Both should already be there from the previous step.

Help me debug this import error

Test streaming and compare latency

Now you will send the same message to both endpoints and observe how they deliver the response differently.

  • Open your browser and navigate to http://127.0.0.1:8000/docs.
  • Find the POST /chat/stream endpoint in the Swagger UI.
  • Click Try it out.
  • Paste the following JSON into the request body:
{
  "messages": [
    {"role": "user", "content": "Explain what an API is in two sentences."}
  ]
}
  • Click Execute.

The response body shows multiple data: lines, each containing a JSON object with a content field (one or a few tokens) and a done flag.

What should I see?

The response contains lines like:

data: {"content": "An", "done": false}

data: {"content": " API", "done": false}

data: {"content": " is", "done": false}

...

data: {"content": "", "done": true}

Each line is a separate chunk that arrived incrementally. The final chunk has "done": true signaling that generation is complete.

Now compare the two endpoints side by side.

  • In the Swagger UI, send the same message to POST /chat (the non-streaming endpoint).
  • Notice the response time shown in the Swagger UI. The non-streaming endpoint shows one large response that took the full generation time to arrive.
  • Compare this with the streaming endpoint. The streaming version delivers the first token almost immediately, while the full response takes the same total time.

Why does streaming matter?

Total generation time is roughly the same for both endpoints. The difference is perceived latency. With streaming, the user sees content appearing within milliseconds instead of staring at a loading spinner for several seconds.

This is the same pattern production AI services use. The model generates tokens one at a time regardless. Streaming just exposes that natural flow to the client.

Response appears as a single block?

The Swagger UI buffers streaming responses, so you may see all the data: lines appear at once. This is a display limitation, not a bug in your code. The underlying HTTP connection still delivered tokens incrementally.

To observe true token-by-token delivery, you can use a tool like curl in a terminal. The important confirmation is that the response body contains multiple data: lines rather than one JSON object.

Help me troubleshoot my streaming endpoint

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py and confirmed the streaming response contains multiple data: lines.

ⓧ I'd like to double check the full code

Here is the full main.py including both the original /chat and new /chat/stream endpoints:

import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from ollama import chat

from config import settings
from models import ChatRequest, ChatResponse, Message

app = FastAPI(
    title=settings.app_name,
    version=settings.app_version,
)


@app.get("/health")
def health_check():
    try:
        from ollama import list as ollama_list

        models = ollama_list()
        return {
            "status": "healthy",
            "ollama": "connected",
            "models_available": len(models.models),
        }
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}


@app.post("/chat", response_model=ChatResponse)
def chat_endpoint(request: ChatRequest):
    messages = [{"role": m.role, "content": m.content} for m in request.messages]

    response = chat(
        model=settings.model_name,
        messages=messages,
    )

    return ChatResponse(
        message=Message(
            role=response.message.role,
            content=response.message.content,
        ),
        model=response.model,
        total_duration=response.total_duration,
    )


@app.post("/chat/stream")
def chat_stream_endpoint(request: ChatRequest):
    messages = [{"role": m.role, "content": m.content} for m in request.messages]

    def generate():
        stream = chat(
            model=settings.model_name,
            messages=messages,
            stream=True,
        )
        for chunk in stream:
            data = {
                "content": chunk.message.content,
                "done": chunk.done,
            }
            yield f"data: {json.dumps(data)}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

Your API now supports both batch and streaming responses. This is the same architecture that powers every major AI chat product. Next up, you will extend this backend with conversation memory in the Secret Mission.

Secret mission

Add Conversation Memory

Right now, your API is stateless. Every request starts from scratch with zero context about what came before. Real chat applications remember what you said three messages ago. Can you add conversation memory so your backend maintains chat history across multiple requests?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama with Llama 3.1 8B model (~4.9GB on disk).
  • Python virtual environment in the agentic-ai-backend folder.
  • Project files (main.py, models.py, config.py, requirements.txt) on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend this project with RAG, agents, or guardrails.

  • Ollama continues running in the background and serves models on http://localhost:11434.
  • The Llama 3.1 8B model stays downloaded and ready for instant inference.
  • Your virtual environment and project files remain on your Desktop.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping all files and models intact for later.

  • Stop the FastAPI dev server by pressing Ctrl+C in the terminal where it is running.
  • Quit Ollama by right-clicking the Ollama icon in the system tray and selecting Quit.
  • Deactivate your virtual environment by running:
deactivate

When you're ready to resume, reopen Ollama (it starts automatically on login, or launch it from the Start menu) and reactivate your virtual environment.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Stop the FastAPI dev server by pressing Ctrl+C in the terminal where it is running.
  • Deactivate your virtual environment by running:
deactivate
  • Remove the Llama 3.1 8B model from Ollama by running:
ollama rm llama3.1:8b
  • Delete the project folder from your Desktop by running:
cd ~/Desktop
Remove-Item -Recurse -Force agentic-ai-backend
  • Uninstall Ollama from your computer. Open Settings, select Apps, then Installed apps. Search for Ollama, click the three-dot menu, and select Uninstall. The uninstaller will offer a checkbox to also remove downloaded models.

Nice Work!

Nice Work!

You just built a production-style AI chat backend that runs entirely on your machine. Your API accepts messages, routes them to a local LLM powered by your GPU, and returns structured responses with metadata.

You've learned how to:

  • Install and run Ollama with Llama 3.1 8B on your local GPU, serving LLM inference through an HTTP API.
  • Build a FastAPI backend with /health, /chat, and /chat/stream endpoints, complete with type-safe Pydantic models and auto-generated interactive documentation at /docs.
  • Implement real-time streaming responses using Server-Sent Events, delivering tokens to clients as the model generates them.
  • Secret Mission: Add conversation memory with session management so the API maintains chat history across multiple requests.

Ready to quiz yourself?