Containerize Your RAG API with Docker

Package a FastAPI RAG app into a portable Docker container image.

Introduction

30 Second Summary

You built an app that works perfectly on your machine. But what happens when you need to share it with a teammate, deploy it to a server, or run it on a different computer? Without a way to package your entire environment, things break.

In this project, you will take the RAG API you built in Part 1 and package it into a Docker container. Your API, its dependencies, and its database will all live inside one portable unit that runs identically anywhere.

What You'll Build

A containerized version of your FastAPI RAG API that runs inside a Docker container with ChromaDB and all dependencies pre-installed.

By the end of this project, you'll have:

  • A Dockerfile that defines how your RAG API is packaged into a portable image.
  • A running Docker container serving your AI-powered API at http://localhost:8000.
  • A working container-to-host networking setup so your containerized API connects to Ollama on your host machine.
  • Secret Mission: Push your Docker image to Docker Hub and learn how to distribute containers publicly.

Are there any prerequisites?

This is Part 2 of the DevOps x AI series. You should have completed Part 1 (Build a RAG API with FastAPI) and have a working nextwork-rag-api project folder with app.py, embed.py, k8s.txt, and a db/ folder.

Set Up Your RAG API and Install Docker

Before you can containerize your RAG API, you need to confirm that it runs correctly on your local machine.

In this step, you will verify your existing project from Part 1, test that the API returns answers, and then install Docker Desktop so you are ready to build your first container image.

In this step, get ready to:
  • Verify your RAG API project files and dependencies.
  • Test the API locally to confirm it returns AI-generated answers.
  • Install Docker Desktop and confirm it works.
Verify your RAG API setup

Your project from Part 1 should already have all the files and dependencies in place. Let's confirm everything is where it needs to be before we start containerizing.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Cursor and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your nextwork-rag-api folder on your Desktop and click Open.
  • Confirm your project files exist by checking the Explorer sidebar in Cursor. You should see app.py, embed.py, k8s.txt, and the db/ folder listed.

Missing files?

If any of these files are missing, revisit Part 1 of the DevOps x AI series to create them. You need all four items before continuing.

help me find my missing project files from Part 1

  • Open a terminal in Cursor by pressing Ctrl+` (backtick key, same shortcut on macOS and Windows).
  • Activate your virtual environment by running this command:
source venv/bin/activate

On Windows?

If you are on Windows, run venv\Scripts\activate instead.

  • Verify your Python packages are installed by running:
pip show fastapi uvicorn chromadb ollama

You should see version information printed for each of the four packages.

Package not found?

If any package is missing, reinstall it by running pip install fastapi uvicorn chromadb ollama in your terminal.

help me fix missing Python packages in my virtual environment

  • Confirm Ollama is running and the tinyllama model is available by running:
ollama list

You should see tinyllama listed in the NAME column of the output.

Ollama not running or tinyllama missing?

If you get a connection error, open the Ollama application first. If tinyllama is not in the list, pull it by running ollama pull tinyllama in your terminal.

help me get Ollama running with the tinyllama model

Now let's test that the full API works end to end.

  • Start your API server by running:
uvicorn app:app --host 127.0.0.1 --port 8000

What does this command do?

This starts your FastAPI application on port 8000. The terminal will stay occupied showing server logs. You will need a second terminal to send a test request.

  • Open a second terminal by clicking the + icon in Cursor's terminal panel.
  • Test the API by running this command:
curl -X POST "http://127.0.0.1:8000/query" -G --data-urlencode "q=What is Kubernetes?"

You should see a JSON response like {"answer": "Kubernetes is a container orchestration platform..."} with an AI-generated answer about Kubernetes.

Not getting a response?

Make sure uvicorn is still running in your first terminal. Check that Ollama is running in the background. The first request may take a few seconds while tinyllama loads into memory.

help me debug why my RAG API isn't responding to curl requests

  • Stop the API server by pressing Ctrl+C in the first terminal.
Install Docker Desktop

Docker Desktop is the application that lets you build and run containers on your machine. It includes the Docker engine, the CLI tools, and a graphical dashboard.

  • Check if Docker is already installed by running this command in your terminal:
docker --version

✔️ I see a version number

Docker is already installed. Make sure Docker Desktop is running by looking for the whale icon in your menu bar (macOS) or system tray (Windows). If the icon is not there, open Docker Desktop from your Applications folder.

Skip ahead to the docker run hello-world test below.

ⓧ Command not found

Docker is not installed yet. Follow these steps to install Docker Desktop.

  • Visit the Docker Desktop download page:
  • https://docs.docker.com/desktop/setup/install/mac-install/
  • Click the download button for your chip type (Apple Silicon or Intel).
  • Open the downloaded Docker.dmg file.
  • Drag the Docker icon into your Applications folder.
  • Open Docker from your Applications folder.
  • Accept the subscription service agreement when prompted.
  • Select the Recommended settings option and enter your password if prompted.
  • Wait for the whale icon in your menu bar to become steady (not animated). This means Docker Desktop has finished starting up.

What about Windows?

If you are on Windows, visit https://docs.docker.com/desktop/setup/install/windows-install/ instead and follow the installer prompts.

Is Docker Desktop free?

Docker Desktop is free for personal use, education, and small businesses (fewer than 250 employees AND less than $10 million USD in annual revenue). No cost for this project.

  • Confirm the install worked by running:
docker --version

You should see a version string printed to the terminal.

Now let's run a quick test to make sure Docker can pull images and start containers.

  • Test that Docker works by running:
docker run hello-world

What does this command do?

This downloads a tiny test image from Docker Hub and runs it as a container. The container prints a confirmation message and exits. It proves that your Docker installation can pull images and run containers end to end.

You should see a message that starts with "Hello from Docker!" in your terminal output.

Permission denied or daemon not running?

Make sure Docker Desktop is open and the whale icon in your menu bar is steady. If you see a permission error, try restarting Docker Desktop.

help me fix Docker permission or daemon errors

Your local environment is fully verified and Docker is ready to go. Next up, you will write a Dockerfile that packages your entire RAG API into a portable container image.

Create a Dockerfile

Your environment is ready with Docker installed and verified. Now it's time to package your RAG API into a portable container that can run on any machine.

Right now, your API only runs on your machine because it depends on locally installed Python packages, your specific file structure, and your operating system. A Dockerfile solves this by defining every dependency and configuration step in a single recipe file that Docker uses to build a self-contained container image.

In this step, get ready to:
  • Write a Dockerfile that defines your container image.
  • Build the image and verify it exists locally.
  • Run the container and observe a networking issue with Ollama.
Create the Dockerfile

A Dockerfile is a plain text file with no extension. Each line is an instruction that Docker executes in order to assemble your image.

  • In Cursor, create a new file called Dockerfile in your nextwork-rag-api project folder (the same directory where app.py lives).
  • Make sure the file has no extension. It should be named exactly Dockerfile with a capital D.
  • Paste the following code into your new Dockerfile:
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY app.py embed.py k8s.txt ./
RUN pip install fastapi uvicorn chromadb ollama
RUN python embed.py
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

Each line is a layer in your image:

  • FROM python:3.11-slim starts from a lightweight Python 3.11 base image.
  • WORKDIR /app sets the working directory inside the container.
  • RUN apt-get update && apt-get install -y curl installs curl for health checks and debugging.
  • COPY app.py embed.py k8s.txt ./ copies your three application files into the container.
  • RUN pip install fastapi uvicorn chromadb ollama installs the Python packages your app needs.
  • RUN python embed.py pre-computes the embeddings so they are baked into the image.
  • EXPOSE 8000 documents which port the app listens on.
  • CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] starts the API server when the container launches.
  • Save Dockerfile.

Why 0.0.0.0 instead of 127.0.0.1?

When you tested locally, you used --host 127.0.0.1 which only listens for connections from the same machine. Inside a container, 0.0.0.0 tells uvicorn to accept connections from outside the container. Without this, Docker's port mapping would not be able to forward traffic to your app.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your Dockerfile before moving on.

⨯ I'd like to double check the full code

Here is the complete Dockerfile:

FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY app.py embed.py k8s.txt ./
RUN pip install fastapi uvicorn chromadb ollama
RUN python embed.py
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Build the Docker image

A Docker image is a frozen snapshot of your application and its environment. Building one executes every instruction in your Dockerfile and saves the result.

  • Make sure your terminal is in the nextwork-rag-api folder (the same folder where your Dockerfile lives).
  • Build your Docker image by running this command:
docker build -t rag-app .

What does this command do?

  • docker build tells Docker to read the Dockerfile and execute each instruction.
  • -t rag-app tags (names) the resulting image as rag-app so you can reference it later.
  • . tells Docker to use the current directory as the build context (where it looks for the Dockerfile and files to copy).

The first build takes 3-5 minutes because Docker downloads the python:3.11-slim base image and installs all packages. Future builds are much faster thanks to layer caching.

  • Verify your image was created by running:
docker images | grep rag-app

You should see a row showing rag-app with the tag latest and a size in the hundreds of megabytes.

Build failed with an error?

  • Check that your Dockerfile has no file extension. Some editors add .txt silently.
  • Make sure Docker Desktop is running. Look for the whale icon in your menu bar.
  • Confirm you are running the command from inside the nextwork-rag-api folder where the Dockerfile lives.

Help me debug my Docker build error

Run the container and test

Your image is ready. Now you will launch it as a running container and send it a query to see if your RAG API works inside Docker.

  • Start your container with port mapping by running this command:
docker run -p 8000:8000 rag-app

What does -p 8000:8000 do?

The -p 8000:8000 flag maps port 8000 on your host machine to port 8000 inside the container. Without this, traffic from your browser or terminal would never reach the containerized API.

You should see uvicorn startup logs in the terminal. The container is now running your API.

  • Open a new terminal tab or window (the current one is occupied by the running container).
  • Test your containerized API by running this command:
curl -X POST "http://127.0.0.1:8000/query" -G --data-urlencode "q=What is Kubernetes?"

Instead of getting a JSON answer, you will see a connection error in the container's terminal output. The API crashes because it cannot reach Ollama.

Why does this error happen?

Your app.py calls ollama.generate() which tries to connect to localhost:11434 by default. But localhost inside a container refers to the container itself, not your host machine.

Ollama is running on your Mac (the host), but the container has its own isolated network. The container's localhost is a dead end because nothing is listening on port 11434 inside the container.

  • Stop the container by pressing Ctrl+C in the terminal where it is running.

Don't see any error?

  • If you got a JSON response instead of an error, your app.py may already have a custom Ollama client configured. Check that app.py uses ollama.generate() directly (the module-level function) without specifying a host.
  • Make sure Ollama is running on your host machine. The error only appears when the container tries to reach Ollama and fails.

Help me understand the Docker networking error I am seeing

You have successfully built and run your first Docker container. The networking issue you just observed is one of the most common gotchas in containerized applications. In the next step, you will fix it with a single configuration change.

Fix Container-to-Host Networking

Your Docker image built successfully and the container started serving requests. But when you queried the API, it couldn't reach Ollama.

Inside a container, localhost refers to the container itself, not your Mac. In this step, you'll use Docker's host.docker.internal DNS name to route traffic from the container back to the host machine.

In this step, get ready to:
  • Update app.py to route Ollama requests through host.docker.internal.
  • Rebuild the Docker image and run the updated container.
  • Confirm the containerized API returns AI-generated answers.
Update app.py to use host.docker.internal

When your container tried to call Ollama at localhost:11434, it was looking for Ollama inside the container. Ollama runs on your Mac, not inside the container.

What is host.docker.internal?

Docker Desktop provides the DNS name host.docker.internal specifically for this situation. It automatically resolves to the internal IP address of the host machine, letting containers reach services running on your Mac.

Since Ollama listens on port 11434 on your host, the container can reach it at http://host.docker.internal:11434.

  • In Cursor, open app.py.
  • Find this line:
collection = chroma.get_or_create_collection("docs")
  • Add a blank line below it, then add this new line:
ollama_client = ollama.Client(host="http://host.docker.internal:11434")

What does this line do?

This creates an Ollama client that explicitly targets your host machine at port 11434. Instead of the default behavior (looking for Ollama at localhost), the client now knows to cross the container boundary using Docker's special DNS name.

  • Now scroll down to the query function and find this line:
    answer = ollama.generate(
  • Replace ollama.generate with ollama_client.generate, so the line now looks like this:
    answer = ollama_client.generate(
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

from fastapi import FastAPI
import chromadb
import ollama

app = FastAPI()
chroma = chromadb.PersistentClient(path="./db")
collection = chroma.get_or_create_collection("docs")

ollama_client = ollama.Client(host="http://host.docker.internal:11434")

@app.post("/query")
def query(q: str):
    results = collection.query(query_texts=[q], n_results=1)
    context = results["documents"][0][0] if results["documents"] else ""

    answer = ollama_client.generate(
        model="tinyllama",
        prompt=f"Context:\n{context}\n\nQuestion: {q}\n\nAnswer clearly and concisely:"
    )

    return {"answer": answer["response"]}
Rebuild and test the container

Your code change lives in app.py on your Mac, but the Docker image still has the old version baked in. You need to rebuild the image to include the fix.

  • Rebuild the Docker image by running this command:
docker build -t rag-app .

Why does rebuilding go faster this time?

Docker caches each layer (instruction) from your previous build. Since only app.py changed, Docker reuses the base image and package installation layers. Only the COPY step and everything after it runs fresh.

  • Once the build finishes, start the container by running:
docker run -p 8000:8000 rag-app

You should see uvicorn startup logs in the terminal, confirming the API is running inside the container.

  • Open a new terminal tab or window.
  • Test the API by running this command:
curl -X POST "http://127.0.0.1:8000/query" -G --data-urlencode "q=What is Kubernetes?"

You should see a JSON response like {"answer": "Kubernetes is a container orchestration platform..."} with an AI-generated answer.

Still seeing a ConnectionError?

  • Make sure Ollama is still running on your Mac. Open a separate terminal and run ollama list to verify.
  • Check that app.py uses ollama_client.generate (not ollama.generate). The module-level call still defaults to localhost.
  • Verify the host URL is exactly http://host.docker.internal:11434 with no trailing slash.

Why can't my container reach Ollama?

Stop the container
  • Switch back to the terminal where the container is running.
  • Press Ctrl+C to stop the container.

Your RAG API now runs inside a portable Docker container that can communicate with Ollama on the host. Next up, you'll take on a secret mission to distribute this container image publicly.

Secret mission

Push to Docker Hub

Your Docker image works locally. But what if a teammate needs to run it on their machine? In this secret mission, you'll push your image to Docker Hub and learn how containers are distributed across machines.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with Docker Desktop (free for personal use), so there are no ongoing costs.

Resources you used:

  • Docker Desktop (installed, free for personal use).
  • Docker images stored locally (disk space only).
  • Docker Hub repository (free for public repos, created in the Secret Mission).

Keep everything running

No action needed. Choose this if you plan to keep building with Docker or want to revisit this project later.

  • Docker images stay on your machine and take up disk space only.
  • Your Docker Hub repository stays public at no cost.
  • Docker Desktop remains installed and ready to use.

Pause - I'll come back to this later

Shut down Docker Desktop to free up system memory. Your images and configuration stay intact for when you return.

  • Right-click the whale icon in your menu bar and select Quit Docker Desktop.
  • This frees up RAM but does not delete any images or containers.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

Remove your local Docker images by running:

docker rmi rag-app
docker image prune -a

What do these commands do?

  • docker rmi rag-app removes the rag-app image specifically.
  • docker image prune -a removes all unused images from your machine, freeing disk space.

Remove your Docker Hub repository:

  • Visit https://hub.docker.com and sign in.
  • Click on your rag-app repository.
  • Go to Settings, then scroll down and select Delete repository.
  • Confirm the deletion when prompted.

Uninstall Docker Desktop:

  • Open Docker Desktop.
  • Click the Troubleshoot icon in the top-right corner of the Dashboard.
  • Select Uninstall, then confirm by selecting Uninstall again.
  • Move Docker.app to the Trash and empty the Trash.

Optionally remove residual Docker files by running:

rm -rf ~/Library/Group\ Containers/group.com.docker
rm -rf ~/.docker

Nice Work!

Nice Work!

Nice work! You just containerized a RAG API with Docker, proving you can package an AI application into a portable, reproducible unit that runs identically on any machine.

You've learned how to:

  • Write a Dockerfile and build a Docker image that packages a Python RAG API with all its dependencies.
  • Run your API inside an isolated Docker container with port mapping to serve requests.
  • Debug and fix container-to-host networking using host.docker.internal so your containerized app connects to Ollama on the host machine.
  • (Secret Mission) Push your image to Docker Hub for public distribution, letting anyone pull and run your containerized API.

Ready to quiz yourself?