Automate Testing with GitHub Actions

Introduction

This project has been archived

We're no longer updating this project - we recommend checking out our active projects instead!

If you've already completed this project, don't worry! Your documentation is still available for you to check out and share.

If you have any questions, leave a question in our Discord and we'll get back to you ASAP!

⚡️ 30 second Summary

Welcome to Project #4 (last one!) of our DevOps × AI series 👏

Picture this: you've just learned how to containerize your app with Docker (Project #2) and deploy it to Kubernetes (Project #3), so it runs the same everywhere. Nice!

But what if you forget to run your tests before deploying? What if bugs slip into production because you only manually tested the most obvious scenarios?

Once you've mastered containerization and orchestration, automating testing and deployment is your next challenge - and why CI/CD (Continuous Integration and Continuous Deployment) exists.

CI/CD automatically runs your tests and validates code quality every time you push code - all without you touching a single command! Tools like GitHub Actions (GitHub's CI/CD platform) are used by millions of developers worldwide because they make building reliable software so much easier.

What you'll build

You're about to build a CI/CD pipeline with GitHub Actions that automatically tests your RAG application every time you update it.

I haven't done Projects #1-3 yet

No worries! You only need Project #1: Build a RAG API with FastAPI for this project - Project #2: Containerize with Docker and Project #3: Deploy to Kubernetes aren't required.

You can either complete Project #1 first (which we highly recommend!), or we'll guide you through a quick setup at the start of this project to catch up. Either way works.

By the end, you'll have:

• 🤖 A GitHub Actions workflow that automatically runs your tests every time you push code - no need to remember to test manually!
• 🧪 Automated quality checks that verify your RAG API's answers are accurate and complete - catching issues like missing information or incorrect responses

I don't have experience with CI/CD or GitHub Actions!

Perfect - that's exactly who this project is for. We'll explain every concept as we go. No prior experience needed, and this project is completely free!

If you get stuck, just ask - we're here to help.

Want a complete demo of how to do this project, from start to finish? Check out our 🎬 walkthrough 🎬

If you're up for a bit of a challenge, quiz yourself on the key concepts up ahead in this project.

This project is part of a series:

1. Part 1: Build a RAG API with FastAPI
2. Part 2: Containerize a RAG API with Docker
3. Part 3: Deploy to Kubernetes
4. Part 4: You are here!

Set Up Your RAG API

Before we build our CI/CD pipeline, let's make sure you have a working RAG API ready! We need a working API first because our CI/CD pipeline will automatically test and validate this API - so it needs to exist before we can automate testing for it.

If you've done Project #1: Build a RAG API with FastAPI of this series, this will be super quick. If not, no worries - we'll set up each component as we go.

What did we do in Project #1 again?

In Project #1: Build a RAG API with FastAPI, you learned how to create a RAG (Retrieval-Augmented Generation) API. This means you built an API that can use AI to understand your documents and answer questions about them. You wrote code to load your knowledge base into Chroma, then created a FastAPI app that uses Ollama to give answers using your own data.

💡 What is a RAG API?

An API (Application Programming Interface) is a way for different programs to communicate with each other and answer questions.

A RAG (Retrieval-Augmented Generation) API is an AI-powered API that answers questions by:

• Retrieving relevant information from your knowledge base (documents)
• Augmenting a prompt with that retrieved information
• Generating an answer using an LLM (Large Language Model)

This combines the accuracy of search with the natural language capabilities of AI models!

In this step, you're going to:

• Set up your RAG API's code, database and dependencies
• Test your RAG API locally

Open Cursor

Let's open Cursor on your computer:

🍎 macOS

• Press Cmd+Space to open Spotlight.
• Type Cursor and press Enter.

🖼️ Windows

• Press the Win key.
• Type Cursor and press Enter.

I don't have Cursor!

We'll use Cursor for this step-by-step guide, but you can pick another IDE like VS Code if you prefer.

If you don't have another IDE to use, check out Cursor's download page if you'd like to have a go at doing this on your own, or try Step #1 of our Build a Mini Spotify with Cursor project for step-by-step guidance!

Cursor will open on your screen.

Have you completed Project #1 (Build a RAG API with FastAPI) in this series?

✅ Yes, I completed Project #1

Perfect! Welcome back to the DevOps × AI series!

Let's verify you have your RAG API project ready.

• In Cursor, select Open project from the welcome screen.

• Navigate to your RAG API project folder (likely ~/Desktop/nextwork-rag-api or similar).

• Select Open.

Your project should open in Cursor, showing your app's files.

Verify Your Files

Let's make sure you have all the necessary files from Project #1:

• In Cursor's file explorer (left sidebar), check that you have:
  • app.py - Your FastAPI application
  • embed.py - Script to create embeddings
  • k8s.txt - Your knowledge base document
  • venv/ - Your Python virtual environment
  • db/ - Your Chroma database folder

Why do we need these files?

• app.py contains your RAG API code with the /query endpoint.
• embed.py creates embeddings from your text documents.
• k8s.txt is your knowledge base (we'll expand this to multiple docs!)
• venv/ isolates your Python packages
• db/ stores your embedded documents for retrieval

✅ I have all the files

Excellent! Your RAG API project is ready. Let's test it to make sure it works.

❌ I'm missing some files

No worries - switch tabs over to ❌ No, I didn't complete project #1 to set up your files!

See you there 👋

❌ No, I didn't complete Project #1

No worries! You can still complete this project. We'll set up the essential files you need for your RAG API.

⏰ This setup will take 25-30 minutes

Tip

We strongly recommend doing Project #1: Build a RAG API with FastAPI first! It teaches critical RAG and API concepts that help you understand what we're automating with GitHub Actions.

But if you want to dive into CI/CD now, we've got you covered! ⬇️

In this setup, you'll:

• Create a RAG API project with all essential files
• Set up your Python environment
• Test your API to make sure it works

Create Project Folder

• In Cursor, select Open project from the welcome screen.

• Navigate to your Desktop.

• Create a new folder called nextwork-rag-api.

• Select Open.

Why create a project folder?

A project folder keeps all your code files organized in one place. This makes it easier to manage your API code and later connect it to GitHub for CI/CD automation.

Check Python Version

Before we set up your virtual environment, let's make sure you have Python installed and check which version you're using.

Why do we need Python?

Python is the programming language we'll use to write our API code. FastAPI, Chroma, and all the other libraries we'll use are Python packages. Think of Python as the foundation - without it, we can't run any of our API code.

• In Cursor, select Terminal → New Terminal.

• Run this command to check Python's version number:

🍎 macOS

python3 --version

🖼️ Windows

python --version

Check what you see:

❌ I see 'command not found' or 'python' is not recognized

No worries! Python isn't installed yet. Let's install it:

• Open your web browser and visit the Python 3.13 release page: https://www.python.org/downloads/release/python-31311/

Why Python 3.13?

We're using Python 3.13 for this project because it has full compatibility with all the packages we need for our RAG API to work. Some packages haven't been updated to support newer versions like 3.14 yet, which would cause compatibility errors as we build and test our RAG API!

• In the Files section, download the installer for your operating system:

🍎 macOS

• Select Download macOS installer.

The installer file will download to your Downloads folder.

• Open Finder (the blue and white face icon in your Dock).
• Click Downloads in the left sidebar (or press Cmd + Option + L to open Downloads).
• Look for the downloaded file named python-3.13.11-macos11.pkg (or similar with a version number).

• Double-click the .pkg file to open the installer.

What is a .pkg file?

A .pkg file is a macOS installer package. When you double-click it, macOS opens the installation wizard that guides you through installing Python on your computer.

• The Python installer window will appear. Click Continue.

• Click Continue again on the Install Python screen.

• Read the license agreement and click Continue.

• Click Agree to accept the license terms.
• The Python installer window will appear. Click Continue.

The installer will show you where Python will be installed (usually /Applications).

• Click Install.

• Click Close to finish when installation completes.

🖼️ Windows

• Look for the Windows section and click Download Python install manager, which uses Windows installer (64-bit). This is the recommended version for most Windows computers.

What's the difference between 64-bit and 32-bit?

Think of 64-bit and 32-bit like different sizes of containers for processing information:

• 64-bit systems can handle larger amounts of data at once - like having a bigger container that can hold more items. Most modern computers (from around 2005 onwards) use 64-bit processors because they're faster and can use more memory (RAM).
• 32-bit systems use smaller containers and can only handle limited amounts of data. These were common on older computers but are rarely used today.

💡 Which one should I choose?

Almost everyone should choose 64-bit - it's what 99% of modern Windows computers use. Only choose 32-bit if you have a very old computer (from before 2005) that specifically requires it. If you're unsure, 64-bit is almost always the right choice!

• Run the downloaded installer.
• Follow the installation wizard, making sure to check Add Python to PATH during installation.
• Click Install Now and wait for installation to complete.

Verify your installation

• Head back into your terminal in Cursor.
• Verify Python by running the version command again:

🍎 macOS

python3 --version

🖼️ Windows

python --version

You should now see a Python version number like Python 3.13.11.

✅ I see a version number

Python is installed! Now let's check if you have the right version:

✅ I see Python 3.13.x

Perfect! You have the correct Python version installed. You're all set to continue with the next steps!

⚠️ I see Python 3.14.x

If you see Python 3.14.x instead of Python 3.13.x, don't worry! Python 3.14 is very new and some packages that we'll rely on haven't been updated to support it yet, which will cause compatibility errors later on. But we can fix this!

First, let's check if Python 3.13 is already installed on your computer:

• Run this command to check for Python 3.13:

🍎 macOS

python3.13 --version

🖼️ Windows

py -3.13 --version

Check what you see:

✅ I see Python 3.13.x

Great! Python 3.13 is already installed on your computer. You don't need to install it again!

When you create your virtual environment later in this step, we'll show you how to specify Python 3.13 explicitly so it uses the correct version instead of Python 3.14. For now, you're all set - just continue with the next steps!

❌ I see 'command not found' or 'not recognized'

Python 3.13 isn't installed yet. Let's install it now:

1. Go to the Python 3.13 release page and download the installer for your operating system.
2. Run the installer. If you're running this project on Windows, make sure to check "Add Python to PATH" during installation (this makes sure Python 3.13 is accessible from your terminal).
3. Close and reopen your terminal in Cursor, then run the version check command again to verify you now have Python 3.13.x installed.

Once Python 3.13 is installed, you can continue with the next steps!

Set Up Virtual Environment

What is a virtual environment?

A virtual environment isolates your project's Python packages from other projects on your computer. This prevents conflicts between different versions of the same package and keeps your project dependencies organized.

• Create a virtual environment:

🍎 macOS / Linux

python3 -m venv venv

🖼️ Windows

python -m venv venv

• Activate the virtual environment:

🍎 macOS / Linux

source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt.

🖼️ Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your PowerShell prompt.

Install Dependencies

What packages are we installing?

This command installs all the packages your RAG API needs to run:

• fastapi - Framework for building your API
• uvicorn - Web server to run your API
• chromadb - Database for storing embeddings
• ollama - Python client to interact with Ollama (the AI model)

• Run this command in your terminal:

pip install fastapi uvicorn chromadb ollama

Check the required packages are installed:

• Run this command:

pip list | grep -E "fastapi|uvicorn|chromadb|ollama"

You should see all four packages listed in the terminal response.

Create k8s.txt

What is k8s.txt?

This is your knowledge base - a text file containing information your API will use to answer questions. When someone asks "What is Kubernetes?", your API will search this file for relevant information to generate an answer.

💡 Why "k8s"?

"k8s" is short for Kubernetes! The "8" represents the 8 letters between "K" and "s" in "Kubernetes". It's a common abbreviation developers use.

• In Cursor, select the New File icon in the file explorer.

• Save the file as k8s.txt in your project root.

• Paste this content:

Kubernetes is a container orchestration platform used to manage containers at scale.

• Press Cmd + S (Mac) or Ctrl + S (Windows) to save your file.

Create embed.py

What does embed.py do?

This script converts your text documents (like k8s.txt) into embeddings - numerical representations that help the API quickly find relevant information when answering questions.

💡 What are embeddings?

Think of embeddings like a smart index in a library. Instead of searching word-by-word through every book, embeddings are numbers that help the API understand the meaning behind text, making searches much faster and more accurate.

💡 What is ChromaDB?

ChromaDB is a database that stores your embeddings. When your API receives a question, it searches ChromaDB to find the most relevant information from your knowledge base.

• Create a new file called embed.py.

• Paste this content:

import chromadb

client = chromadb.PersistentClient(path="./db")
collection = client.get_or_create_collection("docs")

# Clear existing documents
existing_ids = collection.get()["ids"]
if existing_ids:
    collection.delete(ids=existing_ids)

with open("k8s.txt", "r") as f:
    text = f.read()

collection.add(documents=[text], ids=["k8s"])

print("Embedding stored in Chroma")

• Save the file.

Create app.py

What does app.py do?

This is your RAG API - it combines retrieval (searching your knowledge base) with generation (creating answers using AI). When someone sends a question, your API:

1. Searches your embedded documents for relevant context
2. Sends that context + the question to an AI model (tinyllama via Ollama)
3. Returns the AI's natural language answer

💡 What is FastAPI?

You might notice that FastAPI is the first thing we import at the top of the code! FastAPI is a Python framework for building APIs. It makes it easy to create endpoints (like /query) that can receive requests and return responses. This is what we'll be automating with GitHub Actions!

• Create a new file called app.py.

• Paste this content:

from fastapi import FastAPI
import chromadb
import ollama
import os

app = FastAPI()
chroma = chromadb.PersistentClient(path="./db")
collection = chroma.get_or_create_collection("docs")

# Use environment variable for Ollama host, default to localhost for local development
ollama_host = os.getenv("OLLAMA_HOST", "http://localhost:11434")
ollama_client = ollama.Client(host=ollama_host)

@app.post("/query")
def query(q: str):
    results = collection.query(query_texts=[q], n_results=1)
    context = results["documents"][0][0] if results["documents"] else ""

    answer = ollama_client.generate(
        model="tinyllama",
        prompt=f"Context:\n{context}\n\nQuestion: {q}\n\nAnswer clearly and concisely:"
    )

    return {"answer": answer["response"]}

• Save the file.

Create Embeddings

Right now, you have your knowledge base file (k8s.txt) with raw text, and you've created the script (embed.py) that will convert it into embeddings. But you haven't run the script yet, so your API doesn't have any embeddings to search through.

Let's run the embedding script to convert your text into embeddings and store them in ChromaDB.

• Paste this command into your terminal and press Enter:

python embed.py

You should see:

Embedding stored in Chroma

What just happened?

The script read your k8s.txt file, converted it into embeddings, and stored them in a ChromaDB database. Now your API can quickly search through this information when answering questions!

Verify Your Database

Let's check that your database was created correctly.

• In your project directory, check that the db/ folder exists.
• Expand the db/ folder.

You should see Chroma database files inside.

Install Ollama

Your RAG API needs Ollama running to generate answers.

What is Ollama?

Ollama is a tool that lets you run large language models locally on your own computer. Instead of sending requests to cloud services like OpenAI or Anthropic, Ollama runs the AI model directly on your machine.

This gives you complete privacy, no API costs, and the ability to work offline. In this project, you'll use Ollama to run the "tinyllama" model, which will power the AI responses in your RAG system.

First, let's check if Ollama is already installed:

• In your terminal, run this command:

ollama --version

• Check the response in your terminal:

❌ I see 'command not found'

Let's install Ollama for your operating system:

🍎 macOS

Check if Homebrew is installed:

• Run this command:

brew --version

You should see a version number like Homebrew 4.2.0. This means Homebrew is installed and ready to use!

I see "command not found" instead!

No worries! Install Homebrew first by visiting https://brew.sh/ and following the installation instructions. Once Homebrew is installed, come back and continue with the steps below.

Install Ollama using Homebrew:

• Run this command:

brew install --cask ollama

What does --cask mean?

Homebrew has two types of packages:

• Regular packages (like python or git) - These install command-line tools and libraries that run in Terminal.
• Casks (like ollama with --cask) - These install GUI applications (macOS apps) that you'd normally download from a website and drag into your Applications folder.

The --cask flag tells Homebrew "this is a macOS app, not just a command-line tool." When you use --cask, Homebrew downloads the app bundle (like a .dmg file) and automatically installs it into your Applications folder for you - no manual dragging required!

• Wait for the installation to complete.

You should see a "ollama-app was successfully installed!" message!

Verify Ollama installation

• Verify Ollama is installed correctly by running this command in your terminal:

ollama --version

• You should see output like:

What does 'could not connect to a running Ollama instance' mean?

This warning is totally normal at this point! Here's what's happening:

• You've successfully installed the Ollama command-line tools and the Ollama macOS app via Homebrew (that's why you see the version number 0.13.5). The app includes the Ollama service (the background process that actually runs the AI models and listens for requests).
• However, even though the app is installed, the Ollama service isn't running yet - it needs to be started by launching the Ollama app.
• The ollama --version command is trying to connect to this service, but since the service isn't running yet, you see this warning.

To get the Ollama background service running, you need to launch the Ollama app from your Applications folder. Let's do that now!

Launch the Ollama app

Since brew install --cask ollama already installed the Ollama app into your Applications folder, we just need to launch it to start the background service.

• Open Finder.
• Click Applications in the left sidebar.
• Look for Ollama in the list of applications.

You should see the Ollama app icon in your Applications folder.

• Double-click Ollama to launch it.

You should see the Ollama icon appear in your menu bar (top right corner of your screen). This means the Ollama background service is now running!

• You can close the Ollama panel that appears - it's an extra user interface Ollama provides for interacting with your local AI models (but we won't need it for this project!).

🖼️ Windows

Download and install Ollama

• Open your web browser.
• Navigate to the Ollama download page:

https://ollama.com/download/windows

• Click the Download for Windows button.

• Wait for the download to complete.

• Open your Downloads folder.
• Find the downloaded Ollama installer file (it will have a name like OllamaSetup.exe).
• Double-click OllamaSetup.exe to run the installer.
• Follow the installation wizard, accepting the default settings.
• When installation finishes, click Finish.

What just installed?

The Ollama installer set up both the command-line tools (the ollama command) and the Ollama service on your Windows computer. Unlike macOS, on Windows you need to manually start the Ollama service using a command in PowerShell.

Start the Ollama service

• Open PowerShell (right-click Start menu → Windows PowerShell or search for "PowerShell" in the Start menu).
• Run this command to start the Ollama service:

ollama serve

What does this command do?

ollama serve starts the Ollama server on your computer. It listens on port 11434 for requests from your API. Unlike macOS, on Windows you need to keep the terminal window open for Ollama to keep running.

You should see:

Listening on 127.0.0.1:11434

What does this mean?

This confirms that the Ollama background service is now running and ready to accept requests! The service is listening on 127.0.0.1:11434, which means it's accessible from your local computer.

Keep this window open - Ollama must stay running for your API to work!

It says Ollama app not found!

No worries - let's get it installed:

• Open a browser and go to https://ollama.com/download/windows.
• Select Download for Windows and run the installer (OllamaSetup.exe).
• Accept the defaults so Ollama sets up its Windows service and adds itself to your PATH.
• Re-open PowerShell and run ollama serve again. You should now see Listening on 127.0.0.1:11434.

If you still get "not recognized," restart PowerShell so the updated PATH loads, then retry ollama serve.

✅ I see a version number

Great! Let's verify Ollama is working properly.

⚠️ I see 'could not connect to a running Ollama instance'

Don't worry – this error just means your command is trying to talk to Ollama, but the Ollama service isn't running yet. Let's start it:

🍎 macOS

• Run this command:

open -a "Ollama"

What does this command do?

open -a "Ollama" tells macOS to launch the Ollama application. The -a flag specifies the application name, and macOS will find it in your Applications folder.

You should see the Ollama icon appear in your menu bar.

🖼️ Windows

• Open PowerShell and run:

ollama serve

What does this command do?

ollama serve starts the Ollama server on your computer. It listens on port 11434 for requests from your API. Unlike macOS, on Windows you need to keep the terminal window open for Ollama to keep running.

You should see:

Listening on 127.0.0.1:11434

It says Ollama app not found!

No worries - let's get it installed:

• Open a browser and go to https://ollama.com/download/windows.
• Select Download for Windows and run the installer (OllamaSetup.exe).
• Accept the defaults so Ollama sets up its Windows service and adds itself to your PATH.
• Re-open PowerShell and run ollama serve again. You should now see Listening on 127.0.0.1:11434.

If you still get "not recognized," restart PowerShell so the updated PATH loads, then retry ollama serve.

Keep this window open - Ollama must stay running!

Still having trouble installing or starting Ollama on Windows?

Try troubleshooting your error here, or share your error message with the NextWork community!

• Wait 5-10 seconds for Ollama to start, then rerun the command that showed the error (for example, curl http://localhost:11434 or an ollama pull / ollama run command). You should now see Ollama is running instead of the error message.

What does this error mean?

It simply means there's no background Ollama process listening on port 11434 yet, so your request can't reach it. Once the Ollama service starts, the connection will succeed and you'll see Ollama is running instead.

Verify Ollama is running

To be sure, we can also double check the Ollama service is actually running.

What does this command do?

When you run this command, you are sending a quick message to the Ollama service on your computer (localhost means "this computer"). The number 11434 is the [port] that Ollama uses to receive messages when it starts up.

If Ollama is running, it will respond to your message. If it's not running, you'll see a connection error. It's a fast way to check if the Ollama background service is actually working!

💡 How is this different from checking Ollama's version?

• ollama --version only proves the command-line tools are installed. It does not prove the Ollama background service is running.
• curl http://localhost:11434 tests the service itself. If the service is running, you'll get a response. If it's not running, you'll see a connection error.

• Run the following command in your terminal:

curl http://localhost:11434

You should see:

Ollama is running

✅ Ollama is running

Nice! That tells us Ollama is all ready to go.

❌ Ollama is not running

Ollama is installed but not running. Let's start it:

🍎 macOS

open -a "Ollama"

You should see the Ollama icon appear in your menu bar.

🖼️ Windows

Open PowerShell and run:

ollama serve

What does this command do?

ollama serve starts the Ollama server on your computer. It listens on port 11434 for requests from your API. Unlike macOS, on Windows you need to keep the terminal window open for Ollama to keep running.

You should see:

Listening on 127.0.0.1:11434

Keep this window open - Ollama must stay running!

It says Ollama app not found!

No worries - let's get it installed:

• Open a browser and go to https://ollama.com/download/windows.
• Select Download for Windows and run the installer (OllamaSetup.exe).
• Accept the defaults so Ollama sets up its Windows service and adds itself to your PATH.
• Re-open PowerShell and run ollama serve again. You should now see Listening on 127.0.0.1:11434.

If you still get "not recognized," restart PowerShell so the updated PATH loads, then retry ollama serve.

Test that Ollama is responding:

curl http://localhost:11434

You should see Ollama is running in your terminal - sweet!

Pull tinyllama model

Great! With Ollama installed and running, you're ready to start running AI models locally! We'll use tinyllama for this project - it's a smaller, faster model perfect for learning and local development.

What is tinyllama?

Tinyllama is an AI model - a file (about 600MB) that contains everything the AI needs to understand and generate text. When you run Ollama commands, it uses this model to answer your questions.

First, let's check if you already have the tinyllama model:

• Run this command:

ollama list

This command shows all models currently downloaded on your machine.

Check the results - do you already have tinyllama in your results list?

❌ No - I don't have tinyllama

No worries! Let's get to downloading the tinyllama model (approximately 600MB):

• Run this command:

ollama pull tinyllama

The download may take 2-5 minutes depending on your connection:

pulling manifest
pulling 2af3b81862c6... 45%

While we wait... here's a quick recap and what's ahead!

So far, you've:

• Installed Python and set up your environment.
• Installed and started Ollama so it can run AI models locally.
• Verified that Ollama is installed and its background service is running.

Right now, you're downloading the tinyllama model, which will be the "brain" behind your RAG API's responses. Once the download finishes, you'll:

• Test tinyllama in your terminal as a chat interface.
• Then, in the next steps, you'll build a RAG API using Python and FastAPI.
• You'll connect that API to tinyllama via Ollama so it can generate answers.
• Finally, you'll wire in your own documents so tinyllama can answer questions using your data instead of just its pre-training.

Wait for "success" to appear at the end of your terminal.

My computer's taking too long to install tinyllama (over 10 minutes)!

That's okay! This is likely to happen if your local computer has limited bandwidth or disk speed, but we've got options. You can cancel (Ctrl + C) and try a smaller model instead:

• SmolLM 360M (~229MB) – fastest small-chat option with better coherence than the tiniest models. Install it using ollama pull smollm:360m
• SmolLM 135M (~92MB) – smallest footprint; great when you just need a very lightweight model. Install it using ollama pull smollm:135m

We use tinyllama here because it's still lightweight but more capable for chat than the smaller SmolLM variants. If installing tinyllama is slowing you down, feel free to use SmolLM instead.

✅ Yes - I already have tinyllama

Nice! If you already have tinyllama, we can skip installing it.

You should see tinyllama in the output:

This means the tinyllama model is ready to use! Awesome, your API will use it to generate AI responses.

Test Your RAG API

Let's verify your API works!

What does uvicorn do?

Uvicorn is a web server that runs your FastAPI application. The command uvicorn app:app tells uvicorn to:

• Look for a file called app.py (the first "app")
• Find the FastAPI instance named app inside that file (the second "app")
• Start a web server that runs your API

The --reload flag automatically restarts the server when you change code - perfect for development!

• Make sure your virtual environment is activated (you should see (venv) in your terminal prompt).
• Start your RAG API.
• Paste this command into your terminal and press Enter:

uvicorn app:app --reload

✅ Success

You should see output like:

INFO:     Uvicorn running on http://127.0.0.1:8000
INFO:     Application startup complete.

🙋‍♀️ I see 'command not found' or 'uvicorn' is not recognized

This usually means your virtual environment isn't activated! Since you opened a new terminal in Cursor, it doesn't automatically have your virtual environment activated - each new terminal session starts fresh.

Python needs access to the packages we installed earlier (like uvicorn), but those packages are only available inside your virtual environment. When you try to run uvicorn app:app --reload without activating the environment, Python can't find the installed packages.

Let's fix it:

• Make sure you see (venv) at the beginning of your terminal prompt. If you don't see it, activate your virtual environment:
  • macOS/Linux: source venv/bin/activate
  • Windows: venv\Scripts\activate
• Once you see (venv) in your prompt, try running uvicorn app:app --reload again.

🙋‍♀️ Still having trouble?

If you're still seeing errors, make sure you've installed uvicorn in your virtual environment. Try running pip list | grep uvicorn (macOS/Linux) or pip list | Select-String uvicorn (Windows) to verify it's installed.

Let's test your API in a new terminal tab (keep the server running in the first one).

• Select the New Terminal button at the top of your terminal window.

A new terminal will open up beneath your existing terminal.

• Activate your virtual environment in the new tab.
• Run these commands:

🍎 macOS / Linux

cd ~/Desktop/nextwork-rag-api
source venv/bin/activate

🖼️ Windows

cd ~\Desktop\nextwork-rag-api
venv\Scripts\activate

Having trouble with virtual environments?

Get help reactivating your virtual environment here.

Let's test your API.

• Run this command:

🍎 macOS / Linux

curl -X POST "http://127.0.0.1:8000/query" -G --data-urlencode "q=What is Kubernetes?"

🖼️ Windows

Invoke-WebRequest -Uri "http://127.0.0.1:8000/query?q=What%20is%20Kubernetes%3F" -Method POST

You should see a JSON response with an answer about Kubernetes!

{"answer":"Kubernetes is a container orchestration platform that manages containers at scale..."}

What just happened?

Your RAG API:

1. Received the question "What is Kubernetes?"
2. Searched your embedded documents (k8s.txt) for relevant context
3. Sent that context + question to the tinyllama LLM via Ollama
4. Returned the LLM's natural language answer

This is the core workflow we'll be automating with CI/CD!

Perfect! Your RAG API is working. Now let's automate testing and deployment with GitHub Actions!

Push Your Code to GitHub

Now that your RAG API works locally, let's get it onto GitHub so we can set up automated testing!

What is GitHub?

GitHub is a platform for hosting and collaborating on code using Git version control. Think of it like Google Docs for code - it tracks every change, lets you collaborate with others, and is where most developers store their projects.

We're using GitHub in this project because GitHub Actions - the CI/CD tool we'll set up - is built directly into GitHub, making it easy to automate testing and deployment when we push code.

💡 What is Git?

Git is version control software that tracks changes to your code over time. It lets you save different versions of your project, revert to earlier versions, and work on features without breaking your main code.

In this step, you're going to:

• Set up Git in your project
• Create a GitHub repository
• Push your code to GitHub

Initialize a Git Repository

Let's turn your project folder into a Git repository!

First, we'll make sure you're in your project directory.

• Run this command and press Enter:

pwd

You should see a path ending in nextwork-rag-api, like:

What does pwd mean?

pwd stands for "print working directory" - it shows you which folder your terminal is currently in.

Initialize a Git repository.

• Run this command and press Enter:

git init

Getting a 'Git not found' error?

That's totally okay! This just means you don't have Git installed on your computer.

• Head over to git-scm.com/downloads.
• Download and run the installer for your operating system.
• Once installed, restart your IDE and try again.

Still having issues? Troubleshoot here or share your error with the NextWork community!

You should see:

What just happened?

Git created a hidden .git folder in your project that will track all your code changes. This folder contains the entire history of your project!

💡 What are branches?

Think of a branch like a separate timeline for your code. When you create a branch, you're creating an independent line of development that you can experiment with without affecting your main code.

When you run git init, Git automatically creates your first branch (usually called master or main). This is your main timeline where your stable, working code lives. Later, you can create additional branches to work on new features or experiment with changes!

💡 Extra for Experts: What's that hint about 'master'?

When you run git init, Git creates your first branch and names it master by default. You might see a hint message suggesting you could use a different name like main, trunk, or development instead.

This is just Git letting you know about the default branch name - it's not an error! Many developers and companies have moved away from using master as the default branch name to use more inclusive terminology.

Later in this project, we'll show you how to rename the branch to main

Create .gitignore File

Before we commit our code, let's tell Git which files to ignore (like virtual environments and databases).

Let's create a .gitignore file in your project root directory. This file tells Git which files and folders to ignore when tracking changes.

What is a .gitignore file?

A .gitignore file is a special file that tells Git which files and folders to skip when tracking changes. By setting up .gitignore first, we ensure that these unwanted files (like your virtual environment or sensitive information like API keys) are never tracked by Git in the first place.

💡 Why don't we track all changes?

Not all files in your project need to be tracked by Git! Best practice is that your Git repository should only contain source code, configuration files, and documentation - the essential files that other developers need to understand and run your project.

Files that shouldn't be tracked include:

• Generated files (like compiled code, build artifacts, or virtual environments) - these can be recreated anytime
• Sensitive information (like API keys, passwords, or personal tokens) - these should never be shared
• Local settings (like IDE configurations or personal preferences) - these are specific to your machine
• Large files (like databases or media files) - these bloat your repository and slow down Git operations

• Select New File from the file explorer menu.

• Name the file .gitignore (make sure it starts with a dot - this is important!).
• Press Enter to confirm the filename.

Why does the filename start with a dot?

Files that start with a dot (like .gitignore) are called "hidden files" on Unix-based systems (macOS and Linux). The dot tells the system to hide these files by default in file explorers, but they're still fully functional. Git recognizes .gitignore as a special configuration file.

• Paste this content:

# Python
venv/
__pycache__/
*.pyc
*.pyo
*.egg-info/
.Python

# Chroma database
db/

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# OS
.DS_Store
Thumbs.db

• Save the file.

Why do we ignore these files?

The two most important entries in this .gitignore file are:

• venv/ - Virtual environments are large (hundreds of MB) and machine-specific. Each developer should create their own virtual environment, so we don't need to commit it to the repository.
• db/ - The Chroma database folder contains your embeddings and can be very large (hundreds of MB or even GB depending on your data). This will be regenerated in CI/CD when the workflow runs the embedding script, so we don't need to commit it. Each environment should build its own database.

Keeping these out makes your repository cleaner, faster to clone, and prevents conflicts between different development environments!

💡 Extra for Experts: What about the other entries?

The .gitignore file also includes other patterns to keep your repository clean:

Other Python files:

• __pycache__/ - Python automatically creates these cache folders to speed up imports. They're regenerated automatically and don't need to be versioned.
• *.pyc, *.pyo - Compiled Python bytecode files. These are automatically generated from your .py source files and can be recreated anytime.
• *.egg-info/ - Package metadata folders created when installing packages. They're machine-specific and regenerated during installation.
• .Python - A marker file created by some virtual environment tools. Not needed in version control.

IDE files:

• .vscode/, .idea/ - These are IDE-specific settings and configurations. Different developers use different IDEs, so these shouldn't be shared.
• *.swp, *.swo, *~ - Temporary files created by editors like Vim. These are swap files and backup files that shouldn't be committed.

OS files:

• .DS_Store - macOS automatically creates these files to store folder view settings. They're not needed and clutter the repository.
• Thumbs.db - Windows creates these files to cache thumbnail images. Like .DS_Store, they're OS-specific and unnecessary.

Add Files to Git

Let's stage all your project files for the first commit.

Stage all files.

• Run this command and press Enter:

git add .

What does git add do?

git add . stages all files in your current directory for the next commit. Staging means preparing files to be saved in Git's history. Once it's saved in Git's history, we say the change has been committed.

💡 Extra for Experts: Why does Git use staging? Why can't we save files directly?

Git's staging area (also called the "index") gives you control over what goes into each commit:

• You can check exactly what will be committed using git status and git diff, catching mistakes before they're saved in history
• You can stage only specific files, even if you've modified many. For example, you might want to commit app.py separately from k8s.txt if they represent different changes
• You can stage only some changes from a file (using git add -p), allowing you to split large changes into logical, smaller commits
• Staging forces you to pause and review, preventing accidental commits of temporary files, debug code, or incomplete work

Without staging, every file change would be committed immediately, making it harder to create a clean, organized commit history!

Check what's staged.

• Run this command and press Enter:

git status

You should see your files listed under "Changes to be committed":

I see more files in the list!

That's okay! If you're continuing with the same repository from Project #2: Containerize with Docker or Project #3: Deploy to Kubernetes of this series, you'll see more files in your "Changes to be committed" list, like:

You're just adding the new files from this project to your existing repository.

Notice that venv/ and db/ are not listed - that's because .gitignore is working!

Commit Your Code

Now let's save a snapshot of your project.

Create your first commit.

What is a commit?

A commit is like a save point in a video game - it's a snapshot of all the staged changes at a specific moment.

Since we used git add . to stage files (and .gitignore excluded others), this commit will only include the files we staged - not everything in your project folder!

• Run this command and press Enter:

git commit -m "Initial commit: set up local RAG API project"

What does this command do?

git commit saves all your staged changes to Git's history. The -m flag lets you add a commit message directly in the command (instead of opening an editor). The message in quotes describes what this commit contains - think of it as a label for this save point!

You should see output showing files changed and insertions:

What does this output mean?

Let's break down what Git is telling you:

• [main (root-commit) abc1234] - This shows the branch name (main), that it's your first commit (root-commit), and a unique commit ID (abc1234 - yours will be different). The commit ID is like a fingerprint that identifies this specific commit.
• 4 files changed, 45 insertions(+) - This is a summary: 4 files were added, and 45 lines of code were inserted. The (+) means all changes were additions (no deletions).
• create mode 100644 - This means Git created new files with standard file permissions. The 100644 is a permission code that means "regular file that can be read and written."

Each line lists a file that was created in this commit. Your commit was successful! 🎉

Create GitHub Repository

Now let's create a home for your code on GitHub!

Why are we using GitHub for this project?

We're using GitHub because GitHub Actions - the CI/CD tool we're building in this project - is built directly into GitHub.

When we push our code to GitHub, GitHub Actions can automatically detect changes, run our tests, and deploy our API - all without us having to manually run commands or click buttons. This is exactly what we'll set up in the next steps!

• Open your web browser and go to github.com.
• Sign in to your GitHub account.

Don't have a GitHub account?

No problem!

1. Go to github.com/signup
2. Follow the prompts to create a free account
3. Verify your email address
4. Come back here when you're signed in!

If you'd like step by step guidance for creating your GitHub account, check out Step #3 in Connect a GitHub Repo with AWS.

• Click the + icon in the top-right corner.
• Select New repository.

What is "New repository"?

A repository (or "repo") is like a folder on GitHub where you store your project's code, files, and version history. When you select New repository, you're telling GitHub you want to create a brand new, empty folder to upload your code to.

• Fill in the repository details:
  • Owner: Choose your GitHub profile
  • Repository name: nextwork-rag-api
  • Description: RAG API with CI/CD pipeline

• Visibility: Choose Public (so you'll get to use GitHub Actions for free)
• Click Create repository.

You'll see a page with setup instructions. Keep this page open - we'll use it next!

What did we just create?

You created an empty GitHub repository (repo) - think of it as an empty folder on GitHub's servers. Now we'll upload your local code to this remote repository.

Connect Local Git to GitHub

Let's link your local Git repository to the GitHub repository you just created.

What's the difference between a local repository and a GitHub repository?

Right now, you have two separate repositories:

• Local repository - This is the Git repository on your computer (the .git folder we created with git init). It tracks your code changes, commits, and history, but it only exists on your machine.
• GitHub repository (remote) - This is the empty repository we just created on GitHub's servers. It's like a backup and sharing location in the cloud, but it doesn't have any of your code yet.

💡 Why link them together?

By linking your local repository to the GitHub repository, you create a connection between the two. This lets you:

• Push (upload) your local commits to GitHub so others can see your code
• Pull (download) changes from GitHub to keep your local code in sync
• Back up your code in the cloud
• Collaborate with others by sharing the same remote repository

Think of it like connecting your local photo album to a cloud storage service - your photos stay on your computer, but you can sync them to the cloud whenever you want!

• On the GitHub repository page, find the section "…or push an existing repository from the command line".

• Copy the first line with your SSH repository URL. It should look like:

What does this command do?

git remote add origin creates a connection between your local repository and the GitHub repository. Let's break it down:

• git remote add - This command adds a new remote repository connection
• origin - This is the default name for your remote repository (like a nickname). You could use any name, but origin is the standard convention
• git@github.com:username/repo.git - This is the SSH URL that tells Git where to find your GitHub repository

After running this command, whenever you use origin (like git push origin main), Git knows you're referring to your GitHub repository!

Connect Local Repo to GitHub

Let's connect your local Git to GitHub using SSH!

In your terminal, add the GitHub repository as a remote.

• Paste the command in your terminal:

Why are we using SSH to connect our local and remote repositories?

If you've used GitHub before, you might be familiar with HTTPS as another way to connect repositories. However, we're focusing on SSH for this project because:

• HTTPS is being phased out - GitHub is moving away from password-based HTTPS authentication
• More secure - SSH uses cryptographic keys instead of passwords or tokens
• No repeated authentication - Once set up, you never need to enter credentials again
• Industry standard - Most professional development teams use SSH for Git operations

While the HTTPS method can work, SSH is the recommended approach and what you'll encounter in professional environments!

Now let's test the SSH connection to see if your SSH keys are set up.

• Run this command to test your SSH connection to GitHub:

ssh -T git@github.com

What do you see in your terminal?

❌ I see an error

You might see an error like:

• Permission denied (publickey)
• Host key verification failed
• Could not resolve hostname

This means you need to set up SSH keys for GitHub.

What is an SSH key?

An SSH key is like a secure password that proves you are who you say you are, without typing a password. It comes in two parts:

• Private key (e.g. id_ed25519, my_key, or any name you choose) - Never share this! Stays on your computer. The "id_" prefix is just a common convention, not a requirement.
• Public key (e.g. id_ed25519.pub, my_key.pub) - Safe to share, goes on GitHub. Always has a .pub extension.

Check if you already have an SSH key.

• Run this command in your terminal and press Enter:

🍎 macOS / Linux

ls -la ~/.ssh

What does this command do?

This command lists all files in your ~/.ssh directory (where SSH keys are stored):

• ls - Lists files and folders
• -la - Shows detailed information (-l = long format, -a = all files including hidden ones)
• ~/.ssh - The directory where SSH keys are stored (~ is your home folder)

If you see files ending in .pub (like id_ed25519.pub, id_rsa.pub, or any other .pub file), you already have SSH keys set up! The files don't have to start with "id" - that's just a common naming convention.

🖼️ Windows

Get-ChildItem ~/.ssh

What does this command do?

This command lists all files in your ~/.ssh directory (where SSH keys are stored):

• Get-ChildItem - PowerShell's equivalent of ls - lists files and folders
• ~/.ssh - The directory where SSH keys are stored (~ is your home folder)

If you see files ending in .pub (like id_ed25519.pub, id_rsa.pub, or any other .pub file), you already have SSH keys set up! The files don't have to start with "id" - that's just a common naming convention.

What do you see?

❌ I don't see any keys

You'll see an error like "No such file or directory" or an empty directory. That means you need to generate a new SSH key.

Generate a new SSH key.

• Run this command (replace your_email@example.com with your GitHub email):

ssh-keygen -t ed25519 -C "[[GITHUB_EMAIL="your_email@example.com"]]"

What does this command do?

This command generates a new SSH key pair on your local computer. Here's how it works:

• ssh-keygen - The tool that creates SSH keys
• -t ed25519 - Specifies the key type (Ed25519 is a modern, secure, and fast algorithm)
• -C "your_email@example.com" - Adds a comment/label to your key (your email) to help you identify it later

💡 Extra for Experts: How does it generate the key?

When you run this command, your computer uses cryptographic algorithms to create two mathematically linked files:

1. Private key (e.g., id_ed25519) - Stays on your computer, never share this!
2. Public key (e.g., id_ed25519.pub) - Safe to share, this is what you'll add to GitHub

These keys work together like a lock and key system - your private key "unlocks" access, and your public key is the "lock" that GitHub uses to verify your identity. The mathematical relationship between them ensures that only your private key can work with your public key.

• Press Enter to accept the default file location.
• Press Enter twice to skip the passphrase (or set one if you prefer).

You should see output like:

Your identification has been saved in /Users/yourname/.ssh/id_ed25519
Your public key has been saved in /Users/yourname/.ssh/id_ed25519.pub

✅ I see keys

Perfect! If you see files ending in .pub (like id_rsa.pub, id_ed25519.pub, or any other .pub file), you already have an SSH key! The files don't have to start with "id" - that's just a common naming convention.

Great! You can skip the key generation step and go directly to adding your existing key to GitHub.

Add SSH Key to GitHub

Copy your SSH public key.

• Run this command:

🍎 macOS

pbcopy < ~/.ssh/[[SSH_KEY_FILE="your SSH key"]]

What does pbcopy do?

pbcopy copies the contents of a file to your clipboard (like Cmd+C). The < symbol means "read from file".

💡 What if my key has a different name?

If you see a different key file (like id_rsa.pub), replace id_ed25519.pub with your actual key filename. Use the .pub file (the public key), not the private key!

🖼️ Windows

Get-Content ~/.ssh/[[SSH_KEY_FILE="id_ed25519.pub"]] | Set-Clipboard

What does this command do?

Get-Content reads the file, and the | (pipe) sends it to Set-Clipboard, which copies it to your clipboard (like Ctrl+C).

💡 What if my key has a different name?

If you see a different key file (like id_rsa.pub), replace id_ed25519.pub with your actual key filename. Use the .pub file (the public key), not the private key!

Your public key is now copied to your clipboard (no output is shown).

• Go to GitHub in your browser.
• Click your profile picture (top-right) → Settings.

• In the left sidebar, click SSH and GPG keys.

• Click New SSH key.
• Fill in:
  • Title: "My Laptop" (or any name you'll remember)
  • Key: Paste your public key (Cmd+V or Ctrl+V)

• Click Add SSH key.
• Confirm with your GitHub password if prompted.

Having trouble with SSH keys?

Get help generating or adding SSH keys to GitHub here.

Now test your SSH connection again:

ssh -T git@github.com

You should now see a success message!

✅ I see 'you've successfully authenticated, but...'

You'll see a message like:

Perfect! Your SSH keys are already set up and working. You can skip the SSH key setup steps and proceed to the next section.

What does "GitHub does not provide shell access" mean?

This message is normal and expected! It still means your SSH authentication worked perfectly.

The "but GitHub does not provide shell access" part can sound confusing, but it's just GitHub being clear about what SSH is used for:

• "Shell access" would mean logging into GitHub's servers to run commands directly (like ls or cd). GitHub doesn't allow this for security reasons.
• What SSH actually does here - SSH is only used to authenticate your identity for Git operations like git push and git pull. You're not trying to log into their servers - you're just proving who you are so Git can securely upload/download your code.

Let's verify the remote was added.

• Run this command and press Enter:

git remote -v

You should see:

origin  git@github.com:[[GITHUB_USERNAME="your-github-username"]]/nextwork-rag-api.git (fetch)
origin  git@github.com:[[GITHUB_USERNAME="your-github-username"]]/nextwork-rag-api.git (push)

What does -v mean?

The -v flag means "verbose" - it shows you more detailed information. In this case, it displays the URL of your remote repository twice: once for fetching (downloading) code, once for pushing (uploading) code.

For this project, both URLs are the same, which is normal. This just confirms that your local repository is connected to your GitHub repository!

Push Your Code to GitHub

Time to upload your code to GitHub!

Set your default branch name to main.

• Run this command and press Enter:

git branch -M main

Why "main"?

GitHub's default branch is called main. This command renames your local repository's default branch from master to main to match the way GitHub names its branches.

Push your code to GitHub.

• Run this command and press Enter:

git push -u origin main

Getting push errors?

Get help troubleshooting git push failures here.

You should see output like:

What does -u origin main mean?

• -u: Sets up tracking so future git push commands know where to push
• origin: The name of your remote repository (on GitHub)
• main: The branch you're pushing

💡 Extra for Experts: What does the output mean?

When you run git push, you'll see output showing Git uploading your code. Here's what it means:

• "Enumerating objects" - Git is counting all the files and changes to upload
• "Compressing objects" - Git is compressing your files to make the upload faster
• "Writing objects" - Git is uploading your files to GitHub
• "Total 5 (delta 0)" - This shows how many files were uploaded (5 in this example)
• "* [new branch] main -> main" - This confirms a new branch called main was created on GitHub
• "branch 'main' set up to track 'origin/main'" - This means your local main branch is now linked to the main branch on GitHub

If you see all of this without errors, your code was successfully uploaded to GitHub! 🎉

Configure Git

If this is your first time using Git, you'll also need to configure your name and email. This information will be associated with your commits.

Let's check if Git is already configured on your computer:

• Run these commands to check your current Git configuration:

git config --global user.name
git config --global user.email

What do you see? Select the tab below that matches what you see in your terminal.

❌ I see nothing or an error

This means Git isn't configured on your computer yet. This is totally normal if you're new to Git! You'll need to set up your name and email so Git knows who you are when you make commits.

• Run these commands (replace with your own name and GitHub email):

git config --global user.name "[[GIT_USER_NAME="Your Name"]]"
git config --global user.email "[[GIT_USER_EMAIL="your.email@example.com"]]"

Why do we need to configure Git?

Every time you make a commit, Git records who made that change. It needs your name and email to:

• Attribute commits to you - So you (and others) can see who made each change
• Track contributions - GitHub uses this to show your contribution history
• Enable collaboration - When working with others, everyone needs to know who made which changes

Without this configuration, Git won't know who you are and might reject your commits or show them as anonymous!

💡 What does --global mean?

The --global flag means this configuration applies to all Git repositories on your computer, not just this one project. This way, you only need to configure it once and Git will use your name and email for all future commits!

✅ I see my name and email

Perfect! Your Git is already set up with your name and email. You can skip the configuration steps and proceed to the next section.

• Head back to your GitHub repository page in your browser.
• Refresh your tab.

You should now see your files (app.py, embed.py, k8s.txt, .gitignore) on GitHub!

Congratulations! Your RAG API is now on GitHub and ready for CI/CD!

What did we just accomplish?

Your code now lives in two places:

1. Locally on your computer (where you write code)
2. Remotely on GitHub (where others can see it and where CI will run)

This is the foundation of modern software development!

Create Semantic Tests

Before we automate anything with CI/CD, let's create tests for our RAG system locally and see how they behave!

In this step, you'll create semantic tests and run those tests against your RAG API.

What are semantic tests?

Semantic tests check that your RAG system returns answers with the right meaning, not just the right format. For example, when asking "What is Kubernetes?", we want to verify the answer includes key concepts like "orchestration" and "containers".

This is different from typical unit tests - we're testing data quality, not just code correctness!

💡 Why create tests before automation?

Creating and running tests locally helps us understand how our RAG system behaves before we try to automate testing with CI/CD. This hands-on experimentation will reveal important insights about testing AI systems!

In this step, you're going to:

• Create semantic tests for your RAG API
• Run tests locally and see tests pass
• Experiment with breaking and fixing your knowledge base
• Test your API repeatedly and observe interesting behavior
• Discover an important challenge (that we'll solve in Step #4!)

Create Semantic Test Script

Let's write tests that check if our RAG system retrieves the correct information! We'll create these tests first, then automate them with CI.

• In Cursor, select the New File icon in the file explorer.

• Save the file as semantic_test.py in your project root.

• Paste this content:

import requests

def test_kubernetes_query():
    response = requests.post("http://127.0.0.1:8000/query?q=What is Kubernetes?")
    
    if response.status_code != 200:
        raise Exception(f"Server returned {response.status_code}: {response.text}")
    
    answer = response.json()["answer"]

    # Check for key concepts
    assert "orchestration" in answer.lower(), "Missing 'orchestration' keyword"
    assert "container" in answer.lower(), "Missing 'container' keyword"

    print("✅ Kubernetes query test passed")

if __name__ == "__main__":
    test_kubernetes_query()
    print("All semantic tests passed!")

What does this code do?

This test script verifies that your RAG API returns semantically correct answers:

• import requests: Imports the requests library to make HTTP requests to your API
• test_kubernetes_query() function:
  • Sends a POST request to your RAG API with the question "What is Kubernetes?"
  • Extracts the JSON response and gets the answer field
  • Uses assert statements to check if the answer contains key concepts like "orchestration" and "container" (converted to lowercase for case-insensitive matching)
  • If both keywords are found, prints a success message
• if __name__ == "__main__": Ensures the test only runs when you execute the script directly (not when imported as a module)

This is a semantic test - instead of checking for exact text matches (which would fail due to LLM non-determinism), it checks for the presence of important concepts that should appear in any correct answer about Kubernetes.

• Save the file.

Test the Semantic Test Script Locally

Let's verify the test works before automating it with CI!

First, make sure your API is running.

• In a terminal, start your API:

uvicorn app:app --reload

What do you see? Select the tab below that matches what you see in your terminal.

✅ I see the API starting successfully

Perfect! Your API is starting up. You should see output like "Uvicorn running on http://127.0.0.1:8000". Keep this terminal running and proceed to the next step!

❌ I see 'command not found: uvicorn'

This error means either your virtual environment isn't activated, or uvicorn isn't installed. Here's how to fix it:

1. Make sure your virtual environment is activated - You should see (venv) at the start of your terminal prompt. If you don't see it:
   • macOS/Linux: bash source venv/bin/activate - Windows: bash venv\Scripts\activate
2. If you still see the error after activating venv, install uvicorn:
   1. Try running the uvicorn command again: bash uvicorn app:app --reload
3. If you still have issues, make sure you're in the correct directory (where your app.py file is located):

pip install uvicorn

cd /path/to/your/project

Why test locally first?

Testing locally helps us verify the test script works correctly before automating it. This way, if something's wrong, we can fix it quickly without waiting for CI to run!

Let's test our API in a new terminal tab (keep the API running in the first one).

• Select the New Terminal button at the top of your terminal window.

A new terminal will open up beneath your existing terminal.

• Activate your virtual environment in the new terminal tab.

Why do we need to activate the virtual environment?

Since you opened a new terminal tab, it doesn't automatically have your virtual environment activated - each new terminal session starts fresh. The packages we installed earlier (like requests) are only available inside your virtual environment, so we need to activate it before running the test!

• Run these commands:

🍎 macOS / Linux

cd ~/Desktop/nextwork-rag-api
source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt.

🖼️ Windows

cd ~\Desktop\nextwork-rag-api
venv\Scripts\activate

You should see (venv) appear at the beginning of your PowerShell prompt.

• Run the semantic test:

python semantic_test.py

What should happen?

The test will:

1. Send a query to your API: "What is Kubernetes?"
2. Check if the answer contains "orchestration" and "container"
3. Print "✅ Kubernetes query test passed" if both keywords are found

Check your test result:

✅ Test passed

You should see that the ✅ Test passes since your k8s.txt contains both keywords - "orchestration" and "containers".

What does this mean?

Your test successfully connected to the API and verified that the response contains the expected keywords. The API is running correctly and returning answers based on your knowledge base!

💡 Extra for Experts: How do companies use semantic tests in production systems today?

Semantic testing scales to hundreds of documents and queries:

• Test that medical docs contain required disclaimers
• Verify legal docs include necessary clauses
• Check that product docs mention key features
• Ensure customer support docs cover common issues

Semantic tests guard your RAG system's quality at scale!

❌ Connection refused error

If you see an error like this:

ConnectionRefusedError: [Errno 61] Connection refused
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded

What does this error mean?

This error means the test script can't connect to your API server. The API either isn't running, or it's running on a different port.

Here's how to fix it:

1. Check if your API is running:

• Return to the terminal tab where you started the API (with uvicorn app:app --reload)
• Make sure you see the API running and showing "Application startup complete"
• If the API isn't running, start it again:

1. If you're using the same repo from Projects #2 and #3:

• You might be using an old app.py file that was configured for Docker containers
• In those projects, the API ran inside Docker containers, but in this project we're running the API directly on your local machine
• The app.py from previous projects might have different configuration (like different Ollama connection settings) that doesn't work with the local server setup
• Open app.py and make sure it matches the code from this project (it should use chromadb and ollama with the correct structure for local development)
• If needed, update your app.py to:

from fastapi import FastAPI
import chromadb
import ollama
import os
app = FastAPI()
chroma = chromadb.PersistentClient(path="./db")
collection = chroma.get_or_create_collection("docs")
# Use environment variable for Ollama host, default to localhost for local development
ollama_host = os.getenv("OLLAMA_HOST", "http://localhost:11434")
ollama_client = ollama.Client(host=ollama_host)
@app.post("/query")
def query(q: str):
results = collection.query(query_texts=[q], n_results=1)
context = results["documents"][0][0] if results["documents"] else ""
answer = ollama_client.generate(
 model="tinyllama",
 prompt=f"Context:\n{context}\n\nQuestion: {q}\n\nAnswer clearly and concisely:"
)
return {"answer": answer["response"]}

• After updating app.py, restart your API server (stop it with Ctrl+C, then start it again)

uvicorn app:app --reload

1. Once your API is running, try the test again:

python semantic_test.py

❌ Command not found: python

What does this error mean?

This error means that Python is not installed on your system, or it's not available in your PATH. The terminal can't find the python command to run your test script.

Here's how to fix it:

1. Check if Python is installed:Note: This alias only works in your current terminal session. To make it permanent, add it to your shell configuration file (.zshrc for zsh or .bashrc for bash)
   • Try using python3 instead of python: bash python3 semantic_test.py - On macOS and Linux, Python 3 is often installed as python3 instead of python 2. If python3 works, you have two options: - Option A: Continue using python3 for all Python commands in this project - Option B: Create an alias so python points to python3: bash alias python=python3
2. If neither python nor python3 works, install Python:
   • On macOS:
     • Install Homebrew if you don't have it: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
     • Then install Python: brew install python3
   • On Linux:
     • Ubuntu/Debian: sudo apt update && sudo apt install python3
     • Fedora: sudo dnf install python3
   • On Windows:
     • Download Python from python.org
     • Make sure to check "Add Python to PATH" during installation
3. Verify Python is installed:You should see something like Python 3.x.x 5. Once Python is working, try the test again:

python3 --version

python3 semantic_test.py

Perfect! The test script works and passes because your k8s.txt still contains "containers" and "orchestration". Now let's see what happens when we break the file!

Break the File

Let's intentionally remove a key word from the Kubernetes document to see how the system behaves.

• Open k8s.txt in Cursor.

• The file should currently contain:

Kubernetes is a container orchestration platform used to manage containers at scale.

• Remove the word "orchestration" so it now reads:

Kubernetes is a container platform used to manage containers at scale.

• Save the file.

Why are we breaking the file?

We're intentionally removing "orchestration" - a key concept in Kubernetes! This will help us discover an important problem with LLM-based testing. Don't worry, we'll fix it later!

Rebuild Embeddings

Since we changed the source document, we need to rebuild the embeddings so they reflect the updated content.

Why do we need to rebuild embeddings?

When you change your knowledge base document, you need to regenerate the embeddings so your RAG system can find the updated information correctly.

Without rebuilding, your API would still search using the old embeddings (with "orchestration" included), even though the source document no longer contains that word! This mismatch means your API could answer questions about content that doesn't exist in your knowledge base anymore, leading to incorrect answers and unreliable tests.

• Delete the old database folder to start fresh:

rm -rf db/

Why delete the db/ folder?

Deleting the db/ folder ensures we start with a completely fresh database. This guarantees that the new embeddings reflect the updated k8s.txt content (without "orchestration").

• Run the embedding script to rebuild your database:

python embed.py

Test Your API Again

Now let's test the API directly to see what happens!

• Return to the terminal tab where your API is running.
• Press Ctrl+C to stop the API server.

Why are we stopping the server?

We need to restart the API server so it can load the newly rebuilt embeddings. When you rebuild embeddings, the API needs to be restarted to pick up the changes - the running server is still using the old embeddings from memory!

Pressing Ctrl+C safely stops the currently running process, allowing you to start it again with the updated database.

• Start your API again:

uvicorn app:app --reload

• In your second terminal tab (where you've been teseting your API), test the API directly using curl:

curl -X POST "http://127.0.0.1:8000/query" -G --data-urlencode "q=What is Kubernetes?"

What does this curl command do?

This sends a POST request to your API asking "What is Kubernetes?". The -G flag and --data-urlencode properly encode the query parameter.

• Run the command 3-5 times and watch the output carefully.

You'll notice something interesting:

• Sometimes the answer includes "orchestration" even though we removed it from k8s.txt!
• Sometimes it doesn't include "orchestration"
• The output varies each time you run the same query

What's happening?

This is non-deterministic LLM behavior! The LLM (tinyllama) was trained on data that includes "Kubernetes" and "orchestration" together. Even though your source document no longer contains "orchestration", the LLM sometimes adds it based on its training data.

This is a fundamental problem for testing:

When you send the same question to your API multiple times, you get different answers each time. Your tests can't reliably check if your API is working correctly - they might pass when they should fail, or fail when they should pass, just because the LLM generated a different response.

🤔 Can we automate these tests with non-deterministic behavior?

No! Imagine this scenario:

You set up your tests to run automatically in your CI/CD pipeline. Every time you push code, the tests run and check if everything is working. But because the LLM generates different responses each time, your tests fail randomly - sometimes they pass, sometimes they fail, even when your code hasn't changed at all.

When this happens, you can't tell if your code is actually broken or if the LLM just happened to generate a slightly different response this time. Over time, you start losing trust in your automated tests because they're giving you false alarms. You might start ignoring test failures, thinking "oh, it's probably just the LLM being random again." But then one day, your code actually breaks, and you miss it because you've learned to ignore the test failures. Your automated tests become useless!

This is exactly why we need a solution locally before building automation.

Let's solve this problem by implementing mock LLM mode - a way to test retrieval quality without relying on LLM generation. This will make your tests deterministic and enable reliable automation!

Extra for Experts: Why not just use better prompts?

You might wonder: "Can't I just write better prompts to make the LLM deterministic?" Even with perfect prompts and low temperature settings, LLMs remain fundamentally probabilistic. They use random sampling during generation, which means:

• The same input can produce different outputs
• Different model versions can produce different outputs
• Even "deterministic" settings vary across platforms

For reliable CI/CD, we need mock mode to test retrieval independently from generation.

Implement Mock LLM Mode

In Step #3, you discovered that LLM outputs are non-deterministic - the same query produces different responses each time. This makes automated testing hard to trust!

The solution? Mock LLM mode!

Instead of using the LLM to generate answers during testing, we'll make the API return the retrieved context directly.

What is retrieved context?

Retrieved context is the raw text chunks from your source documents (like k8s.txt) that the system finds when you ask a question.

In mock LLM mode, instead of sending this context to the LLM to generate an answer, we'll return the raw retrieved text directly. This lets us test whether the right information was found, without the LLM adding variability!

This solves the non-determinism problem:

• Deterministic output - Same query → Same retrieved document → Same response every time
• Tests retrieval quality - We verify that the right documents are retrieved, without LLM variability
• Faster tests - No need to wait for LLM generation during CI
• No Ollama dependency - Tests can run anywhere without installing Ollama

What is a mock?

A mock is a simplified version of a component used for testing. Think of it like using a dummy credit card for testing payments - you want to test the checkout flow without charging real money!

In our case, we'll "mock" the LLM by returning retrieved text directly, so we can test retrieval quality without needing a real AI model or dealing with non-deterministic outputs.

In this step, you're going to:

• Test your API locally with mock mode
• Commit and push the mock mode implementation

Add Mock LLM Flag to app.py

Let's modify app.py to support a mock mode that returns retrieval results directly.

• Open app.py in Cursor.
• Replace the entire content with this updated version:

import os
from fastapi import FastAPI
import chromadb

# Mock LLM mode for CI testing
USE_MOCK_LLM = os.getenv("USE_MOCK_LLM", "0") == "1"

if not USE_MOCK_LLM:
    import ollama

app = FastAPI()
chroma = chromadb.PersistentClient(path="./db")
collection = chroma.get_or_create_collection("docs")

@app.post("/query")
def query(q: str):
    results = collection.query(query_texts=[q], n_results=1)
    context = results["documents"][0][0] if results["documents"] else ""

    if USE_MOCK_LLM:
        # In mock mode, return the retrieved context directly
        return {"answer": context}

    # In production mode, use Ollama
    answer = ollama.generate(
        model="tinyllama",
        prompt=f"Context:\n{context}\n\nQuestion: {q}\n\nAnswer clearly and concisely:"
    )

    return {"answer": answer["response"]}

• Save the file.

What changed?

We've added a mock LLM mode that lets your app run in CI without needing Ollama installed. Instead of generating AI responses, it returns the retrieved context directly - making tests fast, deterministic, and focused on verifying that your retrieval system works correctly.

Here's how it works:

1. Environment variable check: The code uses USE_MOCK_LLM = os.getenv("USE_MOCK_LLM", "0") == "1" to read the USE_MOCK_LLM environment variable. If the environment variable isn't set, it defaults to "0" (off), which means mock mode is disabled by default.
2. Conditional import: The code only imports Ollama when mock mode is off (if not USE_MOCK_LLM: import ollama). This means when mock mode is enabled, Ollama isn't imported at all, which lets CI run without Ollama installed!
3. Mock response: When USE_MOCK_LLM=1, the API returns the retrieved context directly instead of sending it to the LLM for generation. This means no LLM generation happens, which gives you deterministic output that's perfect for testing retrieval quality!

Test Mock Mode Locally

Let's verify mock mode works!

• Start your API in mock mode.
• Run this command:

🍎 macOS / Linux

USE_MOCK_LLM=1 uvicorn app:app --reload

🖼️ Windows

$env:USE_MOCK_LLM=1
uvicorn app:app --reload

What does USE_MOCK_LLM=1 do?

This sets an environment variable that tells your app to use mock mode. Environment variables are like settings you pass to your program when starting it.

• Switch to your second terminal (that you're using for running tests).
• Paste this command into your terminal and press Enter (expect an error!):

python semantic_test.py

You should see this error:

AssertionError: Missing 'orchestration' keyword

• Run the test 1-2 more times.

This time, the test will fail consistently - and that's exactly what we want!

Why is this good?

This demonstrates that mock mode is deterministic! The test consistently fails because we removed "orchestration" from k8s.txt earlier.

In mock mode, the API returns the exact retrieved context from your documents. Since we removed "orchestration" from k8s.txt and rebuilt the embeddings, the retrieved context no longer contains that word. The test fails every single time - no randomness, no variability!

This is exactly what we need for reliable automated testing. Unlike the non-deterministic behavior we saw earlier (where tests sometimes passed and sometimes failed), mock mode gives us consistent, predictable results that we can trust.

Perfect! We now have deterministic tests!

Commit and Push Mock Mode

Now that mock mode works locally, let's save it to GitHub!

Note: We're only committing app.py right now. The k8s.txt file still has "orchestration" removed (from Step #3) - we'll keep this change locally for now (which will come in handy in Step #6)!

Add app.py to staging.

• Run this command and press Enter:

git add app.py

What does this command do?

git add app.py stages only the app.py file for the next commit. This is different from git add . (which we used earlier) - that command stages all files in your current directory, except those listed in .gitignore.

By using git add app.py, we're being selective and only staging the mock mode implementation. This means other changed files (like the "broken" k8s.txt that's missing "orchestration") won't be included in this commit. We'll handle those files later!

Commit the changes.

• Run this command and press Enter:

git commit -m "Add mock LLM mode for deterministic testing"

Push to GitHub.

• Run this command and press Enter:

git push

Great! Your mock mode implementation is now on GitHub.

Awesome!

You've successfully implemented mock LLM mode, which solves the non-determinism problem we discovered in Step #3. Your API can now:

• Run in production mode with real LLM generation for users
• Run in mock mode with deterministic output for testing

What's the difference between Production and Mock mode?

To summarize what we just implemented:

• Production (USE_MOCK_LLM=0): Uses real Ollama LLM for natural answers
• Mock mode (USE_MOCK_LLM=1): Returns retrieved context for deterministic testing

This pattern lets you test data quality (retrieval) without the variability of LLM generation!

In Step #5, you'll put this all together by creating a GitHub Actions workflow that uses mock mode for automated testing.

Create GitHub Actions Workflow

Now it's time to automate everything with GitHub Actions! In this step, you'll create a CI/CD workflow that automatically tests your RAG API whenever you push code.

The best part? You've already solved the hardest problem (non-deterministic LLM output) with mock mode in Step #4. Now we just need to set up the automation!

What is GitHub Actions?

GitHub Actions is GitHub's automation platform. You write a YAML file describing what to do (e.g., "install Python, run tests"), and GitHub runs it automatically on their servers when you push updated code to your repository.

It's how companies deploy code hundreds of times per day!

💡 Why automate now?

Without automation, every change requires manual steps:

• 📝 You update a doc → manually rebuild embeddings → manually test → manually verify
• 👥 A teammate changes content → you don't know if it breaks anything until users complain
• 💻 Tests work on your laptop → but might fail in production

With GitHub Actions, all these steps happen automatically on every push!

In this step, you're going to:

• Create a GitHub Actions workflow
• Trigger the workflow and watch it run in GitHub!

Create Workflows Directory

GitHub Actions workflows live in a special .github/workflows/ folder in your repo.

• In the file explorer (left sidebar), select New Folder.

• Name it .github/workflows and press Enter.

Why .github/workflows?

GitHub looks for automation workflows in .github/workflows/ by convention. Any .yml file you add to this folder defines a workflow!

Create CI Workflow File with Mock Mode

Let's write the workflow that will test your RAG API! Notice that we're including mock mode from the start - this is the solution to the non-determinism problem we discovered in Step #3!

• In your new folder, create a new file called ci.yml.

• Paste this YAML content:

name: RAG CI Pipeline

on:
  push:
    paths:
      - 'k8s.txt'
      - 'app.py'
      - 'embed.py'

jobs:
  test-and-build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install fastapi uvicorn chromadb requests

      - name: Rebuild embeddings
        run: python embed.py

      - name: Start API in mock mode
        run: |
          USE_MOCK_LLM=1 uvicorn app:app --host 0.0.0.0 --port 8000 &
          sleep 5

      - name: Run semantic tests
        run: python semantic_test.py

      - name: Success
        run: echo "✅ All tests passed! RAG quality maintained."

• Save the file.

What does this workflow do?

This GitHub Actions workflow automatically tests your RAG system whenever you push code changes. It rebuilds embeddings, starts your API in mock mode (for deterministic testing), and runs semantic tests to verify that your RAG system still returns correct answers.

Let's break down each section:

Trigger (on:):

• The workflow runs when you push code that changes k8s.txt, app.py, or embed.py. These are the critical files that affect your RAG system's behavior - if any of these change, we need to test to make sure everything still works!
• The workflow ignores changes to other files (like README.md or documentation). This saves time and resources by only running tests when files that actually affect your RAG system are modified.

Environment (runs-on:):

• The workflow runs on Ubuntu Linux, which is GitHub's free runner (a virtual machine that executes your workflow). Think of it as a fresh, clean computer that GitHub provides just for your tests - it starts fresh each time, so you know your tests aren't affected by leftover files or configurations from previous runs.

Steps (steps:):

These are the actions the workflow performs in order:

1. Checkout - Downloads your code from GitHub so the runner has access to all your files (like app.py, embed.py, k8s.txt, etc.)
2. Set up Python - Installs Python 3.11 on the runner, since your code needs Python to run
3. Install dependencies - Installs the Python packages your project needs (like fastapi, chromadb, requests) from your requirements.txt file
4. Rebuild embeddings - Runs embed.py to create fresh embeddings from your source documents. This ensures the tests use the most up-to-date knowledge base
5. Start API in mock mode - Starts your API server with USE_MOCK_LLM=1 environment variable. This makes the API return retrieved context directly (deterministic!) instead of using the LLM
6. Run tests - Executes semantic_test.py to verify that your RAG system returns the expected answers. If any test fails, the workflow stops and reports the failure
7. Success - If all tests pass, prints a success message confirming everything works correctly

Commit and Push Workflow

Let's commit the workflow file and test script to GitHub!

Why commit these files to GitHub?

GitHub Actions only runs workflows that are committed to your repository. By committing the workflow file (.github/workflows/ci.yml), you're telling GitHub to automatically run your CI pipeline whenever you push code. The test script (semantic_test.py) also needs to be in the repository so the workflow can access and execute it during the workflow run!

Add the workflow file and test script to staging.

• Run these commands and press Enter:

git add .github/workflows/ci.yml
git add semantic_test.py

Commit the workflow and test script.

• Run this command and press Enter:

git commit -m "Add GitHub Actions CI workflow and semantic test script"

Push to GitHub.

• Run this command and press Enter:

git push

Verify your GitHub Actions Workflow

Let's check that your workflow file is on GitHub!

• Go to your GitHub repository in your browser:

https://github.com/[[GITHUB_USERNAME="your-github-username"]]/nextwork-rag-api

• Click the Actions tab at the top of your repository page.

Yay! Since we've pushed the workflow, you should see it running... right?

Why hasn't the workflow run?

Remember how we set up the workflow to only run when certain files change? The workflow will only start when GitHub sees changes to k8s.txt, app.py, or embed.py.

When we pushed the workflow file (.github/workflows/ci.yml) and the test script (semantic_test.py), those files weren't on the list, so GitHub didn't start the workflow.

No worries! We've set up your GitHub Actions workflow. In the next step, we'll make it run and see CI/CD in action!

Quick recap of where we are

• ✅ Created semantic_test.py (checks for "orchestration" and "container")
• ✅ Implemented mock LLM mode in app.py
• ✅ Created .github/workflows/ci.yml workflow
• 📝 k8s.txt currently has: "Kubernetes is a container platform used to manage containers at scale." (missing "orchestration" from Step #3)

The workflow is ready to test our code whenever we push changes to k8s.txt, app.py, or embed.py!

Trigger CI and Watch Semantic Tests Fail

Now let's see your GitHub Actions workflow in action! We'll trigger it by changing the knowledge base and observe how CI automatically catches data quality issues.

In this step, you're going to:

• Trigger your GitHub Actions workflow
• Watch GitHub Actions automatically test your code
• Observe the workflow fail due to data quality issues

Trigger the Workflow

Remember from Step #3, we already broke k8s.txt by removing "orchestration". That change hasn't been pushed to GitHub yet, so let's push it now to trigger the workflow!

Why are we pushing the broken version?

We're intentionally pushing the broken k8s.txt to demonstrate how CI catches data quality issues. When the workflow runs, it will rebuild embeddings from this broken file, run the tests, and fail because the retrieved context no longer contains "orchestration". This shows how CI protects your knowledge base quality!

• Run this command and press Enter:

git add k8s.txt

What does this command do?

git add k8s.txt stages only the k8s.txt file for the next commit. This prepares the broken version of k8s.txt (missing "orchestration") to be committed and pushed to GitHub, which will trigger the workflow.

Commit the change.

• Run this command and press Enter:

git commit -m "Update k8s.txt to trigger CI workflow"

Push to GitHub.

• Run this command and press Enter:

git push

What happens now?

Since you changed k8s.txt (which is in the paths: filter), GitHub Actions will automatically run the workflow you defined in ci.yml. As a quick recap, the steps in the workflow are:

1. Detect the push
2. Check out your code
3. Set up Python and dependencies
4. Rebuild embeddings
5. Start the API in mock mode
6. Run semantic tests

Let's watch it happen!

Watch Your First CI Run!

Let's see GitHub Actions automatically test your code!

• In your browser, head to the tab with GitHub Actions open.

You should see your workflow "RAG CI Pipeline" appear and start running!

When will the workflow start?

After you push, GitHub will detect that k8s.txt changed and start running the workflow. It may take a few seconds to appear - refresh the page if you don't see it immediately!

• Click on the workflow run to see details.

• Click on test-and-build to see each step executing in real-time.

Watch CI Catch the Issue

Watch as the steps progress...

• ✅ Checkout code
• ✅ Set up Python
• ✅ Install dependencies
• ✅ Rebuild embeddings
• ✅ Start API in mock mode
• ❌ Run semantic tests ← This step will FAIL!

The workflow fails! Let's see why.

Perfect! CI caught the data quality issue!

What just happened?

Your k8s.txt is missing "orchestration" (removed in Step #3). The semantic test checks for this keyword, and CI automatically caught that it's missing!

Mock mode was crucial here! Remember the non-determinism problem from Step #3? With mock mode, CI gets exactly the retrieved text: "Kubernetes is a platform used to manage at scale." This is deterministic - it will always fail the test because the keywords are genuinely missing.

💡 What did CI just prevent?

Without CI, you might have:

1. Pushed this degraded content
2. Deployed to production
3. Had your RAG system give incomplete answers
4. Only noticed when users complained

Instead, CI caught it in seconds, before any harm was done!

Congratulations! You've built a complete CI/CD pipeline for your RAG API that automatically catches data quality issues!

What you've accomplished

You've learned the full workflow:

1. Create semantic tests - Define what "quality" means for your RAG system
2. Solve non-determinism - Use mock mode for reliable testing
3. Automate testing - Set up GitHub Actions to test every change
4. Catch quality issues - Watch CI prevent data quality issues

This is how professional teams maintain RAG system quality at scale! Every push is automatically validated, preventing degraded content from reaching production.

Secret mission

Ready to level up?

Your current setup works great with one file (k8s.txt in the root). But real-world RAG systems often have hundreds or thousands of documents. Let's restructure your RAG system to handle multiple knowledge documents and see how CI automatically tests them all!

Restructure for Multiple Documents

Delete your resources

Delete your resources

Now that we've built and tested our CI/CD pipeline, it's time to clean up the resources we created. This is important to free up space on your computer and keep things organized.

Note

Heads up! Your GitHub repository is portfolio-worthy work! Consider keeping it to showcase your CI/CD skills to employers. You can always delete it later if you change your mind.

Resources to delete:

• Delete the local project folder (including venv/, db/, and all code files).
• Delete the local Git repository (.git folder).
• (Optional) Delete the GitHub repository.

Local Project Files

• Head to your project folder (likely ~/Desktop/nextwork-rag-api or similar).

Why delete local project files?

Your project folder contains the virtual environment (venv/), Chroma database (db/), and all your code files. These can take up several hundred megabytes of disk space. Since you've pushed everything to GitHub, you can always clone the repository again later if needed.

• Delete the entire project folder:

🍎 macOS

• In Finder, navigate to your Desktop (or wherever you created the project).
• Right-click on the nextwork-rag-api folder.
• Select Move to Trash.

Alternative: Using Terminal

You can also delete the folder using Terminal:

rm -rf ~/Desktop/nextwork-rag-api

⚠️ Warning: The rm -rf command permanently deletes files. Make sure you've pushed everything to GitHub first!

🖼️ Windows

• Open File Explorer.
• Navigate to your Desktop (or wherever you created the project).
• Right-click on the nextwork-rag-api folder.
• Select Delete.

Alternative: Using PowerShell

You can also delete the folder using PowerShell:

Remove-Item -Recurse -Force "$env:USERPROFILE\Desktop\nextwork-rag-api"

⚠️ Warning: This command permanently deletes files. Make sure you've pushed everything to GitHub first!

GitHub Repository

Should I delete my GitHub repository?

Your GitHub repository showcases your CI/CD skills and is great for your portfolio! Many employers look at GitHub profiles to see real projects with automated testing.

We recommend keeping your repository unless you're certain you won't need it. You can always delete it later from GitHub's settings.

If you do want to delete it, here's how:

• Head to your GitHub repository in your web browser.
• Select Settings from the repository's top menu.

• Scroll down to the Danger Zone section at the bottom of the page.
• Select Delete this repository.
• Type your repository name in the confirmation field (e.g., nextwork-rag-api).
• Select I understand the consequences, delete this repository.

Why delete a GitHub repository?

If you're cleaning up old projects or want to start fresh, deleting unused repositories helps keep your GitHub profile organized. However, remember that this action is permanent - you won't be able to recover the repository or its commit history once deleted.

Final reflection

That's a wrap!

That's a wrap!

Nice work! 🔄 You've just built a complete CI/CD pipeline for your RAG API using GitHub Actions.

You've learned how to:

• 🚀 Set up Git version control for your code.
• 🔗 Create and manage a GitHub repository.
• 📁 Structure your knowledge base with a scalable docs folder.
• 🤖 Implement mock LLM mode for deterministic testing.
• 🧪 Build semantic tests to catch data quality issues.
• ⚙️ Create GitHub Actions workflows for automated testing.
• 🛡️ Detect data quality issues before they reach production.
• 💎 Scale your knowledge base with multiple documents and CI validation.

Congratulations! 🎉

You now have the skills to build robust, automated RAG systems like professional AI engineers. Your CI/CD pipeline automatically tests every change, catches bugs before they reach production, and ensures data quality stays high as your knowledge base grows.

Ready to quiz yourself? 💪

Congratulations!

You've completed all four projects in the DevOps × AI Series! 🎊

Explore other NextWork projects to continue building your AI skills:

• Build a Mini Spotify with Cursor - Learn AI-assisted development
• Data Engineering with Postgres & Docker MCP - Build databases with AI

p.s. Does it say "Still tasks to complete!" at the bottom of the screen?

This means you still have screenshots left to upload, or questions left to answer!

1. Press Ctrl+F (Windows) or Command+F (Mac) on your keyboard.
2. Search for the text Return to later.
3. Jump straight to your incomplete tasks!
4. 🙋‍♀️ Still stuck? Ask the community!