Build an Incident Runbook Retrieval API

Build a local API that retrieves incident runbooks with embeddings and pgvector.

Introduction

30 Second Summary

Incident reports rarely use the same words as the runbook that contains the fix. A literal search can lose useful evidence when the meaning matches but the wording changes.

In this project, you will build a local incident runbook retrieval API with FastAPI. You will watch exact phrase matching fail on a paraphrase before semantic retrieval returns ranked evidence from PostgreSQL.

What You'll Build

You submit an incident description through browser-based documentation to see the most relevant runbook passages ranked with similarity scores.

By the end of this project, you'll have:

  • A literal incident search you can run from FastAPI's interactive REST documentation. The matching PostgreSQL runbook appears in the response.
  • A visible phrase-matching failure that returns "results": [] when the same incident uses different wording.
  • A semantic runbook search that stores embeddings beside operational text. pgvector ranks the evidence by meaning. Each result includes a numeric similarity score.
  • Secret Mission: Add a labeled benchmark that prints four retrieval outcomes before calculating Hit@1 quality.

Are there any prerequisites?

You need a Mac running macOS Sonoma 14 or newer.

You should already have Python 3.10 or newer plus Visual Studio Code.

Your machine needs at least 4 GB of RAM.

Before We Start

Before any setup, take a moment to commit to the incident question your API will answer. Your goal is to return relevant runbook evidence when incidents are described in different words.

Set Up the Python and Git Workspace

Dependency versions can drift between machines. A reproducible workspace keeps this retrieval API stable as you build it.

Git gives the API the version-control foundation expected in production AI work. A project-local Python environment keeps its packages isolated.

In this step, get ready to:
  • Confirm Git is available in the terminal.
  • Confirm Python 3.10 or newer is installed.
  • Create a Git-tracked workspace with isolated dependencies.
Check Git and Python

Git records changes to your project over time. Python runs the retrieval API in later steps.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your system search.
  • Type Terminal into the search field.
  • Press Enter to open Terminal.
  • Check whether Git is available by running this command:
git --version

What Does This Check Prove?

The command asks Git to report its installed version. A version response confirms that your terminal can find Git.

✔️ Git prints a version

Git is ready. You can initialize the project repository in the next substep.

ⓧ macOS shows an install prompt

macOS may offer the developer tools that provide Git. The installation can take several minutes, so a quiet progress window is expected.

  • Complete the developer tools installation offered by macOS.
  • Return to Terminal after the installation finishes.
  • Confirm Git by running the version check again:
git --version

What Should This Confirm?

A version response confirms that the developer tools made Git available to your terminal.

ⓧ Git is unavailable

The documented macOS installer for Git comes with the Xcode Command Line Tools. You can request that installer from Terminal.

  • Start the developer tools installer by running this command:
xcode-select --install

What Does This Command Do?

The command asks macOS to install the Xcode Command Line Tools. Those tools provide Git.

  • Complete the installer shown by macOS.
  • Return to Terminal after the installation finishes.
  • Confirm Git by running:
git --version

What Should This Confirm?

A version response confirms that Git is installed. It also confirms that your terminal can run it.

  • Check your Python version by running this command:
python3 --version

Why Check Python Now?

The pinned packages require Python 3.10 or newer. Checking the interpreter now prevents dependency installation failures later.

✔️ I see version 3.10 or higher

Good. Your Python interpreter meets the requirement for every pinned package in this project.

ⓧ I see an older version

macOS can include an older system-managed Python interpreter. The official Python installer adds a supported interpreter without removing the system copy.

  • Open the official Python releases for macOS in your browser.
  • Select a macOS installer for Python 3.10 or newer.
  • Open the downloaded .pkg file.
  • Complete the macOS Installer prompts.
  • Run Install Certificates.command from the installed Python folder in Applications.
  • Return to the VS Code terminal.
  • Confirm the supported Python version by running:
python3 --version

What Should This Confirm?

The output should now report Python 3.10 or newer. This interpreter can create the project environment.

ⓧ Command not found

Your terminal cannot find a Python installation. The official macOS installer provides the interpreter required by this project.

  • Open the official Python releases for macOS in your browser.
  • Select a macOS installer for Python 3.10 or newer.
  • Open the downloaded .pkg file.
  • Complete the macOS Installer prompts.
  • Run Install Certificates.command from the installed Python folder in Applications.
  • Return to the VS Code terminal.
  • Confirm Python by running:
python3 --version

What Should This Confirm?

A version response confirms that Python is installed. The reported version should be 3.10 or newer.

Still Blocked by a Version Check?

Close Terminal after an installation finishes. Open it again so the new command paths are loaded.

Ask for help with the version output.

Create the isolated workspace

A virtual environment gives this project its own package installation area. Packages installed there stay separate from your other Python projects.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your system search.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.

You should see the Visual Studio Code welcome window.

  • Click File in the top menu bar.
  • Click Open Folder.
  • Select Desktop in the folder dialog.
  • Click New Folder.
  • Enter incident-runbook-retrieval as the folder name.
  • Click Create.
  • Select the incident-runbook-retrieval folder.
  • Click Open.

You should see incident-runbook-retrieval at the top of the Explorer sidebar.

  • Click Terminal in the Visual Studio Code menu bar.
  • Click New Terminal.
  • Initialize the open folder as a Git repository by running:
git init

What Does Git Initialize?

Git creates repository metadata inside incident-runbook-retrieval. Future file changes can now be tracked from this folder.

The terminal confirms that it initialized an empty Git repository in the project folder.

  • Create the project-local environment by running:
python3 -m venv .venv

What Does the Environment Contain?

Python creates the .venv directory inside your project. It contains an isolated interpreter plus its own package installation area.

You should see the .venv directory appear in the Explorer sidebar.

  • Activate the virtual environment by running:
source .venv/bin/activate

What Does Activation Change?

Activation points the current terminal at the environment's Python interpreter. Package commands now affect this project environment.

Your terminal prompt should now include (.venv). That prefix confirms that the environment is active.

Virtual Environment Not Activating?

  • Check that the Explorer shows .venv inside incident-runbook-retrieval.
  • Check that the terminal prompt ends in the incident-runbook-retrieval folder.

Ask for help activating the environment.

Pin and install the dependencies

The dependency file records the exact package versions used by this API. The ignore file keeps generated Python files out of Git.

  • Select the Explorer icon in the left activity bar.
  • Click the New File control.
  • Enter requirements.txt as the file name.
  • Press Enter to create the file.

You should see an empty requirements.txt tab in the editor.

  • Fill requirements.txt with the pinned dependencies by pasting this content:
fastapi==0.162.0
uvicorn==0.54.0
psycopg==3.3.8
psycopg-binary==3.3.8
pydantic==2.14.3

What Do These Packages Provide?

  • FastAPI 0.162.0 provides the REST API framework.
  • Uvicorn 0.54.0 runs the API as a local web service.
  • Psycopg 3.3.8 connects Python to PostgreSQL.
  • psycopg-binary 3.3.8 provides the matching binary implementation.
  • Pydantic 2.14.3 validates incoming request data.
  • Save requirements.txt with Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm that requirements.txt remains visible in the Explorer sidebar.

Dependency File Looks Different?

Check that every package occupies its own line. Check that every version uses two equals signs.

Ask for help comparing the file.

  • Click the New File control in the Explorer sidebar.
  • Enter .gitignore as the file name.
  • Press Enter to create the file.

You should see an empty .gitignore tab in the editor.

  • Fill .gitignore with the generated Python paths by pasting this content:
.venv/
__pycache__/
*.pyc

What Does Git Ignore?

The .venv/ rule excludes the local environment. The remaining rules exclude Python cache files.

These files can be recreated from the project source. Excluding them keeps the repository focused on files you maintain.

  • Save .gitignore with Cmd+S (macOS) or Ctrl+S (Windows).
  • Confirm that .gitignore remains visible in the Explorer sidebar.

Ignore File Missing?

Check that the filename begins with one dot. Check that it has no .txt suffix.

Ask for help fixing the filename.

✔️ Awesome, I've got everything!

Your two workspace files are ready. Double-check that both files are saved before installing the packages.

ⓧ I'd like to double check the full code

Compare requirements.txt with this complete reference.

fastapi==0.162.0
uvicorn==0.54.0
psycopg==3.3.8
psycopg-binary==3.3.8
pydantic==2.14.3

What Should Match?

The package names must appear in this order. Every version pin must match exactly.

Compare .gitignore with this complete reference.

.venv/
__pycache__/
*.pyc

What Should Match?

The file must contain these three ignore patterns. Each pattern occupies its own line.

The pinned file now defines the workspace packages. Installing from that file turns the recorded versions into your active environment.

  • Install the pinned dependencies into the active environment by running:
python -m pip install -r requirements.txt

What Does This Installation Use?

Python runs the copy of pip associated with the active environment. Pip reads every pinned entry from requirements.txt.

The terminal prints download progress before returning to the prompt. Your isolated environment now contains the API dependencies.

Dependency Installation Failed?

  • Confirm that the terminal prompt still includes (.venv).
  • Confirm that requirements.txt is saved inside incident-runbook-retrieval.
  • Check that every version pin matches the reference above.

Ask for help with the installation output.

Before you run the final checks, which output will prove that the repository is initialized and the pinned packages are installed?

  • Verify the repository state plus the installed packages by running these checks:
git status
python -m pip list

What Should You See?

Git reports the state of the new repository. It lists .gitignore plus requirements.txt as untracked files.

The ignored .venv directory stays out of the Git report.

The package list includes FastAPI 0.162.0. It also includes Uvicorn 0.54.0.

The package list includes Psycopg 3.3.8. It includes psycopg-binary 3.3.8 plus Pydantic 2.14.3.

That is the workspace locked down. Your repository now has an isolated environment plus repeatable dependency pins.

Your Python workspace is ready. Next, you will start the local database plus the embedding service that powers semantic retrieval.

Start PostgreSQL and Ollama Locally

Your Python workspace is ready. The retrieval API now needs local services that can store vectors and turn incident text into embeddings.

In this step, Docker Desktop runs PostgreSQL 18 with pgvector 0.8.7. Ollama runs the EmbeddingGemma model on your Mac.

In this step, get ready to:
  • Install Docker Desktop and Ollama on your Mac.
  • Start a persistent PostgreSQL container with pgvector.
  • Download EmbeddingGemma and confirm it returns embeddings.
Install Docker Desktop and Ollama

These services run outside your Python virtual environment. Docker Desktop manages the database container. Ollama hosts the embedding model through a local HTTP API.

  • Check whether Docker Desktop is already available from the VS Code terminal by running:
docker version

What does this check do?

The command asks Docker for its client and server versions. A server response confirms that Docker Desktop is installed and running.

✔️ Docker reports client and server versions

Docker Desktop is ready to run the PostgreSQL container.

  • Leave Docker Desktop running in the background.

ⓧ Docker is unavailable or not running

Docker Desktop is free for personal learning under its current terms. Larger commercial use can require a paid subscription.

  • Open the official Docker Desktop installation guide in your browser.
  • Download the macOS disk image for your Mac.
  • Double-click Docker.dmg in your Downloads folder.
  • Drag Docker into the Applications folder.
  • Select Applications in Finder.
  • Double-click Docker to launch it.
  • Review the Docker Desktop subscription agreement.
  • Accept the agreement to finish launching Docker Desktop.
  • Check whether Ollama is already responding by running:
curl http://localhost:11434/api/version

What does this check do?

The command sends a request to Ollama's local version endpoint. A JSON response proves the Ollama application is running and accepting local requests.

✔️ I see version 0.40.14 or higher

Ollama is ready to host EmbeddingGemma.

  • Leave the Ollama application running in the background.

ⓧ I see an older version

This project uses Ollama 0.40.14. Installing the current release updates the local application.

  • Open the official Ollama macOS installation guide in your browser.
  • Download the current macOS disk image.
  • Double-click the Ollama disk image in your Downloads folder.
  • Move Ollama into the Applications folder.
  • Select Applications in Finder.
  • Double-click Ollama to launch the updated application.
  • Allow Ollama to add its CLI to your path if it asks for permission.

ⓧ Ollama is not responding

Ollama needs its macOS application running before the local API can answer requests.

  • Open the official Ollama macOS installation guide in your browser.
  • Download the current macOS disk image.
  • Double-click the Ollama disk image in your Downloads folder.
  • Move Ollama into the Applications folder.
  • Select Applications in Finder.
  • Double-click Ollama to launch it.
  • Allow Ollama to add its CLI to your path if it asks for permission.

Still having trouble with the local apps?

Confirm that both applications remain open after installation. Docker commands need the Docker engine running. Ollama requests need the Ollama application running.

If either check still fails, help me troubleshoot my Docker Desktop or Ollama installation.

Start PostgreSQL with Docker Compose

A Compose file records the database image and connection settings beside your code. This gives the project a repeatable PostgreSQL service with persistent local storage.

  • Return to the Explorer sidebar in VS Code.
  • Select the incident-runbook-retrieval folder.
  • Use the new-file control at the top of the Explorer sidebar.
  • Enter compose.yaml as the file name.
  • Paste this configuration into compose.yaml:
services:
  postgres:
    image: pgvector/pgvector:0.8.7-pg18-trixie
    environment:
      POSTGRES_USER: incident
      POSTGRES_PASSWORD: incident
      POSTGRES_DB: incident_copilot
    ports:
      - "5432:5432"
    volumes:
      - incident_data:/var/lib/postgresql

volumes:
  incident_data:

What does this configuration do?

  • The postgres service uses PostgreSQL 18 with pgvector 0.8.7 already included.
  • The environment values create the incident_copilot database with the local incident account.
  • The port mapping makes PostgreSQL available at localhost:5432.
  • The incident_data volume keeps database files after the container stops.
  • The path /var/lib/postgresql matches the persistent volume location used by the PostgreSQL 18 image.
  • Save compose.yaml.
  • Start the PostgreSQL service in the background by running:
docker compose up -d

What does this command do?

Docker Compose reads compose.yaml and creates the PostgreSQL container. The -d flag keeps the service running in the background.

On the first start, Docker may download the database image. The terminal displays download progress before it reports that the service has started.

PostgreSQL container did not start?

Confirm that Docker Desktop is still running. Check that compose.yaml is saved inside the incident-runbook-retrieval folder.

If the service still fails, help me troubleshoot my PostgreSQL Compose service.

✔️ Awesome, I've got everything!

Your Compose configuration is saved. Docker now has the instructions it needs to recreate the local database service.

ⓧ I'd like to double check the full code

Compare your complete compose.yaml file with this reference.

services:
  postgres:
    image: pgvector/pgvector:0.8.7-pg18-trixie
    environment:
      POSTGRES_USER: incident
      POSTGRES_PASSWORD: incident
      POSTGRES_DB: incident_copilot
    ports:
      - "5432:5432"
    volumes:
      - incident_data:/var/lib/postgresql

volumes:
  incident_data:
Download and test EmbeddingGemma

An embedding represents text as a list of numbers. EmbeddingGemma creates those vectors locally so the next steps can compare incident descriptions by meaning.

The model download is 622 MB. Ollama displays progress while it saves the model on your Mac.

  • Download EmbeddingGemma through Ollama by running:
ollama pull embeddinggemma

What does this command do?

Ollama downloads the official embeddinggemma model into its local model store. Once the pull completes, the local API can use that model to create embeddings.

The terminal shows download progress followed by a completion message. That confirms the model is available locally.

Model download did not complete?

Confirm that Ollama is still running. Check that your internet connection has enough time to download the 622 MB model.

If the pull still fails, help me troubleshoot the EmbeddingGemma download.

Before you send the first embedding request, what kind of result do you expect from text processed by an embedding model?

  • Send Ollama's sample request to the embedding endpoint by running:
curl http://localhost:11434/api/embed -d '{
  "model": "embeddinggemma",
  "input": "Why is the sky blue?"
}'

What happens in this request?

  • The request sends text to Ollama's local /api/embed endpoint.
  • The model field selects EmbeddingGemma.
  • The input field contains the text that Ollama converts into a vector.
  • The response places the generated vector inside the embeddings array.

You should see JSON containing an embeddings array with many decimal values. This proves that your local model can turn text into vectors.

Embedding request did not return an array?

Confirm that the Ollama application is still running. Check that the model name is exactly embeddinggemma.

If the endpoint still fails, help me debug my local Ollama embedding request.

Before the final check, do you expect both local services to respond while your Python virtual environment remains activated?

  • Verify the PostgreSQL container and Ollama version endpoint by running:
docker compose ps
curl http://localhost:11434/api/version

What should you see?

The Compose output lists the postgres service as running with port 5432 published. The Ollama request returns JSON containing its version.

You have both local services working. PostgreSQL can now store runbook vectors while Ollama creates them.

Your vector database and embedding service are ready. Next up, you will build a FastAPI keyword-search baseline over fictional incident runbooks.

Build the Keyword Retrieval Baseline

Your local PostgreSQL database is running inside its container. Ollama can already turn text into embeddings.

Now you need a FastAPI layer that accepts incident descriptions through a REST API. This gives you a result you can inspect from your browser.

A phrase-matching baseline creates an early visible win. Its strict wording requirement also gives you a baseline that semantic retrieval can improve later.

In this step, get ready to:
  • Store five fictional incident runbooks in a local dataset.
  • Build a FastAPI application with health and keyword-search endpoints.
  • Submit a literal incident phrase through the interactive documentation.
Create the incident runbook dataset

The keyword search needs operational text to inspect. A JSON file keeps each runbook title beside the response guidance for that incident.

  • In the VS Code Explorer sidebar, select the incident-runbook-retrieval repository.
  • Click the new-folder icon in the Explorer sidebar.
  • Enter data as the folder name.
  • Select the new data folder in the Explorer sidebar.
  • Click the new-file icon in the Explorer sidebar.
  • Enter runbooks.json as the file name.
  • Fill data/runbooks.json with the five incident runbooks by pasting this code:
[
  {
    "title": "PostgreSQL connection pool exhausted",
    "content": "When the application cannot acquire a database connection, inspect active sessions, close abandoned transactions, and tune the connection pool."
  },
  {
    "title": "Sustained CPU saturation",
    "content": "When compute usage remains near capacity and requests slow down, identify the busiest process, inspect recent releases, and scale or roll back safely."
  },
  {
    "title": "Disk capacity exhausted",
    "content": "When a host cannot write files because storage is full, find rapidly growing directories, rotate logs, and increase the volume only after removing avoidable data."
  },
  {
    "title": "TLS certificate expiry",
    "content": "When HTTPS clients reject a service because its certificate is expired, confirm the certificate dates, renew it, deploy the replacement, and verify the complete chain."
  },
  {
    "title": "Message queue backlog",
    "content": "When queued work grows faster than consumers process it, inspect consumer errors, compare arrival and completion rates, and scale healthy consumers."
  }
]

How is the dataset structured?

  • Each title names the incident that an operator needs to resolve.
  • Each content value contains the evidence that the API returns.
  • The five incidents give your search function distinct operational situations to inspect.
  • Save data/runbooks.json.
  • Confirm that the Explorer sidebar lists runbooks.json inside the data folder.

You should see five objects in the file. Each object should contain one title value and one content value.

Seeing JSON warnings?

Check that every object has matching braces. Confirm that a comma separates each object except the final one.

Compare every quotation mark around the title and content values.

Ask for help checking the file without changing its runbook text: help me find the JSON syntax problem in my runbook dataset.

✔️ Awesome, I've got everything!

Your runbook dataset is saved. The keyword API now has five operational passages to search.

ⓧ I'd like to double check the full code

Compare your complete data/runbooks.json file with this reference.

[
  {
    "title": "PostgreSQL connection pool exhausted",
    "content": "When the application cannot acquire a database connection, inspect active sessions, close abandoned transactions, and tune the connection pool."
  },
  {
    "title": "Sustained CPU saturation",
    "content": "When compute usage remains near capacity and requests slow down, identify the busiest process, inspect recent releases, and scale or roll back safely."
  },
  {
    "title": "Disk capacity exhausted",
    "content": "When a host cannot write files because storage is full, find rapidly growing directories, rotate logs, and increase the volume only after removing avoidable data."
  },
  {
    "title": "TLS certificate expiry",
    "content": "When HTTPS clients reject a service because its certificate is expired, confirm the certificate dates, renew it, deploy the replacement, and verify the complete chain."
  },
  {
    "title": "Message queue backlog",
    "content": "When queued work grows faster than consumers process it, inspect consumer errors, compare arrival and completion rates, and scale healthy consumers."
  }
]

How to compare this file

Check the punctuation first. Then confirm that all five titles and content values match the reference.

Build the keyword search API

The repository is already open in Visual Studio Code. You will add a Python application that loads the dataset before checking each incident phrase.

A Pydantic model defines the request body that the endpoint accepts. FastAPI uses that model to validate the query and limit before your search function runs.

  • Select the incident-runbook-retrieval repository in the Explorer sidebar.
  • Click the new-file icon in the Explorer sidebar.
  • Enter example.py as the file name.
  • Set up the application and its request schema by pasting this code into example.py:
import json

from fastapi import FastAPI
from pydantic import BaseModel

DATA_PATH = "data/runbooks.json"

app = FastAPI(title="Incident Runbook Retrieval API")


class SearchRequest(BaseModel):
    query: str
    limit: int = 3

What does this code set up?

  • The imports provide JSON loading, the web application, and request validation.
  • DATA_PATH points the application to the runbook dataset you created.
  • SearchRequest requires a query while giving each request a default result limit of 3.
  • Save example.py.
  • Switch back to the VS Code terminal from earlier.
  • Start the application shell by running this command:
uvicorn example:app

What does this command do?

Uvicorn imports the app object from example.py. It keeps the terminal busy while the local API server handles browser requests.

  • Open http://127.0.0.1:8000/docs in your browser.

You should see interactive documentation titled Incident Runbook Retrieval API.

Your FastAPI application is live. That first browser page proves the Python module loads successfully.

API server not starting?

Confirm that the terminal still shows the activated .venv environment. Check that the terminal is inside the incident-runbook-retrieval folder.

Ask for help with the startup output: help me troubleshoot why Uvicorn cannot load my FastAPI application.

  • Return to the VS Code terminal.
  • Press Ctrl+C to stop the temporary server.
  • Return to example.py in the editor.
  • Add the data-loading and phrase-matching helpers below SearchRequest by pasting this code:
def read_runbooks() -> list[dict]:
    with open(DATA_PATH, encoding="utf-8") as file:
        return json.load(file)


def normalized_limit(limit: int) -> int:
    return max(1, min(limit, 5))


def keyword_results(query: str, limit: int) -> list[dict]:
    phrase = query.lower().strip()
    matches = []
    for runbook in read_runbooks():
        searchable = f"{runbook['title']} {runbook['content']}".lower()
        if phrase in searchable:
            matches.append(
                {
                    "title": runbook["title"],
                    "content": runbook["content"],
                    "score": 1.0,
                }
            )
    return matches[: normalized_limit(limit)]

How does phrase matching work?

  • read_runbooks() loads the five JSON objects into Python.
  • normalized_limit() keeps the requested result count between 1 and 5.
  • keyword_results() converts both sides of the comparison to lowercase.
  • The matcher returns a runbook only when the complete query phrase appears in its title or content.
  • Save example.py.
  • Add the two API routes below keyword_results() by pasting this code:
@app.get("/health")
def health() -> dict:
    return {"status": "ok", "embedding_model": "not indexed yet"}


@app.post("/search/keyword")
def search_keyword(request: SearchRequest) -> dict:
    return {
        "query": request.query,
        "results": keyword_results(request.query, request.limit),
    }

What do these routes expose?

  • GET /health confirms that the API process is available.
  • POST /search/keyword accepts a validated SearchRequest body.
  • The keyword route passes the request values to keyword_results() before returning the matches.
  • Save example.py.

✔️ Awesome, I've got everything!

Your application now contains the request model, search helpers, and two API routes. Keep example.py saved for the browser test.

ⓧ I'd like to double check the full code

Compare your complete example.py file with this reference.

import json

from fastapi import FastAPI
from pydantic import BaseModel

DATA_PATH = "data/runbooks.json"

app = FastAPI(title="Incident Runbook Retrieval API")


class SearchRequest(BaseModel):
    query: str
    limit: int = 3


def read_runbooks() -> list[dict]:
    with open(DATA_PATH, encoding="utf-8") as file:
        return json.load(file)


def normalized_limit(limit: int) -> int:
    return max(1, min(limit, 5))


def keyword_results(query: str, limit: int) -> list[dict]:
    phrase = query.lower().strip()
    matches = []
    for runbook in read_runbooks():
        searchable = f"{runbook['title']} {runbook['content']}".lower()
        if phrase in searchable:
            matches.append(
                {
                    "title": runbook["title"],
                    "content": runbook["content"],
                    "score": 1.0,
                }
            )
    return matches[: normalized_limit(limit)]


@app.get("/health")
def health() -> dict:
    return {"status": "ok", "embedding_model": "not indexed yet"}


@app.post("/search/keyword")
def search_keyword(request: SearchRequest) -> dict:
    return {
        "query": request.query,
        "results": keyword_results(request.query, request.limit),
    }

How to compare this file

Check the order of each function first. Then compare indentation inside keyword_results() and search_keyword().

Run the literal keyword search

FastAPI generates interactive documentation from the routes and request model in your application. You can use that page to send a real request without building a separate client.

  • Switch back to the VS Code terminal from earlier.
  • Start the completed keyword API by running this command:
uvicorn example:app

Keep this terminal running

Uvicorn now serves both routes from example.py. Leave this terminal running so the browser can reach the API.

  • Open http://127.0.0.1:8000/docs in your browser.
  • Expand POST /search/keyword in the interactive documentation.
  • Click Try it out.

Before you send the request, which runbook do you expect the literal phrase database connection to match?

  • Replace the example request body with this literal keyword query:
{"query":"database connection","limit":3}

What does this request ask for?

The query value is the complete phrase that the matcher searches for. The limit value allows up to three matching runbooks in the response.

  • Click Execute to submit the request.

You should see a result titled PostgreSQL connection pool exhausted. Its content contains the literal phrase database connection.

The result should also contain a score of 1.0.

That is your first retrieval result working. The API can now turn a literal incident phrase into relevant operational evidence.

No runbook in the response?

Confirm that the request uses the exact lowercase phrase database connection. Check that data/runbooks.json is saved inside the repository.

Review the terminal for a file-loading or Python error. Keep the terminal open while you submit the request again.

Ask for help tracing the empty response: help me debug why my keyword endpoint cannot find the database runbook.

Your literal phrase now retrieves the matching database runbook. Next, you will reword the same incident and expose the weakness hidden inside exact phrase matching.

Test the Paraphrase Failure

Your FastAPI keyword endpoint already returns the PostgreSQL connection runbook for a literal search. That gives you a working control result.

Now you need to test whether exact phrase matching can handle the same incident with different wording. This comparison reveals whether your API follows meaning or only repeated text.

In this step, get ready to:
  • Reconfirm the known literal query as your control result.
  • Submit the same incident using paraphrased wording.
  • Document what the two responses prove about phrase matching.
Confirm the literal control result

A control query checks that the endpoint still works before you change the wording. It gives the paraphrase test a reliable result to compare against.

  • Return to http://127.0.0.1:8000/docs in the browser from earlier.
  • Expand POST /search/keyword in the interactive documentation.
  • Activate the request editor for the endpoint.
  • Enter database connection in the query field.
  • Set the limit field to 3.
  • Submit the request.

You should see PostgreSQL connection pool exhausted in the response. Your baseline is still healthy.

Test the paraphrased incident

The next query describes the same database incident with different words. It gives the phrase matcher a direct test of whether meaning alone is enough.

  • Keep POST /search/keyword expanded.
  • Replace the query value with The application cannot borrow a database session.
  • Leave the limit value set to 3.

Before you submit the request, do you think the phrase matcher can connect this wording to the database runbook?

  • Submit the paraphrased request.

You should see "results": [] in the response. You found the baseline's weakness.

Why Does the Paraphrase Fail?

  • The keyword_results() function converts the complete query into a lowercase phrase.
  • The full phrase must appear as one continuous substring inside a runbook title or its content.
  • The runbook uses cannot acquire a database connection. The paraphrase uses cannot borrow a database session.
  • Those different words produce no substring match. The endpoint therefore returns an empty results list.

Seeing a Different Result?

  • Confirm the query field contains the complete paraphrase exactly.
  • Confirm the request still targets POST /search/keyword.
  • Return to the VS Code terminal from earlier to confirm the Uvicorn process remains active.

Help me diagnose an unexpected keyword search result.

Compare the two responses

A controlled comparison changes one input at a time. Here, only the incident wording changed.

  • Compare database connection with The application cannot borrow a database session.
  • Compare the matched runbook title with the empty results list.

The literal phrase returns the relevant runbook. The paraphrase returns "results": [] even though it describes the same incident.

What Does This Prove?

The endpoint works correctly according to its current substring rule. That rule only recognizes wording already present in the runbook.

The empty response documents lost evidence caused by a wording change. Your next retrieval approach needs to compare meaning.

You now have clear evidence that the keyword baseline loses relevant runbook information when wording changes. Next, you will make retrieval follow meaning so the same paraphrase can recover the missing evidence.

Add Vector Retrieval

The empty paraphrase result exposed the keyword baseline's blind spot. Your API has useful evidence, but phrase matching cannot reach it when the wording changes.

Now you'll create embeddings with Ollama. PostgreSQL will store them through pgvector. Cosine similarity will rank runbooks by meaning.

In this step, get ready to:
  • Connect the FastAPI application to Ollama's embedding endpoint.
  • Embed the five runbooks into PostgreSQL with pgvector.
  • Return ranked semantic matches from the vector-search endpoint.
Connect the API to Ollama

An embedding represents the meaning of text as a list of numbers. Your API needs the Ollama endpoint details before it can create those vectors.

  • In example.py, replace the top section ending at DATA_PATH = "data/runbooks.json" with this code:
import json
from urllib.request import Request, urlopen

import psycopg
from fastapi import FastAPI
from pydantic import BaseModel

DATA_PATH = "data/runbooks.json"
DB_DSN = (
    "dbname=incident_copilot user=incident password=incident "
    "host=localhost port=5432"
)
OLLAMA_EMBED_URL = "http://localhost:11434/api/embed"
OLLAMA_EMBED_MODEL = "embeddinggemma"

What does this configuration do?

  • The Request and urlopen imports let Python send embedding requests to Ollama.
  • The psycopg import connects the application to PostgreSQL.
  • DB_DSN identifies the local incident database.
  • OLLAMA_EMBED_URL identifies the local embedding endpoint.
  • OLLAMA_EMBED_MODEL keeps the selected model in one reusable value.
  • Save example.py.
  • Switch back to the terminal running Uvicorn.
  • Press Ctrl+C to stop the current server.
  • Restart the API with the new configuration by running:
uvicorn example:app

What does this check prove?

Uvicorn imports the updated application before serving it. A successful start confirms that the new imports and configuration load correctly.

  • Return to http://127.0.0.1:8000/docs in your browser.
  • Run GET /health from the interactive documentation.

You'll still see embedding_model set to not indexed yet. The API is loading the new connection settings successfully.

  • In example.py, add these helpers below keyword_results():
def embed_texts(texts: list[str]) -> list[list[float]]:
    body = json.dumps(
        {"model": OLLAMA_EMBED_MODEL, "input": texts}
    ).encode("utf-8")
    request = Request(
        OLLAMA_EMBED_URL,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urlopen(request, timeout=120) as response:
        payload = json.load(response)
    return payload["embeddings"]


def vector_literal(values: list[float]) -> str:
    return "[" + ",".join(str(value) for value in values) + "]"

How do these helpers prepare vectors?

  • embed_texts() sends one or more text inputs to the local Ollama embedding endpoint.
  • The response provides each embedding inside the embeddings array.
  • vector_literal() converts each numeric list into the bracketed format accepted by pgvector.
  • Find the health() function in example.py.
  • Replace its return line with this line:
    return {"status": "ok", "embedding_model": OLLAMA_EMBED_MODEL}

Why update the health response?

The health endpoint now reports the model selected by the application. This gives you a quick runtime check for the embedding configuration.

  • Save example.py.
  • Switch back to the terminal running Uvicorn.
  • Press Ctrl+C to stop the server.
  • Restart the API with the embedding helpers by running:
uvicorn example:app

What happens when the server restarts?

Uvicorn loads the helper functions as part of example.py. The API can now prepare Ollama requests and pgvector values.

  • Refresh the interactive documentation.
  • Run GET /health again.

You'll see embedding_model set to embeddinggemma. Your API now exposes its active embedding model.

Does the API fail to restart?

Check that the new imports remain above the FastAPI import. Confirm that each configuration string matches the code block.

Make sure the terminal still shows your activated .venv environment.

Ask for help with the embedding configuration.

Index the runbooks in pgvector

Vector search needs stored evidence before it can rank anything. The indexing path creates the pgvector schema and upserts one embedding for each runbook.

  • In example.py, add this function below vector_literal():
def ensure_schema(cursor) -> None:
    cursor.execute("CREATE EXTENSION IF NOT EXISTS vector")
    cursor.execute(
        """
        CREATE TABLE IF NOT EXISTS runbooks (
            id bigserial PRIMARY KEY,
            title text UNIQUE NOT NULL,
            content text NOT NULL,
            embedding vector NOT NULL
        )
        """
    )

What does the schema contain?

  • CREATE EXTENSION IF NOT EXISTS vector enables pgvector in the incident database.
  • The runbooks table stores each title beside its content.
  • The embedding column stores the vector used for retrieval.
  • The unique title lets later indexing runs update an existing runbook.
  • Save example.py.
  • Switch back to the terminal running Uvicorn.
  • Press Ctrl+C to stop the server.
  • Check that the schema function loads by restarting the API:
uvicorn example:app

What does this restart check?

The restart checks the new function's Python syntax. The database schema is created later when the indexing endpoint calls it.

  • Run GET /health from the interactive documentation.

You'll receive the healthy response with embeddinggemma still selected. The schema helper is loaded without disrupting the API.

Does the schema helper stop the server?

Check the indentation around both cursor.execute() calls. Confirm that the SQL string has matching triple quotes.

Ask for help checking the schema function.

  • In example.py, add this function below ensure_schema():
def index_all_runbooks() -> int:
    runbooks = read_runbooks()
    texts = [
        f"{runbook['title']}\n{runbook['content']}"
        for runbook in runbooks
    ]
    embeddings = embed_texts(texts)

    with psycopg.connect(DB_DSN) as connection:
        with connection.cursor() as cursor:
            ensure_schema(cursor)
            for runbook, embedding in zip(runbooks, embeddings):
                cursor.execute(
                    """
                    INSERT INTO runbooks (title, content, embedding)
                    VALUES (%s, %s, %s::vector)
                    ON CONFLICT (title) DO UPDATE SET
                        content = EXCLUDED.content,
                        embedding = EXCLUDED.embedding
                    """,
                    (
                        runbook["title"],
                        runbook["content"],
                        vector_literal(embedding),
                    ),
                )
    return len(runbooks)

How does indexing work?

  • The function combines each runbook title with its operational content before embedding it.
  • embed_texts() sends all five combined texts to Ollama in one request.
  • ensure_schema() prepares PostgreSQL before any rows are written.
  • ON CONFLICT updates existing runbooks when you index them again.
  • Add this endpoint below search_keyword() near the bottom of example.py:
@app.post("/index")
def index_runbooks() -> dict:
    indexed = index_all_runbooks()
    return {"indexed": indexed, "embedding_model": OLLAMA_EMBED_MODEL}

What does the index endpoint return?

The POST /index endpoint runs the complete indexing process. Its response reports how many runbooks were stored and which embedding model created their vectors.

  • Save example.py.
  • Switch back to the terminal running Uvicorn.
  • Press Ctrl+C to stop the server.
  • Start the API with the indexing endpoint by running:
uvicorn example:app

What happens during this start?

Uvicorn registers the new indexing endpoint in the FastAPI application. The database work begins only when you run that endpoint.

  • Refresh the interactive documentation.
  • Expand POST /index.

Before you run the endpoint, how many runbooks do you expect it to index?

  • Run POST /index from the interactive documentation.

The first embedding request can take a little longer while Ollama prepares the model. A short wait here is expected.

You'll see indexed set to 5. You'll also see embedding_model set to embeddinggemma.

That's the storage layer working. All five runbooks now have embeddings in PostgreSQL.

Does indexing fail?

Confirm that the PostgreSQL container is still running. Confirm that the Ollama application is still active.

Check that data/runbooks.json remains inside the incident-runbook-retrieval folder.

Ask for help tracing the indexing request.

Search by meaning

The stored vectors give the paraphrase a new path to the evidence. The API can embed the query and rank every runbook by cosine similarity.

  • In example.py, add this function below index_all_runbooks():
def vector_results(query: str, limit: int) -> list[dict]:
    query_vector = vector_literal(embed_texts([query])[0])
    result_limit = normalized_limit(limit)

    with psycopg.connect(DB_DSN) as connection:
        with connection.cursor() as cursor:
            ensure_schema(cursor)
            cursor.execute(
                """
                SELECT title, content,
                       1 - (embedding <=> %s::vector) AS score
                FROM runbooks
                ORDER BY embedding <=> %s::vector
                LIMIT %s
                """,
                (query_vector, query_vector, result_limit),
            )
            rows = cursor.fetchall()

    return [
        {
            "title": title,
            "content": content,
            "score": round(float(score), 4),
        }
        for title, content, score in rows
    ]

How does vector ranking work?

  • The query receives an embedding from the same model used for the runbooks.
  • The <=> operator calculates cosine distance between the query and each stored embedding.
  • Subtracting that distance from 1 produces the similarity score returned by the API.
  • The SQL query orders smaller cosine distances first so the closest meaning ranks highest.
  • Add this endpoint below index_runbooks() at the bottom of example.py:
@app.post("/search/vector")
def search_vector(request: SearchRequest) -> dict:
    return {
        "query": request.query,
        "results": vector_results(request.query, request.limit),
    }

What does the vector endpoint add?

The POST /search/vector endpoint accepts the same validated request shape as keyword search. Its results come from semantic ranking in PostgreSQL.

  • Save example.py.
  • Switch back to the terminal running Uvicorn.
  • Press Ctrl+C to stop the server.
  • Start the complete API by running:
uvicorn example:app

What is available now?

The running application now exposes keyword search and vector search. It also keeps the indexing endpoint available for refreshing stored runbooks.

  • Refresh http://127.0.0.1:8000/docs.
  • Expand POST /search/vector.
  • Set the request body by pasting:
{
  "query": "The application cannot borrow a database session",
  "limit": 3
}

Why reuse this request?

This is the same paraphrase that produced an empty keyword result. Reusing it makes the comparison between phrase matching and semantic retrieval visible.

Before you run the vector search, do you think the paraphrase will recover any runbook evidence?

  • Run POST /search/vector from the interactive documentation.

You'll see a non-empty results list. Each result contains a title, operational content, and a numeric score.

You fixed the failure you observed earlier. The same paraphrase now retrieves ranked runbook evidence by meaning.

✔️ Awesome, I've got everything!

Great work. Save example.py and leave Uvicorn running while you review the result.

ⓧ I'd like to double check the full code

Compare your complete example.py with this version.

import json
from urllib.request import Request, urlopen

import psycopg
from fastapi import FastAPI
from pydantic import BaseModel

DATA_PATH = "data/runbooks.json"
DB_DSN = (
    "dbname=incident_copilot user=incident password=incident "
    "host=localhost port=5432"
)
OLLAMA_EMBED_URL = "http://localhost:11434/api/embed"
OLLAMA_EMBED_MODEL = "embeddinggemma"

app = FastAPI(title="Incident Runbook Retrieval API")


class SearchRequest(BaseModel):
    query: str
    limit: int = 3


def read_runbooks() -> list[dict]:
    with open(DATA_PATH, encoding="utf-8") as file:
        return json.load(file)


def normalized_limit(limit: int) -> int:
    return max(1, min(limit, 5))


def keyword_results(query: str, limit: int) -> list[dict]:
    phrase = query.lower().strip()
    matches = []
    for runbook in read_runbooks():
        searchable = f"{runbook['title']} {runbook['content']}".lower()
        if phrase in searchable:
            matches.append(
                {
                    "title": runbook["title"],
                    "content": runbook["content"],
                    "score": 1.0,
                }
            )
    return matches[: normalized_limit(limit)]


def embed_texts(texts: list[str]) -> list[list[float]]:
    body = json.dumps(
        {"model": OLLAMA_EMBED_MODEL, "input": texts}
    ).encode("utf-8")
    request = Request(
        OLLAMA_EMBED_URL,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urlopen(request, timeout=120) as response:
        payload = json.load(response)
    return payload["embeddings"]


def vector_literal(values: list[float]) -> str:
    return "[" + ",".join(str(value) for value in values) + "]"


def ensure_schema(cursor) -> None:
    cursor.execute("CREATE EXTENSION IF NOT EXISTS vector")
    cursor.execute(
        """
        CREATE TABLE IF NOT EXISTS runbooks (
            id bigserial PRIMARY KEY,
            title text UNIQUE NOT NULL,
            content text NOT NULL,
            embedding vector NOT NULL
        )
        """
    )


def index_all_runbooks() -> int:
    runbooks = read_runbooks()
    texts = [
        f"{runbook['title']}\n{runbook['content']}"
        for runbook in runbooks
    ]
    embeddings = embed_texts(texts)

    with psycopg.connect(DB_DSN) as connection:
        with connection.cursor() as cursor:
            ensure_schema(cursor)
            for runbook, embedding in zip(runbooks, embeddings):
                cursor.execute(
                    """
                    INSERT INTO runbooks (title, content, embedding)
                    VALUES (%s, %s, %s::vector)
                    ON CONFLICT (title) DO UPDATE SET
                        content = EXCLUDED.content,
                        embedding = EXCLUDED.embedding
                    """,
                    (
                        runbook["title"],
                        runbook["content"],
                        vector_literal(embedding),
                    ),
                )
    return len(runbooks)


def vector_results(query: str, limit: int) -> list[dict]:
    query_vector = vector_literal(embed_texts([query])[0])
    result_limit = normalized_limit(limit)

    with psycopg.connect(DB_DSN) as connection:
        with connection.cursor() as cursor:
            ensure_schema(cursor)
            cursor.execute(
                """
                SELECT title, content,
                       1 - (embedding <=> %s::vector) AS score
                FROM runbooks
                ORDER BY embedding <=> %s::vector
                LIMIT %s
                """,
                (query_vector, query_vector, result_limit),
            )
            rows = cursor.fetchall()

    return [
        {
            "title": title,
            "content": content,
            "score": round(float(score), 4),
        }
        for title, content, score in rows
    ]


@app.get("/health")
def health() -> dict:
    return {"status": "ok", "embedding_model": OLLAMA_EMBED_MODEL}


@app.post("/search/keyword")
def search_keyword(request: SearchRequest) -> dict:
    return {
        "query": request.query,
        "results": keyword_results(request.query, request.limit),
    }


@app.post("/index")
def index_runbooks() -> dict:
    indexed = index_all_runbooks()
    return {"indexed": indexed, "embedding_model": OLLAMA_EMBED_MODEL}


@app.post("/search/vector")
def search_vector(request: SearchRequest) -> dict:
    return {
        "query": request.query,
        "results": vector_results(request.query, request.limit),
    }

Does vector search return an error or no results?

Run POST /index again to confirm the runbooks exist in PostgreSQL. Keep Ollama active because every search request needs a fresh query embedding.

Check that both uses of query_vector appear in the SQL parameters. Confirm that vector_results() calls embed_texts() before connecting to the database.

Ask for help tracing the vector query.

Secret mission

Measure Hit@1 Retrieval Quality

Your vector search already recovers one paraphrased incident. This mission turns four labeled incidents into a repeatable benchmark that exposes first-place hits and misses through Hit@1.

Clean Up Your Resources

Clean Up Your Resources

This local project has no ongoing service charges for personal learning. Decide whether to keep your resources running, pause them, or delete them entirely.

Resources you used:

  • A running FastAPI retrieval API in your terminal.
  • A pgvector-enabled PostgreSQL container with the incident_data volume and indexed runbooks table.
  • A local Ollama service with the EmbeddingGemma model stored on your Mac.
  • The incident-runbook-retrieval folder containing your Git history, virtual environment, API code, runbook data, and retrieval benchmark.

Keep everything running

No action is needed. Choose this if you want to keep testing the API or improving your retrieval benchmark.

  • Your FastAPI retrieval API remains available while its terminal process keeps running.
  • Your PostgreSQL container keeps the indexed runbooks available in the incident_data volume.
  • Your Ollama service keeps embeddinggemma ready for future embedding requests.
  • Your project files, virtual environment, and Git history remain available.

Pause - I'll come back to this later

Stop the running processes to free up memory. Your project files, database volume, model, and Git history stay on your Mac.

  • Switch back to the terminal where FastAPI is running.
  • Press Ctrl+C to stop the API.
  • Switch back to the activated terminal inside incident-runbook-retrieval.
  • Stop the PostgreSQL container by running this command:
docker compose stop

What Does This Command Do?

The command stops the PostgreSQL container. It preserves the incident_data volume with your indexed runbooks.

  • Quit the Ollama application.

Your local services are paused. The database volume and downloaded model remain ready for your next session.

Delete - I don't want to use this again

Remove every project-specific resource and start fresh. These actions permanently delete your indexed database, downloaded model, code, virtual environment, and Git history.

  • Switch back to the terminal where FastAPI is running.
  • Press Ctrl+C to stop the API.
  • Switch back to the activated terminal inside incident-runbook-retrieval.
  • Remove the PostgreSQL container and named volume by running this command:
docker compose down -v

What Does This Command Remove?

The command removes the PostgreSQL container and the incident_data named volume. The indexed runbooks table disappears with that volume.

  • Remove the local embedding model by running this command:
ollama rm embeddinggemma

What Does This Command Remove?

The command removes EmbeddingGemma from Ollama's local model storage. The Ollama application remains installed.

  • Quit the Ollama application.
  • Move outside the project folder and delete it by running these commands:
cd ~/Desktop
rm -rf incident-runbook-retrieval

What Do These Commands Remove?

The first command moves your terminal to the Desktop. The second command removes the entire incident-runbook-retrieval folder.

Your source files, .venv environment, runbook data, benchmark, and Git history are removed with the folder.

Docker Desktop and Ollama remain installed because this cleanup removes only the resources created for this project.

Nice Work!

Nice Work!

Project complete! Your local incident runbook retrieval API now turns paraphrased incident descriptions into ranked operational evidence.

You've learned how to:

  • Build a FastAPI service that accepts validated JSON bodies through REST endpoints. Inspect each route through browser-based interactive documentation.
  • Prove the limits of phrase matching by running a successful literal search. Re-run the same endpoint with a paraphrase to expose the empty result.
  • Generate local embeddings with Ollama. Store them beside runbook text in PostgreSQL. Rank relevant evidence with pgvector cosine distance.
  • Secret Mission: Create a labeled Hit@1 benchmark in evaluate.py. Use four paraphrased queries to print individual PASS or FAIL outcomes. Calculate a repeatable retrieval-quality score.

Ready to quiz yourself?