Build and Evaluate the Retrieval Core

Build and benchmark keyword, semantic, and hybrid incident runbook search.

Introduction

30 Second Summary

When a service fails at 2 a.m., the right troubleshooting guide only helps if someone can find it from a rushed description. Different wording can send a basic search toward the wrong fix.

In this project, you will build an evaluated retrieval core for an AI Incident Response Assistant. The finished tool compares a TF-IDF keyword baseline with dense embeddings across six labeled incidents.

What You'll Build

Picture typing a paraphrased database incident into your terminal to see semantic search rank the correct runbook first while keyword search points to the wrong one.

By the end of this project, you'll have:

  • A deliberate keyword-search failure that shows why paraphrased database incidents can defeat exact-term matching.
  • A side-by-side retrieval demo that recovers the relevant runbook from a natural-language incident description.
  • A reproducible benchmark report showing Top-1 accuracy plus average query latency across six labeled incidents.
  • Secret Mission: A weighted hybrid retriever that combines lexical scores with semantic scores for a three-way benchmark.

Are there any prerequisites?

There are no prerequisites beyond a Windows computer. The guide includes setup for every required tool.

Before We Start

Before the hands-on work begins, commit to building an evaluated incident-runbook retriever. You will use it to demonstrate retrieval engineering skills for AI engineering roles.

Set Up the Windows Retrieval Lab

A retrieval benchmark is only trustworthy when its runtime can be recreated. Uncontrolled Python or package versions can change what your code does.

Visual Studio Code will hold the lab workspace. Its Microsoft Python extension will connect Python 3.12.10 to an isolated virtual environment for this project.

In this step, get ready to:
  • Confirm the Windows development tools are available.
  • Create an isolated .venv inside the lab workspace.
  • Install the pinned retrieval packages from requirements.txt.
Install the Windows tools

Python runs the project code. Visual Studio Code provides the workspace controls that create the project environment.

  • Open Windows Command Prompt from Windows search.
  • Check the installed Python version by running this command:
py -3 --version

What does this command do?

  • The py launcher finds Python installations registered with Windows.
  • The -3 option selects an installed Python 3 interpreter.
  • The --version option prints the selected interpreter version.

✔️ I see the required Python version

Your output shows Python 3.12.10. The interpreter is ready for the project environment.

ⓧ I see another Python version

This project uses Python 3.12.10 so both pinned packages use a compatible interpreter.

  • Open the official Python 3.12.10 release page.
  • Select Windows installer (64-bit) from the files list.
  • Run the downloaded installer.
  • Allow the installer to make changes if Windows asks.
  • Complete the installer using its default selections.
  • Close the existing Command Prompt window.
  • Open a new Windows Command Prompt from Windows search.

Run the same version check again if Python 3.12.10 replaced an older installation. If another installed version remains the default, you will select Python 3.12.10 explicitly when creating .venv.

ⓧ No Python version appears

Python needs to be installed before Visual Studio Code can create the project environment.

  • Open the official Python 3.12.10 release page.
  • Select Windows installer (64-bit) from the files list.
  • Run the downloaded installer.
  • Allow the installer to make changes if Windows asks.
  • Complete the installer using its default selections.
  • Close the existing Command Prompt window.
  • Open a new Windows Command Prompt from Windows search.
  • Run the same version check again.

You should now see Python 3.12.10 in the output.

The editor is the next tool to check. Windows search gives you a quick way to confirm whether Visual Studio Code is already installed.

  • Search Windows for Visual Studio Code.

✔️ Visual Studio Code appears

  • Open Visual Studio Code from the search result.

You should see the Visual Studio Code welcome screen.

ⓧ Visual Studio Code is missing

Visual Studio Code supports Windows 10 or 11 on a 64-bit computer.

  • Open the official Visual Studio Code download page.
  • Download Visual Studio Code for Windows.
  • Run the downloaded installer.
  • Allow the installer to make changes if Windows asks.
  • Complete the installer using its default selections.
  • Search Windows for Visual Studio Code.
  • Open Visual Studio Code from the search result.

You should now see the Visual Studio Code welcome screen.

The Python extension adds interpreter selection and environment management to Visual Studio Code. Its publisher must be Microsoft so you install the intended extension.

  • Press Ctrl+Shift+X to open the Extensions view.
  • Enter Python in the extension search box.
  • Select the Python extension published by Microsoft.

✔️ I see the Manage gear button

The Manage gear button confirms the Microsoft Python extension is installed.

ⓧ I see the Install button

  • Select the Install button.

The button changes to the Manage gear button when installation completes.

That is the Windows toolchain sorted. Visual Studio Code can now use your pinned Python interpreter.

Create the workspace and virtual environment

A workspace gives every project file one named home. You will place this workspace on your Desktop so it stays easy to find.

  • Select File from the Visual Studio Code menu bar.
  • Select Open Folder....
  • Choose your Desktop as the folder location.
  • Select New Folder.
  • Enter incident-retrieval-lab as the folder name.
  • Select Select Folder.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

The Explorer view should show incident-retrieval-lab as the open workspace.

What is a virtual environment?

A virtual environment keeps this project's Python packages separate from packages installed elsewhere on your computer. The .venv folder becomes the controlled runtime for every script in this lab.

  • Press Ctrl+Shift+P to open the Command Palette.
  • Enter Python: Create Environment in the search field.
  • Select Python: Create Environment.
  • Select Venv as the environment type.
  • Select the Python 3.12.10 interpreter.

Visual Studio Code shows creation progress. The .venv folder should appear under incident-retrieval-lab when the environment is ready.

  • Select Terminal from the menu bar.
  • Select New Terminal.
  • Select the Launch Profile dropdown in the terminal panel.
  • Select Command Prompt.

You should see a Command Prompt terminal whose prompt starts with (.venv). This prefix confirms that commands now use the project environment.

Missing the activated environment?

  • Open the Command Palette again.
  • Select Python: Select Interpreter.
  • Choose the interpreter inside .venv.
  • Create another terminal with Terminal followed by New Terminal.

If the prefix is still missing, help me activate the .venv environment in Visual Studio Code on Windows.

Pin and install the retrieval packages

A requirements file records the exact package versions that define the lab. Exact pins keep future installations aligned with the environment you test today.

  • Select the New File button in the Explorer toolbar.
  • Enter requirements.txt as the file name.
  • Add both dependency pins by pasting this content:
sentence-transformers==6.2.0
scikit-learn==1.9.1

What do these package pins provide?

  • Sentence Transformers 6.2.0 provides the local embedding model interface used for semantic retrieval later in the project.
  • scikit-learn 1.9.1 provides the TF-IDF vectorizer and cosine similarity calculation used by the keyword baseline.
  • The double equals signs require the listed version instead of accepting any available release.
  • Save requirements.txt with Ctrl+S.
  • Confirm the Explorer lists requirements.txt inside incident-retrieval-lab.

Requirements file missing?

Confirm you created the file under the incident-retrieval-lab workspace. Check that its name is exactly requirements.txt.

If the file still does not appear, help me create requirements.txt in my Visual Studio Code workspace.

✔️ Awesome, I've got everything!

Your saved requirements.txt now contains both exact package pins.

ⓧ I'd like to double check the full code

  • Compare your saved requirements.txt with this complete file:
sentence-transformers==6.2.0
scikit-learn==1.9.1

What should match?

The package names and version pins must match exactly. This file is the complete project artifact for this step.

This installation can take several minutes while larger dependencies download.

Before you run it, where do you expect these packages to be installed?

  • Install the pinned dependencies by running this command:
python -m pip install -r requirements.txt

What does this command do?

The command runs pip through the active Python interpreter. It reads each exact dependency pin from requirements.txt.

The public embedding model is not downloaded during this installation. Its local cache appears only when the first semantic search runs later.

The terminal should return to the (.venv) prompt without an installation error.

That is the lab foundation secured. Both retrieval libraries now live inside the project environment.

Package installation failed?

  • Confirm the terminal prompt starts with (.venv).
  • Confirm requirements.txt contains both package pins exactly.
  • Retry the command after confirming your internet connection is available.

For help with the failure, help me debug the requirements installation in my Windows virtual environment.

Your reproducible retrieval lab is ready. Next, you will build a keyword baseline and watch a paraphrased incident expose its limitation.

Make Keyword Search Fail

Your isolated Python environment is ready. The incident-retrieval-lab workspace can now support a controlled retrieval experiment.

Exact-term retrieval looks useful until a user describes the right incident with different words. You will build a TF-IDF baseline that exposes this weakness with a paraphrased database incident.

In this step, get ready to:
  • Define six incident runbooks with six labeled evaluation queries.
  • Build a keyword retriever with TF-IDF scoring.
  • Run a paraphrased query that the keyword retriever ranks incorrectly.
Define the runbooks and evaluation cases

A retrieval test needs a fixed collection of documents. Stable runbook IDs let you compare each predicted result with the expected result.

  • Use the file creation icon at the top of Visual Studio Code's file sidebar.
  • Enter runbooks.py as the file name.
  • Create the first half of RUNBOOKS by pasting this code:
RUNBOOKS = [
    {
        "id": "memory-pressure",
        "title": "Memory Pressure",
        "content": (
            "A service becomes progressively slower as memory usage rises. "
            "Capture a heap profile, inspect retained objects, confirm whether a leak exists, "
            "and restart only as a temporary recovery action."
        ),
    },
    {
        "id": "database-connections",
        "title": "Database Connection Pool Exhaustion",
        "content": (
            "The database connection pool is exhausted. Application workers cannot acquire "
            "a connection because all pooled connections are in use. Check for leaked "
            "connections, long transactions, and an undersized pool."
        ),
    },
    {
        "id": "certificate-expiry",
        "title": "Expired TLS Certificate",
        "content": (
            "Clients reject HTTPS connections because the TLS certificate has expired. "
            "Renew the certificate, deploy the replacement, and verify the full certificate chain."
        ),
    },
]

What does this data represent?

  • RUNBOOKS holds the searchable incident documents.
  • Each id provides a stable value for automated evaluation.
  • Each title names the incident pattern.
  • Each content value contains the response guidance that retrieval ranks.
  • Save runbooks.py.
  • Confirm runbooks.py appears in Visual Studio Code's file sidebar.

File missing from the sidebar?

Check that you created runbooks.py inside the open incident-retrieval-lab workspace. A file created outside that folder does not appear in this workspace.

Help me find a Python file that is missing from my Visual Studio Code workspace.

  • In runbooks.py find the closing line shown below:
]

Why locate this bracket?

This bracket currently closes the RUNBOOKS list. Replacing it lets you add the remaining documents inside the same collection.

  • Replace that closing line with the remaining runbooks plus a new closing bracket:
    {
        "id": "disk-space",
        "title": "Disk Space Exhaustion",
        "content": (
            "The host disk is full and applications cannot write files. Identify large logs "
            "or temporary files, recover space safely, and add retention or capacity alerts."
        ),
    },
    {
        "id": "high-cpu",
        "title": "High CPU Saturation",
        "content": (
            "CPU utilization remains near capacity and request latency increases. Profile hot "
            "code paths, inspect runaway workers, and scale only after identifying the cause."
        ),
    },
    {
        "id": "authentication-failures",
        "title": "Authentication Failures After Key Rotation",
        "content": (
            "Users receive 401 responses after token-signing keys rotate. Confirm the active key "
            "identifier, refresh cached public keys, and verify issuer and audience configuration."
        ),
    },
]

What did this add?

The collection now covers disk exhaustion, CPU saturation, and authentication failures. These entries complete the six-runbook search corpus.

You should now see six id fields inside RUNBOOKS.

A labeled evaluation case pairs a natural-language query with the runbook ID that should rank first. Paraphrased wording makes the evaluation test retrieval quality beyond exact phrase matching.

  • Add EVAL_CASES below RUNBOOKS by pasting this code:
EVAL_CASES = [
    {
        "query": "The backend cannot reach its data store because every client slot is occupied.",
        "expected_id": "database-connections",
    },
    {
        "query": "Browsers reject the site because its secure identity is no longer valid.",
        "expected_id": "certificate-expiry",
    },
    {
        "query": "The service gets slower over time and a restart temporarily fixes it.",
        "expected_id": "memory-pressure",
    },
    {
        "query": "The host cannot create files because storage is completely full.",
        "expected_id": "disk-space",
    },
    {
        "query": "Sign-in calls started returning unauthorized responses after credentials rotated.",
        "expected_id": "authentication-failures",
    },
    {
        "query": "Workers are consuming all processor capacity and requests are timing out.",
        "expected_id": "high-cpu",
    },
]

How do these cases work?

  • query contains the incident description submitted to a retriever.
  • expected_id identifies the runbook that should rank first.
  • The six cases cover every runbook once. This creates a balanced evaluation set for the later benchmark.
  • Save runbooks.py.
  • Open the Command Palette from Visual Studio Code's top menu.
  • Choose Python: Run Python File in Terminal.

The terminal returns to the active (.venv) prompt without a traceback. This confirms Python can parse all six runbooks plus all six evaluation cases.

Seeing a syntax problem?

Check that every runbook dictionary ends with a comma. Confirm that RUNBOOKS closes before EVAL_CASES begins.

Help me fix the syntax in my runbooks.py data file.

Build the keyword retriever

A keyword retriever converts document text into weighted term features. Cosine similarity then measures how closely a query vector points toward each document vector.

Why start with a keyword baseline?

A lexical baseline gives you a simple reference point for later comparisons. Its mistakes reveal which queries depend on shared words.

The baseline also makes future improvements measurable. Evidence from the same corpus carries more weight than an isolated demo.

  • Use the file creation icon at the top of Visual Studio Code's file sidebar.
  • Enter retrievers.py as the file name.
  • Create the shared retrieval foundation by pasting this code:
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"


def document_texts(documents):
    return [f"{document['title']}. {document['content']}" for document in documents]


def rank_results(documents, scores, top_k):
    ranked = sorted(
        zip(documents, scores),
        key=lambda item: item[1],
        reverse=True,
    )
    return [
        {**document, "score": float(score)}
        for document, score in ranked[:top_k]
    ]

What does this foundation do?

  • document_texts() combines each title with its response guidance. The retriever can score both fields as one document.
  • rank_results() sorts documents from the highest score to the lowest score.
  • top_k limits the returned list to the requested number of results.
  • MODEL_ID records the public model used in the next step. This code does not construct the model.
  • Save retrievers.py.
  • Confirm retrievers.py appears in Visual Studio Code's file sidebar.

Seeing an import warning?

Confirm the Visual Studio Code terminal still starts with (.venv). The selected interpreter must point to the project environment that contains the pinned packages.

Help me fix Python imports in my active virtual environment.

The KeywordRetriever learns a vocabulary from the runbooks. Every search transforms the query with that same vocabulary before comparing scores.

  • Add KeywordRetriever below rank_results() by pasting this code:
class KeywordRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.vectorizer = TfidfVectorizer(stop_words="english")
        self.document_matrix = self.vectorizer.fit_transform(document_texts(documents))

    def search(self, query, top_k=3):
        query_vector = self.vectorizer.transform([query])
        scores = cosine_similarity(query_vector, self.document_matrix)[0]
        return rank_results(self.documents, scores, top_k)

How does keyword ranking work?

  • TfidfVectorizer comes from scikit-learn. It turns the runbook text into weighted term features.
  • fit_transform() learns the corpus vocabulary. It also creates the document matrix.
  • transform() represents each query with the learned vocabulary.
  • cosine_similarity() produces one lexical score for every runbook.
  • Save retrievers.py.
  • Open the Command Palette from Visual Studio Code's top menu.
  • Choose Python: Run Python File in Terminal.

The terminal returns to (.venv) without a traceback. This confirms the imports plus the retriever definitions load successfully.

Retriever file fails to load?

Check the indentation inside both __init__() and search(). Confirm KeywordRetriever starts at the far-left edge of the file.

Help me debug my KeywordRetriever definition.

Run the failing baseline

The demo asks for one result from the paraphrased database incident. It prints the expected ID beside the ID selected by the keyword retriever.

  • Use the file creation icon at the top of Visual Studio Code's file sidebar.
  • Enter baseline_demo.py as the file name.
  • Create the baseline comparison by pasting this code:
from retrievers import KeywordRetriever
from runbooks import RUNBOOKS

QUERY = "The backend cannot reach its data store because every client slot is occupied."
EXPECTED_ID = "database-connections"

retriever = KeywordRetriever(RUNBOOKS)
result = retriever.search(QUERY, top_k=1)[0]

print(f"Query: {QUERY}")
print(f"Expected: {EXPECTED_ID}")
print(f"Retrieved: {result['id']}")
print(f"Score: {result['score']:.3f}")

What does this demo test?

  • QUERY describes an exhausted database connection pool with different wording.
  • EXPECTED_ID records the correct runbook ID.
  • top_k=1 keeps only the highest-ranked result.
  • The final print statements expose the prediction plus its score.
  • Save baseline_demo.py.

Before you run the demo, which runbook do you think an exact-term retriever selects from this paraphrased query?

  • Run the baseline in the active (.venv) terminal with this command:
python baseline_demo.py

What does this command do?

Python loads the six runbooks. It creates the keyword retriever before searching for the single highest-ranked result.

You will see Expected: database-connections followed by Retrieved: memory-pressure. The printed score is Score: 0.000.

Why did the baseline fail?

The paraphrased query shares no useful vocabulary with the database runbook after English stop words are removed. Every runbook receives the same zero lexical score.

The stable ranking keeps the first runbook when those scores tie. memory-pressure therefore becomes the incorrect Top-1 result.

That failure is the evidence you needed. Your baseline runs correctly while exposing the exact weakness this project is designed to measure.

Seeing a different result or an error?

Confirm QUERY matches the paraphrased incident exactly. Check that memory-pressure remains the first entry in RUNBOOKS.

If Python cannot import a project file, confirm baseline_demo.py, retrievers.py, and runbooks.py appear together inside incident-retrieval-lab.

Help me debug the failing keyword baseline.

✔️ Awesome, I've got everything!

Your three Python files are saved. The baseline now reproduces the intended keyword retrieval failure.

ⓧ I'd like to double check the full code

  • Compare each saved file with the complete references below.
sentence-transformers==6.2.0
scikit-learn==1.9.1

What should match?

The requirements file keeps both retrieval packages pinned. This file remains unchanged from the setup step.

RUNBOOKS = [
    {
        "id": "memory-pressure",
        "title": "Memory Pressure",
        "content": (
            "A service becomes progressively slower as memory usage rises. "
            "Capture a heap profile, inspect retained objects, confirm whether a leak exists, "
            "and restart only as a temporary recovery action."
        ),
    },
    {
        "id": "database-connections",
        "title": "Database Connection Pool Exhaustion",
        "content": (
            "The database connection pool is exhausted. Application workers cannot acquire "
            "a connection because all pooled connections are in use. Check for leaked "
            "connections, long transactions, and an undersized pool."
        ),
    },
    {
        "id": "certificate-expiry",
        "title": "Expired TLS Certificate",
        "content": (
            "Clients reject HTTPS connections because the TLS certificate has expired. "
            "Renew the certificate, deploy the replacement, and verify the full certificate chain."
        ),
    },
    {
        "id": "disk-space",
        "title": "Disk Space Exhaustion",
        "content": (
            "The host disk is full and applications cannot write files. Identify large logs "
            "or temporary files, recover space safely, and add retention or capacity alerts."
        ),
    },
    {
        "id": "high-cpu",
        "title": "High CPU Saturation",
        "content": (
            "CPU utilization remains near capacity and request latency increases. Profile hot "
            "code paths, inspect runaway workers, and scale only after identifying the cause."
        ),
    },
    {
        "id": "authentication-failures",
        "title": "Authentication Failures After Key Rotation",
        "content": (
            "Users receive 401 responses after token-signing keys rotate. Confirm the active key "
            "identifier, refresh cached public keys, and verify issuer and audience configuration."
        ),
    },
]

EVAL_CASES = [
    {
        "query": "The backend cannot reach its data store because every client slot is occupied.",
        "expected_id": "database-connections",
    },
    {
        "query": "Browsers reject the site because its secure identity is no longer valid.",
        "expected_id": "certificate-expiry",
    },
    {
        "query": "The service gets slower over time and a restart temporarily fixes it.",
        "expected_id": "memory-pressure",
    },
    {
        "query": "The host cannot create files because storage is completely full.",
        "expected_id": "disk-space",
    },
    {
        "query": "Sign-in calls started returning unauthorized responses after credentials rotated.",
        "expected_id": "authentication-failures",
    },
    {
        "query": "Workers are consuming all processor capacity and requests are timing out.",
        "expected_id": "high-cpu",
    },
]

What should match?

The file contains six runbooks in a stable order. It also contains six labeled paraphrase queries.

from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"


def document_texts(documents):
    return [f"{document['title']}. {document['content']}" for document in documents]


def rank_results(documents, scores, top_k):
    ranked = sorted(
        zip(documents, scores),
        key=lambda item: item[1],
        reverse=True,
    )
    return [
        {**document, "score": float(score)}
        for document, score in ranked[:top_k]
    ]


class KeywordRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.vectorizer = TfidfVectorizer(stop_words="english")
        self.document_matrix = self.vectorizer.fit_transform(document_texts(documents))

    def search(self, query, top_k=3):
        query_vector = self.vectorizer.transform([query])
        scores = cosine_similarity(query_vector, self.document_matrix)[0]
        return rank_results(self.documents, scores, top_k)

What should match?

The shared helpers prepare document text plus ranked results. KeywordRetriever provides the only active retrieval method in this step.

from retrievers import KeywordRetriever
from runbooks import RUNBOOKS

QUERY = "The backend cannot reach its data store because every client slot is occupied."
EXPECTED_ID = "database-connections"

retriever = KeywordRetriever(RUNBOOKS)
result = retriever.search(QUERY, top_k=1)[0]

print(f"Query: {QUERY}")
print(f"Expected: {EXPECTED_ID}")
print(f"Retrieved: {result['id']}")
print(f"Score: {result['score']:.3f}")

What should match?

The demo uses the paraphrased database incident. It requests one result before printing the expected ID plus the retrieved ID.

You have proved the lexical gap with a reproducible failure. Next, you will add dense embeddings and test whether meaning can recover the correct runbook.

Recover Meaning with Embeddings

Your keyword baseline exposed the exact weakness this step solves. The paraphrased database incident shared too little vocabulary with the correct runbook.

Dense embeddings represent meaning as vectors. You will use Sentence Transformers to recover the relevant runbook from different wording.

In this step, get ready to:
  • Extend the existing TF-IDF baseline with a semantic retriever.
  • Build an interactive comparison that displays three ranked results from each retriever.
  • Verify that semantic retrieval recovers the database runbook from the paraphrased incident.
Add the semantic retriever

Semantic search compares the meaning of the query with the meaning of each runbook. Normalized vectors make those comparisons consistent across the corpus.

  • Select retrievers.py from the file list in Visual Studio Code.
  • Scroll below the existing KeywordRetriever class.
  • Add the semantic retriever by pasting this class at the end of the file:
class SemanticRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.model = SentenceTransformer(MODEL_ID)
        self.document_embeddings = self.model.encode(
            document_texts(documents),
            normalize_embeddings=True,
        )

    def search(self, query, top_k=3):
        query_embedding = self.model.encode(
            [query],
            normalize_embeddings=True,
        )
        scores = self.model.similarity(
            query_embedding,
            self.document_embeddings,
        )[0].tolist()
        return rank_results(self.documents, scores, top_k)

What does this code do?

  • SentenceTransformer(MODEL_ID) prepares the public embedding model named by MODEL_ID.
  • document_embeddings stores one vector for each combined runbook title and description.
  • normalize_embeddings=True gives every vector the same length. This supports consistent cosine similarity comparisons.
  • search() converts each incident description into a normalized query embedding.
  • SentenceTransformer.similarity scores that query against every stored runbook embedding.
  • rank_results() preserves the same result shape used by the keyword retriever.
  • Save retrievers.py.
  • Confirm the existing keyword path still works by running this command:
python baseline_demo.py

What does this check prove?

This command imports the updated retrievers.py through the existing keyword demo. You should still see Expected: database-connections followed by Retrieved: memory-pressure.

The unchanged result proves the new class has not broken the baseline.

Seeing an import or syntax error?

Check that SemanticRetriever starts below the complete KeywordRetriever class. Confirm that both methods remain indented inside the new class.

Make sure the terminal prompt still starts with (.venv).

Help me debug the SemanticRetriever code in retrievers.py.

✔️ I have the full retriever

Good progress. Your keyword path still runs with SemanticRetriever ready for the interactive app.

ⓧ I'd like to double check the full code

  • Compare your complete retrievers.py file with this reference:
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"


def document_texts(documents):
    return [f"{document['title']}. {document['content']}" for document in documents]


def rank_results(documents, scores, top_k):
    ranked = sorted(
        zip(documents, scores),
        key=lambda item: item[1],
        reverse=True,
    )
    return [
        {**document, "score": float(score)}
        for document, score in ranked[:top_k]
    ]


class KeywordRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.vectorizer = TfidfVectorizer(stop_words="english")
        self.document_matrix = self.vectorizer.fit_transform(document_texts(documents))

    def search(self, query, top_k=3):
        query_vector = self.vectorizer.transform([query])
        scores = cosine_similarity(query_vector, self.document_matrix)[0]
        return rank_results(self.documents, scores, top_k)


class SemanticRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.model = SentenceTransformer(MODEL_ID)
        self.document_embeddings = self.model.encode(
            document_texts(documents),
            normalize_embeddings=True,
        )

    def search(self, query, top_k=3):
        query_embedding = self.model.encode(
            [query],
            normalize_embeddings=True,
        )
        scores = self.model.similarity(
            query_embedding,
            self.document_embeddings,
        )[0].tolist()
        return rank_results(self.documents, scores, top_k)
Create the result formatter

Both retrievers return the same ranked result shape. One formatter can therefore display titles, scores, and runbook text from either strategy.

  • Use the new-file control in the Visual Studio Code file list to create app.py inside incident-retrieval-lab.
  • Add the retriever imports and output formatter by pasting this code:
from retrievers import KeywordRetriever, SemanticRetriever
from runbooks import RUNBOOKS


def print_results(label, results):
    print(f"\n{label}")
    print("-" * len(label))
    for rank, result in enumerate(results, start=1):
        print(f"{rank}. {result['title']} [{result['score']:.3f}]")
        print(f"   {result['content']}")

How does the formatter work?

  • KeywordRetriever and SemanticRetriever give the app both ranking strategies.
  • RUNBOOKS supplies the shared incident corpus.
  • print_results() labels each result group before printing its ranked entries.
  • enumerate() assigns a visible rank starting at one.
  • score is formatted to three decimal places so the two ranking patterns are easy to compare.
  • Save app.py.
  • Check the Visual Studio Code file list for app.py.

You should see app.py listed beside baseline_demo.py and retrievers.py.

Cannot find app.py?

Confirm that you created the file inside incident-retrieval-lab. Check that the filename ends with .py.

Help me find or recreate app.py in my Visual Studio Code workspace.

Build and run the comparison loop

The main loop creates both retrievers once when the app starts. Each incident description then runs through the two ranking strategies.

  • Paste this code directly below print_results() in app.py:
def main():
    print("Loading local retrievers...")
    keyword = KeywordRetriever(RUNBOOKS)
    semantic = SemanticRetriever(RUNBOOKS)
    print("Ready. Type quit to stop.")

    while True:
        query = input("\nDescribe an incident: ").strip()
        if query.lower() in {"quit", "exit"}:
            break
        if not query:
            continue

        print_results("Keyword results", keyword.search(query))
        print_results("Semantic results", semantic.search(query))


if __name__ == "__main__":
    main()

What does the loop do?

  • KeywordRetriever(RUNBOOKS) prepares the lexical baseline.
  • SemanticRetriever(RUNBOOKS) loads the embedding model before encoding every runbook.
  • input() keeps the app waiting for incident descriptions.
  • quit or exit stops the loop.
  • keyword.search(query) produces the top three lexical results.
  • semantic.search(query) produces the top three semantic results.
  • Save app.py.

The first semantic run can pause while the public model downloads into your local Hugging Face cache. Download progress in the terminal confirms the process is active.

Before you run the app, do you expect the semantic ranking to repeat the keyword tie?

  • Start the interactive comparison by running this command:
python app.py

What happens on the first run?

The app creates the keyword retriever before loading sentence-transformers/all-MiniLM-L6-v2. The semantic retriever then encodes all six runbooks.

Later launches can reuse the model from your local cache.

  • Wait for Ready. Type quit to stop. to appear in the terminal.
  • Enter The backend cannot reach its data store because every client slot is occupied. at the incident prompt.

Under Keyword results, you should see Memory Pressure ranked first. Under Semantic results, you should see Database Connection Pool Exhaustion ranked first.

That is the recovery you were aiming for. Your retriever now recognizes the incident even when the query uses different words.

  • Type quit at the next incident prompt.

The interactive prompt should close and return you to the active (.venv) terminal.

Model not loading or results missing?

Confirm that your terminal prompt starts with (.venv). The installed packages belong to that environment.

If the model download stops, confirm that your computer can reach the internet before running the app again.

Help me debug the first semantic search run in app.py.

✔️ I have the comparison app

Your interactive app now compares three keyword results with three semantic results for the same incident description.

ⓧ I'd like to double check the full code

  • Compare your complete app.py file with this reference:
from retrievers import KeywordRetriever, SemanticRetriever
from runbooks import RUNBOOKS


def print_results(label, results):
    print(f"\n{label}")
    print("-" * len(label))
    for rank, result in enumerate(results, start=1):
        print(f"{rank}. {result['title']} [{result['score']:.3f}]")
        print(f"   {result['content']}")


def main():
    print("Loading local retrievers...")
    keyword = KeywordRetriever(RUNBOOKS)
    semantic = SemanticRetriever(RUNBOOKS)
    print("Ready. Type quit to stop.")

    while True:
        query = input("\nDescribe an incident: ").strip()
        if query.lower() in {"quit", "exit"}:
            break
        if not query:
            continue

        print_results("Keyword results", keyword.search(query))
        print_results("Semantic results", semantic.search(query))


if __name__ == "__main__":
    main()

Your semantic retriever now fixes the paraphrase failure that exposed the keyword baseline. Next, you will test both approaches across all six labeled incidents and measure their retrieval quality.

Benchmark Retrieval Quality

Your side-by-side search now shows how semantic retrieval can recover a runbook that TF-IDF misses.

One convincing query cannot prove retrieval quality. In this step, you will use retrieval evaluation to test both approaches against six labeled incidents.

In this step, get ready to:
  • Run both retrievers against all six evaluation cases.
  • Measure retrieval performance with two repeatable metrics.
  • Generate a Markdown report containing every prediction.
Evaluate all six cases

Each evaluation case pairs a paraphrased query with one expected runbook ID. Top-1 accuracy counts a hit only when that runbook ranks first.

  • In the Visual Studio Code file sidebar from earlier, use the new-file control to create evaluate.py inside incident-retrieval-lab.
  • Add the benchmark imports to evaluate.py by pasting this code:
from pathlib import Path
from time import perf_counter

from retrievers import KeywordRetriever, SemanticRetriever
from runbooks import EVAL_CASES, RUNBOOKS

What do these imports provide?

  • Path gives the generated report a local file destination.
  • perf_counter measures how long each search takes.
  • KeywordRetriever plus SemanticRetriever provide the two strategies being compared.
  • EVAL_CASES supplies the labeled queries. RUNBOOKS supplies the searchable documents.
  • Save evaluate.py.
  • Verify that the imports resolve by running this command in the active .venv terminal:
python evaluate.py

What does this check prove?

Python loads the new file plus its imported retrieval modules. A clean return to the prompt confirms that these imports resolve from the current workspace.

You should return to the terminal prompt without a traceback.

Can't resolve the imports?

  • Confirm that the active terminal prompt begins with (.venv).
  • Confirm that evaluate.py sits beside retrievers.py.
  • Confirm that runbooks.py sits in the same folder.

Help me diagnose unresolved imports in evaluate.py.

The benchmark applies one scoring rule to both retrievers. It records correctness plus latency measurement data for every query.

  • Add the benchmark() function below the imports in evaluate.py by pasting this code:
def benchmark(retriever, label):
    predictions = []
    total_seconds = 0.0

    for case in EVAL_CASES:
        started = perf_counter()
        result = retriever.search(case["query"], top_k=1)[0]
        total_seconds += perf_counter() - started
        predictions.append(
            {
                "query": case["query"],
                "expected": case["expected_id"],
                "predicted": result["id"],
                "correct": result["id"] == case["expected_id"],
            }
        )

    correct = sum(item["correct"] for item in predictions)
    return {
        "label": label,
        "accuracy": correct / len(predictions),
        "average_latency_ms": total_seconds * 1000 / len(predictions),
        "predictions": predictions,
    }

How does the benchmark work?

  • perf_counter() captures the elapsed search time for each case.
  • top_k=1 keeps only the highest-ranked runbook.
  • correct records whether the first result matches expected_id.
  • accuracy stores the proportion of correct first results.
  • average_latency_ms stores the average search time in milliseconds.
  • Save evaluate.py.
  • Verify the new function parses by running this command:
python evaluate.py

What does this check prove?

Python now defines benchmark() after loading the imports. A clean return confirms that the function structure is valid.

You should return to the terminal prompt without a traceback.

Seeing a syntax problem?

  • Check that the body of benchmark() is indented once.
  • Check that the body of the for loop is indented twice.
  • Check that the prediction dictionary remains inside predictions.append().

Help me fix the benchmark function.

Generate the Markdown report

Summary metrics make the retrievers easy to compare. Per-query records reveal which incidents produced those measurements.

  • Add the markdown_report() function below benchmark() in evaluate.py by pasting this code:
def markdown_report(results):
    lines = [
        "# Retrieval Benchmark Results",
        "",
        "| Retriever | Top-1 accuracy | Average query latency |",
        "| --- | ---: | ---: |",
    ]

    for result in results:
        lines.append(
            f"| {result['label']} | {result['accuracy']:.0%} | "
            f"{result['average_latency_ms']:.2f} ms |"
        )

    for result in results:
        lines.extend(["", f"## {result['label']} predictions", ""])
        for prediction in result["predictions"]:
            status = "PASS" if prediction["correct"] else "FAIL"
            lines.append(
                f"- {status}: expected `{prediction['expected']}`, "
                f"predicted `{prediction['predicted']}` for "
                f"{prediction['query']}"
            )

    return "\n".join(lines) + "\n"

What does the report contain?

  • The opening table records the accuracy of each retriever.
  • The same table records average query latency.
  • Each prediction section preserves the expected runbook ID.
  • Each prediction section preserves the retrieved runbook ID.
  • PASS plus FAIL expose individual successes or misses.
  • Save evaluate.py.
  • Verify the report function parses by running this command:
python evaluate.py

What does this check prove?

Python now defines the report builder after defining the benchmark. A clean return confirms that its strings plus loops have valid syntax.

You should return to the terminal prompt without a traceback.

Seeing a report syntax problem?

  • Check that each formatted string begins with f.
  • Check that every opening parenthesis has a matching closing parenthesis.
  • Check that the return statement remains inside markdown_report().

Help me debug the report builder.

The final function creates both retrievers under the same process. It prints their measurements before writing the detailed report.

  • Add the main() function below markdown_report() in evaluate.py by pasting this code:
def main():
    print("Loading retrievers...")
    results = [
        benchmark(KeywordRetriever(RUNBOOKS), "Keyword"),
        benchmark(SemanticRetriever(RUNBOOKS), "Semantic"),
    ]

    for result in results:
        print(
            f"{result['label']}: Top-1 accuracy={result['accuracy']:.0%}, "
            f"average latency={result['average_latency_ms']:.2f} ms"
        )

    Path("results.md").write_text(markdown_report(results), encoding="utf-8")
    print("Wrote results.md")


if __name__ == "__main__":
    main()

What does the main function do?

  • main() creates the keyword retriever.
  • main() creates the semantic retriever.
  • benchmark() evaluates each retriever against the same cases.
  • Path("results.md") writes the generated evidence into the project folder.
  • Save evaluate.py.

The cached embedding model still needs to initialize. The loading line can remain on screen for a short while during this local setup.

Before you run the completed benchmark, which retriever do you expect to rank more labeled incidents correctly?

  • Run both benchmarks by entering this command in the active terminal:
python evaluate.py

What does this run prove?

Both retrievers receive the same six queries. Their terminal rows make retrieval quality plus query cost directly comparable.

The generated report preserves the predictions behind those summary measurements. This gives you reproducible evidence for every score.

You should see Loading retrievers... followed by separate Keyword plus Semantic benchmark rows. The final line should read Wrote results.md.

Benchmark stopped before writing the report?

  • Read the first traceback line that references evaluate.py.
  • Compare the named function with its code block above.
  • Confirm that the terminal still uses the project environment from earlier.

Help me debug the failed benchmark.

Inspect the benchmark evidence

The summary rows show the overall trade-off. The prediction records show exactly where each retriever succeeds or fails.

  • Return to the file sidebar from earlier.
  • Select results.md.
  • Confirm that the opening table contains Keyword plus Semantic rows.
  • Confirm that each row contains Top-1 accuracy plus average query latency.
  • Confirm that the Keyword prediction section contains six records.
  • Confirm that the Semantic prediction section contains six records.

You should see machine-specific measurements in the table. Each prediction below it should include an expected ID plus a predicted ID with a status.

That is a meaningful milestone. Your retrieval choices now have repeatable evidence behind them.

✔️ Awesome, I've got everything!

Your evaluate.py file is saved. Your latest run has generated results.md with two summary rows plus twelve prediction records.

ⓧ I'd like to double check the full code

Compare your evaluate.py file with this complete version.

from pathlib import Path
from time import perf_counter

from retrievers import KeywordRetriever, SemanticRetriever
from runbooks import EVAL_CASES, RUNBOOKS


def benchmark(retriever, label):
    predictions = []
    total_seconds = 0.0

    for case in EVAL_CASES:
        started = perf_counter()
        result = retriever.search(case["query"], top_k=1)[0]
        total_seconds += perf_counter() - started
        predictions.append(
            {
                "query": case["query"],
                "expected": case["expected_id"],
                "predicted": result["id"],
                "correct": result["id"] == case["expected_id"],
            }
        )

    correct = sum(item["correct"] for item in predictions)
    return {
        "label": label,
        "accuracy": correct / len(predictions),
        "average_latency_ms": total_seconds * 1000 / len(predictions),
        "predictions": predictions,
    }


def markdown_report(results):
    lines = [
        "# Retrieval Benchmark Results",
        "",
        "| Retriever | Top-1 accuracy | Average query latency |",
        "| --- | ---: | ---: |",
    ]

    for result in results:
        lines.append(
            f"| {result['label']} | {result['accuracy']:.0%} | "
            f"{result['average_latency_ms']:.2f} ms |"
        )

    for result in results:
        lines.extend(["", f"## {result['label']} predictions", ""])
        for prediction in result["predictions"]:
            status = "PASS" if prediction["correct"] else "FAIL"
            lines.append(
                f"- {status}: expected `{prediction['expected']}`, "
                f"predicted `{prediction['predicted']}` for "
                f"{prediction['query']}"
            )

    return "\n".join(lines) + "\n"


def main():
    print("Loading retrievers...")
    results = [
        benchmark(KeywordRetriever(RUNBOOKS), "Keyword"),
        benchmark(SemanticRetriever(RUNBOOKS), "Semantic"),
    ]

    for result in results:
        print(
            f"{result['label']}: Top-1 accuracy={result['accuracy']:.0%}, "
            f"average latency={result['average_latency_ms']:.2f} ms"
        )

    Path("results.md").write_text(markdown_report(results), encoding="utf-8")
    print("Wrote results.md")


if __name__ == "__main__":
    main()

Your benchmark now turns six labeled incidents into measurable evidence. Next, you will package the demo so another person can understand its problem, architecture, results, plus trade-off.

Package the Portfolio Demo

Your six-query retrieval evaluation now turns retrieval quality into measurable evidence. The remaining challenge is making that evidence understandable without asking someone to reverse-engineer your scripts.

A concise README gives reviewers a fast route through the project story. A final demo lets you explain what each retrieval method gains or loses.

In this step, get ready to:
  • Document the project story in README.md.
  • Regenerate the current benchmark evidence in results.md.
  • Rehearse the interactive retrieval demo.
Write the portfolio README

A README connects the scripts in your repository to a clear engineering decision. It also gives a reviewer the commands needed to reproduce your evidence.

  • Click the New File icon beside incident-retrieval-lab in the Explorer sidebar.
  • Type README.md.
  • Press Enter to create the file.

You should now see README.md listed with the existing project files.

  • Add the project story and local commands by pasting this first section:
# Incident Runbook Retrieval Lab

This project benchmarks two retrieval strategies for an AI incident-response assistant:

1. A TF-IDF keyword baseline.
2. Dense semantic retrieval with a local sentence embedding model.

## Why it matters

Retrieval is the evidence-selection stage of a retrieval-augmented generation system. If retrieval selects the wrong runbook, answer generation receives the wrong context. This project evaluates retrieval before adding an LLM.

## Architecture

User query -> Keyword or semantic retriever -> Ranked incident runbooks -> Evaluation report

## Run locally

```text
python -m pip install -r requirements.txt
python baseline_demo.py
python evaluate.py
python app.py
```

The first semantic run downloads the public embedding model and caches it locally.

What does this section communicate?

  • The opening frames the incident-response problem around a TF-IDF baseline.
  • The same opening introduces dense semantic retrieval as the meaning-based comparison.
  • The architecture traces a query through retrieval to the evaluation report.
  • The Run locally section gives reviewers a reproducible command sequence.
  • Press Ctrl+S to save README.md.
  • Click Open Preview to the Side in the editor toolbar.

The Markdown preview should show the project title at the top. The local commands should appear in a fenced block.

Preview missing sections?

  • Confirm the file is named README.md.
  • Save the file before reopening the preview.

Help me fix my README preview.

  • Add one blank line after the sentence about the local cache.
  • Add the evidence and trade-off sections by pasting this content:
## Evaluation

`evaluate.py` runs six labeled paraphrase queries. A prediction passes only when the expected runbook is ranked first. The script reports Top-1 accuracy, measures average query latency, and writes the detailed predictions to `results.md`.

## Design decision

TF-IDF provides an explainable lexical baseline but can fail when the query and runbook use different terms. Dense embeddings can recover semantic similarity, but they add model-loading and inference cost. Both need a labeled evaluation set because neither approach is universally correct.

## Limitations

- The corpus and evaluation set are intentionally small.
- The benchmark measures retrieval, not answer faithfulness.
- No reranker, vector database, API, deployment, or observability layer is included yet.

## Next project

Use the ranked runbook text as grounded context for an answer-generation service, then add citations, unsupported-question handling, and regression tests.

How does this strengthen the portfolio story?

  • Evaluation connects six labeled queries to Top-1 accuracy.
  • The same section makes latency measurement part of the evidence.
  • Design decision compares lexical explainability with embedding cost.
  • Next project points toward a grounded RAG assistant.
  • Press Ctrl+S to save the completed README.md.

The preview should now include Evaluation, Design decision, Limitations, and Next project sections. Each section gives a reviewer a specific claim to inspect.

Formatting looks broken?

  • Confirm there is one blank line before ## Evaluation.
  • Compare each heading marker with the full file below.

Help me find the Markdown formatting mistake.

✔️ Awesome, I've got everything!

Your README now connects the working code to the evidence a reviewer needs.

ⓧ I'd like to double check the full code

  • Compare README.md against this complete file:
# Incident Runbook Retrieval Lab

This project benchmarks two retrieval strategies for an AI incident-response assistant:

1. A TF-IDF keyword baseline.
2. Dense semantic retrieval with a local sentence embedding model.

## Why it matters

Retrieval is the evidence-selection stage of a retrieval-augmented generation system. If retrieval selects the wrong runbook, answer generation receives the wrong context. This project evaluates retrieval before adding an LLM.

## Architecture

User query -> Keyword or semantic retriever -> Ranked incident runbooks -> Evaluation report

## Run locally

```text
python -m pip install -r requirements.txt
python baseline_demo.py
python evaluate.py
python app.py
```

The first semantic run downloads the public embedding model and caches it locally.

## Evaluation

`evaluate.py` runs six labeled paraphrase queries. A prediction passes only when the expected runbook is ranked first. The script reports Top-1 accuracy, measures average query latency, and writes the detailed predictions to `results.md`.

## Design decision

TF-IDF provides an explainable lexical baseline but can fail when the query and runbook use different terms. Dense embeddings can recover semantic similarity, but they add model-loading and inference cost. Both need a labeled evaluation set because neither approach is universally correct.

## Limitations

- The corpus and evaluation set are intentionally small.
- The benchmark measures retrieval, not answer faithfulness.
- No reranker, vector database, API, deployment, or observability layer is included yet.

## Next project

Use the ranked runbook text as grounded context for an answer-generation service, then add citations, unsupported-question handling, and regression tests.
Refresh the benchmark evidence

Performance evidence can drift as the implementation changes. Running the benchmark again keeps results.md aligned with the code a reviewer sees.

Before you run the benchmark, predict which values are more likely to change:

  • Regenerate results.md from the active (.venv) Command Prompt terminal by running:
python evaluate.py

What does this refresh?

The evaluate.py script runs both retrievers against all six labeled cases. It rewrites results.md with measurements from the current machine.

  • Check the terminal for the Keyword benchmark row.
  • Check the terminal for the Semantic benchmark row.
  • Confirm the final line reads Wrote results.md.
  • Return to results.md from earlier.
  • Confirm the report contains the current table plus the detailed predictions.

Benchmark does not complete?

  • Confirm the terminal prompt starts with (.venv).
  • Check that the terminal is still in the incident-retrieval-lab folder.

Help me debug the benchmark run.

Rehearse the interactive demo

The interactive app turns the benchmark story into something a reviewer can watch. Two incident types give you enough evidence to explain the behavior without relying on one lucky query.

  • Start the interactive demo from the active (.venv) terminal by running:
python app.py

What does this command start?

The app.py script loads the keyword retriever plus the semantic retriever. It keeps the incident prompt open for repeated comparisons.

You should see Ready. Type quit to stop. before the incident prompt appears.

App does not reach the prompt?

  • Confirm the terminal still uses the active (.venv) environment.
  • Check that retrievers.py remains in the incident-retrieval-lab folder.
  • Check that runbooks.py remains in the same folder.

Help me get the interactive prompt running.

Before you paste this query, predict which retriever is more likely to understand its paraphrase:

  • Paste this database incident at the Describe an incident: prompt:
The backend cannot reach its data store because every client slot is occupied.

What did this query prove?

The semantic results rank Database Connection Pool Exhaustion first. The keyword results rank Memory Pressure first.

The TF-IDF baseline has no useful shared terms for this paraphrase. Dense embeddings recover the relationship between the incident description and the database runbook.

Before the next query, predict whether both retrievers can produce a complete three-result display:

  • Paste this authentication incident at the next Describe an incident: prompt:
Sign-in calls started returning unauthorized responses after credentials rotated.

What does the complete demo show?

This query gives you a second incident type for the demo. You should see three numbered results in each column.

Every result includes a title plus a score. The runbook text appears beneath it.

  • Lexical matching can miss paraphrases when the query shares no useful terms with the runbook.
  • Dense embeddings can rank related meanings despite different wording.
  • Retrieval evaluation tests evidence selection before answer generation can hide a mistake.
  • Practice the three retrieval explanations aloud.
  • Stop the interactive app cleanly by entering:
quit

What does this input do?

The app.py loop recognizes quit as its exit input. Your terminal returns to the active (.venv) prompt.

That closes the loop. Your repository now pairs a working demo with fresh evidence.

Secret mission

Build a Hybrid Retriever

Blend keyword scores with semantic scores after min-max normalization. Then benchmark the weighted hybrid against both existing retrievers to see whether it improves, matches, or reduces retrieval accuracy.

Clean Up Your Resources

Clean Up Your Resources

Your retrieval lab runs entirely on your Windows computer. No ongoing charges accrue when you leave the workspace or model cache in place.

Choose whether to keep the workspace, pause your work, or delete the local resources.

Resources you used:

  • The incident-retrieval-lab folder on your Desktop.
  • The .venv inside incident-retrieval-lab with the pinned Python packages.
  • The shared Hugging Face cache entry for sentence-transformers/all-MiniLM-L6-v2.

Keep everything running

No action needed. Choose this if you plan to continue developing your incident-response retrieval system.

  • Keep the incident-retrieval-lab folder on your Desktop for future retrieval experiments.
  • Keep the cached model to avoid downloading it again during your next semantic retrieval run.
  • Return to the workspace when you want to extend the benchmark or build the answer-generation layer.

Pause - I'll come back to this later

Close the active development session to free local memory. Your files and cached model stay on disk.

  • Close the active Command Prompt terminal in Visual Studio Code.
  • Close Visual Studio Code.
  • Leave the incident-retrieval-lab folder on your Desktop.
  • Leave the shared model cache in place for your next semantic retrieval run.

No retrieval process remains running. No charges accrue while the project is paused.

Delete - I don't want to use this again

Deleting the workspace is permanent. No cloud resources or accounts are affected.

  • Close Visual Studio Code.
  • Use Windows File Explorer to locate the incident-retrieval-lab folder on your Desktop.
  • Remove the incident-retrieval-lab folder from your Desktop.
  • Confirm that incident-retrieval-lab no longer appears on your Desktop.
  • Check whether another local project uses sentence-transformers/all-MiniLM-L6-v2.
  • Remove sentence-transformers/all-MiniLM-L6-v2 through your local Hugging Face cache management workflow only when no other project uses it.
  • Keep the shared model cache entry if you are unsure whether another project depends on it.

Nice Work!

Nice Work!

You did it! You built a local incident-runbook search tool with a repeatable benchmark for retrieval quality.

You've learned how to:

  • Build a deterministic TF-IDF baseline that exposes a paraphrase failure before stronger retrieval methods enter the comparison.
  • Use dense embeddings for local semantic search that recovers the database runbook from natural-language incident descriptions.
  • Measure Top-1 accuracy across six labeled queries. Record average query latency with detailed predictions in results.md. Present the evidence through an interactive CLI with a technical README.
  • Secret Mission: Build a weighted hybrid retriever that combines min-max-normalized lexical scores with semantic scores. Compare its benchmark row with the keyword retriever. Compare it with the semantic retriever too.

Ready to quiz yourself?