Build a RAG Sufficiency Gate

Train a text classifier that routes RAG answers by sufficiency.

Introduction

30 Second Summary

A confident answer can still leave an important question unresolved. Without a clear checkpoint, weak answers can reach people who expect something useful.

In this project, you will train a binary text classifier in Google Colab to judge question and candidate-answer pairs from your existing RAG system. The finished gate returns a sufficiency decision with an estimated probability.

What You'll Build

When you paste a live RAG question and answer into your notebook, your quality gate prints a clear decision with the model's estimated probability.

By the end of this project, you'll have:

  • A validated dataset summary showing usable rows plus counts for INSUFFICIENT and SUFFICIENT.
  • A trained sufficiency classifier that scores held-out question-answer pairs using TF-IDF with logistic regression.
  • A reusable quality gate that prints a decision with its sufficient-class probability before downloading rag_sufficiency_gate.pkl.
  • Secret Mission: Add a human-review band that routes uncertain predictions to HUMAN_REVIEW.

Are there any prerequisites?

You need a Google account with Colab access. You also need Python basics plus an existing RAG system.

Your labeled CSV must contain the exact columns question, answer, and sufficiency_label.

Before We Start

Before any hands-on work, lock in what your RAG quality gate will judge. You are building a classifier that learns from labeled question and candidate-answer pairs without claiming to verify truth without evidence.

Set Up the Colab Notebook

Your quality gate depends on a consistent notebook environment. Package drift can turn a first machine-learning experiment into dependency troubleshooting.

Google Colab provides a hosted Python 3.13 runtime. Pinned package versions keep every later result reproducible inside this notebook.

In this step, get ready to:
  • Create a blank Colab notebook in the standard CPU runtime.
  • Install the pinned machine-learning packages.
  • Verify the runtime versions plus the project imports.
Create the blank notebook

A Colab notebook holds executable code in separate cells. The hosted runtime runs those cells without requiring a local Python installation.

  • Go to Google Colab in your browser.
  • Create a blank notebook from the landing page.

You'll see an empty notebook with its first code cell ready.

  • Rename the notebook to rag_sufficiency_gate.ipynb by editing the notebook title.
  • Keep the standard CPU runtime selected.

The notebook title now shows rag_sufficiency_gate.ipynb. Your first code cell is ready for the environment setup.

Why keep the CPU runtime?

This project's linear baseline does not require a GPU. The standard CPU runtime provides everything needed for training plus evaluation.

Install the pinned packages

pandas handles the labeled table data used by the gate. scikit-learn provides the text-classification tools used by the model.

The exact pins keep pandas on 3.1.0. They also keep scikit-learn on 1.9.1.

  • Paste the package installation cell into the first code cell using the code below:
# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

What does this cell do?

  • The %pip command installs packages into the active notebook runtime.
  • The -q flag reduces the installation output.
  • The pandas==3.1.0 pin selects the required pandas release.
  • The scikit-learn==1.9.1 pin selects the required scikit-learn release.

The first package installation can stay busy for a little while. That delay comes from preparing the pinned libraries in the hosted runtime.

  • Run Cell 1 using the run icon beside the code cell.

You'll see the cell finish without a package installation error. The runtime now contains both pinned packages.

Package installation not completing?

  • Confirm that the notebook is connected to its hosted runtime.
  • Check that your browser still has an internet connection.
  • Run Cell 1 again after the connection is active.
  • Get help with the installation error.
Verify the runtime and imports

A successful installation only proves that the package command finished. Importing the project APIs confirms that Python can load those packages inside the active runtime.

  • Add a second code cell below Cell 1 using the notebook's code-cell control.
  • Paste the environment imports plus version checks into Cell 2 using this code:
# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

What does this cell do?

  • The standard library imports expose the Python runtime version.
  • They also prepare model saving for the final notebook.
  • The package imports expose the library versions you pinned.
  • The remaining imports prepare the machine-learning APIs used by later cells.

Before you run Cell 2, which three version lines do you expect the notebook to print?

  • Run Cell 2 using the run icon beside the code cell.

✔️ I see the expected versions

That is your environment locked down. Your notebook can now run every later cell against the expected libraries.

The output begins with Python: 3.13. The pandas line shows 3.1.0.

The scikit-learn line shows 1.9.1.

ⓧ I see an older Python version

The current hosted Colab runtime uses Python 3.13. A lower version indicates that this notebook is attached to an older runtime.

  • Return to the Colab landing page.
  • Create another blank notebook in the current hosted runtime.
  • Rename the fresh notebook to rag_sufficiency_gate.ipynb.
  • Paste Cell 1 into the fresh notebook.
  • Run Cell 1.
  • Paste Cell 2 below Cell 1.
  • Run Cell 2.

Still seeing an older runtime?

Use a fresh hosted notebook instead of continuing in the outdated runtime. Help me verify my Colab runtime version.

ⓧ A package version is different

A version mismatch means the active runtime has not loaded one of the pinned packages. Refreshing the runtime state gives the imports a clean start.

  • Run Cell 1 again.
  • End the current session using Colab's runtime controls.
  • Connect the notebook to a fresh standard CPU runtime.
  • Run Cell 1 in the fresh runtime.
  • Run Cell 2 after Cell 1 finishes.

Versions still do not match?

Check that Cell 1 contains the exact package pins from this step. Help me diagnose this Colab package mismatch.

✔️ Awesome, I've got everything!

Great. Double-check that both cells ran successfully in rag_sufficiency_gate.ipynb.

ⓧ I'd like to double check the full code

# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

Your notebook now has the verified runtime plus the project imports. Next, you'll load the labeled CSV and protect the classifier from unusable rows.

Load and Validate the Answers

Your Google Colab notebook now has a verified environment. The next input is your labeled CSV.

A binary text classifier can only learn the intended decision from usable examples. Missing columns can stop training. Blank text or inconsistent labels can teach the wrong pattern.

In this step, get ready to:
  • Upload one labeled CSV into the notebook.
  • Normalize the labels after cleaning the question-answer examples.
  • Verify the usable row count plus both class counts.
Upload and validate the CSV schema

The upload function returns the selected file by name. pandas then reads that file into a table named df.

  • Add a code cell beneath the version cell by using the notebook toolbar.
  • Upload one CSV and enforce its required columns by pasting this code into the new cell:
# Cell 3: Upload one CSV and validate its schema
uploaded = files.upload()
if len(uploaded) != 1:
    raise ValueError("Upload exactly one CSV file.")

filename = next(iter(uploaded))
if not filename.lower().endswith(".csv"):
    raise ValueError("The uploaded dataset must be a CSV file.")

df = pd.read_csv(filename)
required_columns = ["question", "answer", "sufficiency_label"]
missing_columns = [column for column in required_columns if column not in df.columns]
if missing_columns:
    raise ValueError(f"Missing required columns: {missing_columns}")

What does this code do?

  • The files.upload() function opens the browser file picker.
  • The upload count check requires one file.
  • The file extension check requires a .csv file.
  • The required_columns list defines the dataset contract used by the classifier.
  • Run the cell with the control beside it.
  • Select your labeled CSV in the Windows file picker.

The upload completes when the selected file meets the extension and schema checks. The cell finishes with the dataset loaded into df.

Did the upload stop?

  • Select exactly one file if the upload count check stops the cell.
  • Choose a filename ending in .csv if the extension check stops the cell.
  • Rename the dataset columns to question, answer, and sufficiency_label if the schema check reports missing columns.

The column names are case-sensitive. Help me diagnose why my CSV is failing the upload or schema checks.

Normalize labels and clean examples

Label normalization gives common binary spellings one consistent meaning. Text cleanup removes examples that contain no usable question or candidate answer.

  • Return to Cell 3.
  • Append the normalization and cleanup logic beneath the schema check by pasting this code:
label_map = {
    "0": 0,
    "0.0": 0,
    "false": 0,
    "no": 0,
    "insufficient": 0,
    "1": 1,
    "1.0": 1,
    "true": 1,
    "yes": 1,
    "sufficient": 1,
}

def normalize_label(value):
    key = str(value).strip().lower()
    if key not in label_map:
        raise ValueError(
            f"Unsupported sufficiency_label value: {value!r}. "
            f"Supported values are: {sorted(label_map)}"
        )
    return label_map[key]

df = df[required_columns].copy()
df["question"] = df["question"].fillna("").astype(str).str.strip()
df["answer"] = df["answer"].fillna("").astype(str).str.strip()
df = df[(df["question"] != "") & (df["answer"] != "")].copy()
df["label"] = df["sufficiency_label"].map(normalize_label)
df = df.drop_duplicates(subset=["question", "answer", "label"]).reset_index(drop=True)

How does the cleanup work?

  • The label_map dictionary assigns insufficient spellings to 0.
  • The same dictionary assigns sufficient spellings to 1.
  • The normalize_label() function standardizes each value before looking it up.
  • The text cleanup trims whitespace before removing blank examples.
  • The duplicate check keeps one copy of each matching question-answer-label example.
  • Run Cell 3 again.
  • Select the same CSV when the file picker opens.

A compatible dataset completes the cleanup without an exception. Your df table now contains normalized label values.

Seeing an unsupported label?

  • Correct any label that falls outside the spellings listed in label_map.
  • Remove accidental notes from cells in the sufficiency_label column.
  • Check the source CSV for labels with extra words.

The function already handles surrounding spaces plus capitalization differences. Help me find the unsupported values in my sufficiency_label column.

Validate and inspect the cleaned dataset

The final guards protect the training step from a dataset that is too small or missing a class. The summary makes each requirement visible before modeling begins.

  • Return to the bottom of Cell 3.
  • Append the dataset checks and summary output by pasting this code:
if len(df) < 10:
    raise ValueError("At least 10 usable labeled rows are required for this prototype.")
if set(df["label"].unique()) != {0, 1}:
    raise ValueError("The cleaned dataset must contain both sufficient and insufficient examples.")
if df["label"].value_counts().min() < 2:
    raise ValueError("Each class needs at least two usable examples.")

label_counts = df["label"].value_counts().sort_index().rename(
    index={0: "INSUFFICIENT", 1: "SUFFICIENT"}
)
print(f"Cleaned rows: {len(df)}")
print("\nSample rows:")
print(df[["question", "answer", "label"]].head(3).to_string(index=False))
print("\nClass counts:")
print(label_counts.to_string())

What do these checks prove?

  • The row-count guard requires at least ten usable examples.
  • The label-set guard requires normalized classes 0 and 1.
  • The class-count guard requires at least two examples in each class.
  • The final output shows the cleaned size plus three examples plus the class balance.

Before you run the complete cell, do you expect both classes to survive the cleanup?

  • Run the completed Cell 3.
  • Select the same CSV when the file picker opens.

The output starts with Cleaned rows: followed by a count of at least ten.

The Sample rows: section shows three cleaned question-answer examples.

The Class counts: section shows nonzero counts for INSUFFICIENT and SUFFICIENT.

You have secured the data foundation for the model. Every remaining row now matches the classifier's expected contract.

Did a dataset check stop the cell?

  • Add more usable examples if fewer than ten rows remain after cleanup.
  • Add the missing class if the cleaned labels contain only one binary value.
  • Add more examples to any class with fewer than two usable rows.

Run the complete cell again after updating the CSV. Help me interpret which dataset validation check failed.

✔️ Awesome, I've got everything!

Great. Confirm that Cell 3 is saved in rag_sufficiency_gate.ipynb.

ⓧ I'd like to double check the full code

# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

# Cell 3: Upload one CSV and validate its schema
uploaded = files.upload()
if len(uploaded) != 1:
    raise ValueError("Upload exactly one CSV file.")

filename = next(iter(uploaded))
if not filename.lower().endswith(".csv"):
    raise ValueError("The uploaded dataset must be a CSV file.")

df = pd.read_csv(filename)
required_columns = ["question", "answer", "sufficiency_label"]
missing_columns = [column for column in required_columns if column not in df.columns]
if missing_columns:
    raise ValueError(f"Missing required columns: {missing_columns}")

label_map = {
    "0": 0,
    "0.0": 0,
    "false": 0,
    "no": 0,
    "insufficient": 0,
    "1": 1,
    "1.0": 1,
    "true": 1,
    "yes": 1,
    "sufficient": 1,
}


def normalize_label(value):
    key = str(value).strip().lower()
    if key not in label_map:
        raise ValueError(
            f"Unsupported sufficiency_label value: {value!r}. "
            f"Supported values are: {sorted(label_map)}"
        )
    return label_map[key]


df = df[required_columns].copy()
df["question"] = df["question"].fillna("").astype(str).str.strip()
df["answer"] = df["answer"].fillna("").astype(str).str.strip()
df = df[(df["question"] != "") & (df["answer"] != "")].copy()
df["label"] = df["sufficiency_label"].map(normalize_label)
df = df.drop_duplicates(subset=["question", "answer", "label"]).reset_index(drop=True)

if len(df) < 10:
    raise ValueError("At least 10 usable labeled rows are required for this prototype.")
if set(df["label"].unique()) != {0, 1}:
    raise ValueError("The cleaned dataset must contain both sufficient and insufficient examples.")
if df["label"].value_counts().min() < 2:
    raise ValueError("Each class needs at least two usable examples.")

label_counts = df["label"].value_counts().sort_index().rename(
    index={0: "INSUFFICIENT", 1: "SUFFICIENT"}
)
print(f"Cleaned rows: {len(df)}")
print("\nSample rows:")
print(df[["question", "answer", "label"]].head(3).to_string(index=False))
print("\nClass counts:")
print(label_counts.to_string())

Your labeled answers are clean and ready for modeling. Next, you will train the first sufficiency classifier and expose the limits of accuracy alone.

Train the Sufficiency Classifier

Your cleaned dataset now gives the Google Colab notebook consistent examples of sufficient answers and insufficient answers. Those labeled examples give the model a decision pattern to learn.

This step trains a binary text classifier with TF-IDF plus logistic regression. You will test its first accuracy score before deciding how much evidence that score provides.

In this step, get ready to:
  • Create marked question-answer records for the model.
  • Make a reproducible stratified training and test split.
  • Fit the classifier and challenge its accuracy-only result.
Prepare marked text pairs

Each model input needs the question and candidate answer in one consistent record. A stratified train-test split keeps both labels represented while reserving rows that the model does not train on.

  • Add a new code cell directly beneath Cell 3 in the notebook.
  • Prepare the model inputs by pasting this code into the new cell:
# Cell 4: Build text pairs, split the data, train, and expose accuracy's limitation

def build_model_text(question, answer):
    return f"Question: {question}\nAnswer: {answer}"


df["model_text"] = [
    build_model_text(question, answer)
    for question, answer in zip(df["question"], df["answer"])
]

train_df, test_df = train_test_split(
    df,
    test_size=0.20,
    random_state=42,
    stratify=df["label"],
)

print(f"Training rows: {len(train_df)}")
print(f"Test rows: {len(test_df)}")

What does this code do?

  • The build_model_text() function marks the question and answer sections so their roles remain visible in each model input.
  • The model_text column stores one formatted record for every cleaned row.
  • The test_size=0.20 setting reserves one fifth of the rows for testing.
  • The random_state=42 setting makes the shuffled split repeatable.
  • The stratify=df["label"] setting uses the class labels when dividing the rows.
  • Run Cell 4 using the button beside the cell.

You will see separate training and test row counts. Together, the two counts match the cleaned row count from Cell 3.

Split not completing?

  • Rerun Cell 3 to restore the cleaned df dataset and its label column.
  • Confirm that the class counts from Cell 3 contain both INSUFFICIENT and SUFFICIENT examples.
  • Use this prompt for targeted help: Help me diagnose why my stratified train-test split fails with this cleaned dataset.
Build the baseline pipeline

The vectorizer converts words and two-word phrases into numeric features. The classifier learns how those features relate to the two sufficiency labels.

Why TF-IDF and logistic regression?

This combination trains quickly on the notebook's CPU runtime. It also provides class probabilities that a quality gate can use later.

A Transformer classifier adds tokenization and fine-tuning machinery. This baseline keeps your attention on the sufficiency decision itself.

  • Click inside Cell 4 directly above the line that prints Training rows:.
  • Insert the model pipeline at that location by pasting this code:
model = Pipeline(
    steps=[
        (
            "tfidf",
            TfidfVectorizer(
                ngram_range=(1, 2),
                min_df=1,
                max_features=20000,
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                l1_ratio=0.0,
                class_weight="balanced",
                max_iter=1000,
            ),
        ),
    ]
)

How is the pipeline structured?

  • The tfidf step creates features from individual words and two-word phrases.
  • The max_features=20000 setting caps the feature vocabulary.
  • The sublinear_tf=True setting reduces the influence of very frequent terms.
  • The classifier step uses balanced class weights so each label influences training according to its frequency.
  • The max_iter=1000 setting gives the classifier room to complete its optimization.
  • Run Cell 4 again using the button beside the cell.

The training and test row counts print again without an exception. This confirms that the text preparation and model definition can run together.

Pipeline definition failing?

  • Confirm that the entire model = Pipeline( block sits above the two row-count lines.
  • Check that every opening parenthesis and bracket has its matching closing character.
  • Use this prompt for targeted help: Help me compare my Pipeline definition with the expected TF-IDF and logistic-regression structure.
Fit and challenge the accuracy score

The pipeline is now defined against the training split. Fitting it creates the learned model that can predict labels for the held-out rows.

  • Select the two row-count lines at the bottom of Cell 4.
  • Replace those lines with the complete training and output section below:
model.fit(train_df["model_text"], train_df["label"])
test_predictions = model.predict(test_df["model_text"])
baseline_accuracy = accuracy_score(test_df["label"], test_predictions)

print(f"Training rows: {len(train_df)}")
print(f"Test rows: {len(test_df)}")
print(f"Baseline accuracy: {baseline_accuracy:.3f}")
print("Accuracy alone is incomplete because it does not show which class is failing.")

What happens during training?

  • The fit() call learns feature weights from the training records and their labels.
  • The predict() call assigns one label to each held-out record.
  • The accuracy_score() call calculates the fraction of held-out predictions that match their labels.
  • The .3f format displays the score with three decimal places.

Before you run the cell, predict whether one accuracy score can expose which label the model gets wrong.

  • Run Cell 4 to train the model and reveal the baseline result.

You have your first fitted sufficiency classifier. You will see training rows, test rows, a Baseline accuracy: value, and the accuracy-only warning.

Why does the result fall short?

Accuracy combines every correct prediction into one number. It does not reveal whether the model is missing insufficient answers or sufficient answers.

That missing label-level evidence is the intended gap in this baseline. The model is trained, but the single score cannot show which type of mistake needs attention.

Training cell not finishing?

  • Confirm that Cell 4 still begins with the build_model_text() function.
  • Confirm that the model.fit() line appears after the complete Pipeline definition.
  • Use this prompt for targeted help: Help me diagnose why Cell 4 does not fit my sufficiency classifier or print baseline accuracy.

✔️ Awesome, I've got everything!

Great work. Double-check that Cell 4 is saved with its latest output in the notebook.

ⓧ I'd like to double check the full code

# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

# Cell 3: Upload one CSV and validate its schema
uploaded = files.upload()
if len(uploaded) != 1:
    raise ValueError("Upload exactly one CSV file.")

filename = next(iter(uploaded))
if not filename.lower().endswith(".csv"):
    raise ValueError("The uploaded dataset must be a CSV file.")

df = pd.read_csv(filename)
required_columns = ["question", "answer", "sufficiency_label"]
missing_columns = [column for column in required_columns if column not in df.columns]
if missing_columns:
    raise ValueError(f"Missing required columns: {missing_columns}")

label_map = {
    "0": 0,
    "0.0": 0,
    "false": 0,
    "no": 0,
    "insufficient": 0,
    "1": 1,
    "1.0": 1,
    "true": 1,
    "yes": 1,
    "sufficient": 1,
}


def normalize_label(value):
    key = str(value).strip().lower()
    if key not in label_map:
        raise ValueError(
            f"Unsupported sufficiency_label value: {value!r}. "
            f"Supported values are: {sorted(label_map)}"
        )
    return label_map[key]


df = df[required_columns].copy()
df["question"] = df["question"].fillna("").astype(str).str.strip()
df["answer"] = df["answer"].fillna("").astype(str).str.strip()
df = df[(df["question"] != "") & (df["answer"] != "")].copy()
df["label"] = df["sufficiency_label"].map(normalize_label)
df = df.drop_duplicates(subset=["question", "answer", "label"]).reset_index(drop=True)

if len(df) < 10:
    raise ValueError("At least 10 usable labeled rows are required for this prototype.")
if set(df["label"].unique()) != {0, 1}:
    raise ValueError("The cleaned dataset must contain both sufficient and insufficient examples.")
if df["label"].value_counts().min() < 2:
    raise ValueError("Each class needs at least two usable examples.")

label_counts = df["label"].value_counts().sort_index().rename(
    index={0: "INSUFFICIENT", 1: "SUFFICIENT"}
)
print(f"Cleaned rows: {len(df)}")
print("\nSample rows:")
print(df[["question", "answer", "label"]].head(3).to_string(index=False))
print("\nClass counts:")
print(label_counts.to_string())

# Cell 4: Build text pairs, split the data, train, and expose accuracy's limitation

def build_model_text(question, answer):
    return f"Question: {question}\nAnswer: {answer}"


df["model_text"] = [
    build_model_text(question, answer)
    for question, answer in zip(df["question"], df["answer"])
]

train_df, test_df = train_test_split(
    df,
    test_size=0.20,
    random_state=42,
    stratify=df["label"],
)

model = Pipeline(
    steps=[
        (
            "tfidf",
            TfidfVectorizer(
                ngram_range=(1, 2),
                min_df=1,
                max_features=20000,
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                l1_ratio=0.0,
                class_weight="balanced",
                max_iter=1000,
            ),
        ),
    ]
)

model.fit(train_df["model_text"], train_df["label"])
test_predictions = model.predict(test_df["model_text"])
baseline_accuracy = accuracy_score(test_df["label"], test_predictions)

print(f"Training rows: {len(train_df)}")
print(f"Test rows: {len(test_df)}")
print(f"Baseline accuracy: {baseline_accuracy:.3f}")
print("Accuracy alone is incomplete because it does not show which class is failing.")

Your baseline now turns question-answer text into held-out predictions. Next, you will inspect each label separately to uncover the mistakes that accuracy hides.

Evaluate Both Sufficiency Classes

Your fitted model now makes held-out predictions in Google Colab. Its accuracy score compresses both classes into one number.

Before this gate reaches your RAG workflow, you need to see how it treats each label. Per-class precision, recall, F1, plus a confusion matrix reveal false approvals separately from false rejections.

In this step, get ready to:
  • Generate a classification report for both sufficiency labels.
  • Build a labeled two-by-two confusion table.
  • Inspect insufficient-answer recall plus false-sufficient predictions.
Generate both evaluation views

A class-level report measures each label independently. The labeled table then shows exactly where the model placed every held-out example.

  • Add Cell 5 below Cell 4 by pasting this code:
# Cell 5: Evaluate both classes and print a labeled confusion matrix
label_order = [0, 1]
label_names = ["INSUFFICIENT", "SUFFICIENT"]

report = classification_report(
    test_df["label"],
    test_predictions,
    labels=label_order,
    target_names=label_names,
    digits=3,
    zero_division=0,
)
print(report)

matrix = confusion_matrix(
    test_df["label"],
    test_predictions,
    labels=label_order,
)
matrix_df = pd.DataFrame(
    matrix,
    index=[f"actual_{name}" for name in label_names],
    columns=[f"predicted_{name}" for name in label_names],
)
print("Confusion matrix:")
print(matrix_df.to_string())

What does this evaluation show?

  • The label_order list keeps both evaluation outputs in the same class order.
  • The classification_report call calculates precision, recall, F1, plus support for each class.
  • The digits=3 setting displays each decimal metric to three places.
  • The zero_division=0 setting gives undefined division results a numeric value of zero.
  • The confusion_matrix call counts each actual-label plus predicted-label combination.
  • The pandas DataFrame adds readable row names plus column names to those counts.
  • Run Cell 5 using its run control.

You'll see separate INSUFFICIENT plus SUFFICIENT rows in the report. A labeled two-by-two table appears below them.

Evaluation cell not running?

Run Cell 4 again if the notebook no longer has test_df or test_predictions in memory. Cell 5 depends on both objects.

Check that classification_report plus confusion_matrix were imported in Cell 2. Help me diagnose why my evaluation cell is failing.

✔️ Awesome, I've got everything!

Great. Cell 5 now prints both class-level metrics plus the labeled confusion matrix.

ⓧ I'd like to double check the full code

# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

# Cell 3: Upload one CSV and validate its schema
uploaded = files.upload()
if len(uploaded) != 1:
    raise ValueError("Upload exactly one CSV file.")

filename = next(iter(uploaded))
if not filename.lower().endswith(".csv"):
    raise ValueError("The uploaded dataset must be a CSV file.")

df = pd.read_csv(filename)
required_columns = ["question", "answer", "sufficiency_label"]
missing_columns = [column for column in required_columns if column not in df.columns]
if missing_columns:
    raise ValueError(f"Missing required columns: {missing_columns}")

label_map = {
    "0": 0,
    "0.0": 0,
    "false": 0,
    "no": 0,
    "insufficient": 0,
    "1": 1,
    "1.0": 1,
    "true": 1,
    "yes": 1,
    "sufficient": 1,
}


def normalize_label(value):
    key = str(value).strip().lower()
    if key not in label_map:
        raise ValueError(
            f"Unsupported sufficiency_label value: {value!r}. "
            f"Supported values are: {sorted(label_map)}"
        )
    return label_map[key]


df = df[required_columns].copy()
df["question"] = df["question"].fillna("").astype(str).str.strip()
df["answer"] = df["answer"].fillna("").astype(str).str.strip()
df = df[(df["question"] != "") & (df["answer"] != "")].copy()
df["label"] = df["sufficiency_label"].map(normalize_label)
df = df.drop_duplicates(subset=["question", "answer", "label"]).reset_index(drop=True)

if len(df) < 10:
    raise ValueError("At least 10 usable labeled rows are required for this prototype.")
if set(df["label"].unique()) != {0, 1}:
    raise ValueError("The cleaned dataset must contain both sufficient and insufficient examples.")
if df["label"].value_counts().min() < 2:
    raise ValueError("Each class needs at least two usable examples.")

label_counts = df["label"].value_counts().sort_index().rename(
    index={0: "INSUFFICIENT", 1: "SUFFICIENT"}
)
print(f"Cleaned rows: {len(df)}")
print("\nSample rows:")
print(df[["question", "answer", "label"]].head(3).to_string(index=False))
print("\nClass counts:")
print(label_counts.to_string())

# Cell 4: Build text pairs, split the data, train, and expose accuracy's limitation

def build_model_text(question, answer):
    return f"Question: {question}\nAnswer: {answer}"


df["model_text"] = [
    build_model_text(question, answer)
    for question, answer in zip(df["question"], df["answer"])
]

train_df, test_df = train_test_split(
    df,
    test_size=0.20,
    random_state=42,
    stratify=df["label"],
)

model = Pipeline(
    steps=[
        (
            "tfidf",
            TfidfVectorizer(
                ngram_range=(1, 2),
                min_df=1,
                max_features=20000,
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                l1_ratio=0.0,
                class_weight="balanced",
                max_iter=1000,
            ),
        ),
    ]
)

model.fit(train_df["model_text"], train_df["label"])
test_predictions = model.predict(test_df["model_text"])
baseline_accuracy = accuracy_score(test_df["label"], test_predictions)

print(f"Training rows: {len(train_df)}")
print(f"Test rows: {len(test_df)}")
print(f"Baseline accuracy: {baseline_accuracy:.3f}")
print("Accuracy alone is incomplete because it does not show which class is failing.")

# Cell 5: Evaluate both classes and print a labeled confusion matrix
label_order = [0, 1]
label_names = ["INSUFFICIENT", "SUFFICIENT"]

report = classification_report(
    test_df["label"],
    test_predictions,
    labels=label_order,
    target_names=label_names,
    digits=3,
    zero_division=0,
)
print(report)

matrix = confusion_matrix(
    test_df["label"],
    test_predictions,
    labels=label_order,
)
matrix_df = pd.DataFrame(
    matrix,
    index=[f"actual_{name}" for name in label_names],
    columns=[f"predicted_{name}" for name in label_names],
)
print("Confusion matrix:")
print(matrix_df.to_string())
Read the class-level metrics

The INSUFFICIENT row shows how reliably the model handles answers that should be withheld. Recall is especially important because it measures how many actual insufficient answers the model catches.

How do the report columns differ?

  • Precision measures how often a prediction for that class is correct.
  • Recall measures how many actual examples from that class the model finds.
  • F1 balances precision plus recall in one score.
  • Support reports how many held-out examples belong to that class.
  • Locate the INSUFFICIENT row in the classification report.
  • Read its value under the recall column.
  • Compare that value with the recall shown for SUFFICIENT.

A lower INSUFFICIENT recall means more inadequate answers escaped that class. The model may approve some of those answers as sufficient.

Inspect false approvals in the matrix

The matrix uses actual classes as rows. Its columns show the classes predicted by the model.

Before you rerun Cell 5, predict which matrix position represents an insufficient answer approved as sufficient.

  • Rerun Cell 5 using its run control.
  • Find the actual_INSUFFICIENT row in the confusion table.
  • Read the count under the predicted_SUFFICIENT column.

You'll see report rows for both labels followed by a two-by-two confusion table. The actual_INSUFFICIENT plus predicted_SUFFICIENT cell is the false-sufficient count.

Why focus on false approvals?

A false approval lets an insufficient candidate answer continue through the gate. This cell exposes that risk directly before you experiment with live RAG answers.

That's the accuracy blind spot closed. Your notebook now separates missed insufficient answers from rejected sufficient answers.

Your evaluation now shows where the prototype succeeds plus where it fails. Next, you'll turn its sufficient-class probability into a reusable RAG quality-gate decision.

Create the RAG Quality Gate

Your class-level evaluation now shows how the prototype handles each sufficiency label. Those results reveal the cost of false approvals before the model reaches your existing RAG workflow.

The workflow still needs a callable decision instead of a table of metrics. In this step, you will convert the fitted Pipeline probability into a decision before saving the trusted model bundle.

In this step, get ready to:
  • Create a reusable function that assesses candidate RAG answers.
  • Check the gate against held-out evidence before trying a live RAG answer.
  • Package the fitted model for later integration.
Build the probability-based decision function

A probability threshold turns the model's sufficient-class probability into an actionable label. The default threshold of 0.5 approves answers at or above that cutoff.

  • Use the notebook's add-cell control to create a code cell beneath Cell 5.
  • Start Cell 6 with the reusable assessment function by pasting this code:
# Cell 6: Create the reusable RAG quality gate

def assess_answer(question, rag_answer, threshold=0.5):
    question = str(question).strip()
    rag_answer = str(rag_answer).strip()
    if not question or not rag_answer:
        raise ValueError("question and rag_answer must both contain text.")
    if not 0.0 <= threshold <= 1.0:
        raise ValueError("threshold must be between 0.0 and 1.0.")

    model_text = build_model_text(question, rag_answer)
    sufficient_index = list(model.classes_).index(1)
    sufficient_probability = float(
        model.predict_proba([model_text])[0][sufficient_index]
    )
    decision = (
        "SUFFICIENT" if sufficient_probability >= threshold else "INSUFFICIENT"
    )
    return {
        "decision": decision,
        "sufficient_probability": round(sufficient_probability, 4),
        "threshold": threshold,
    }

What does this function do?

  • The input checks reject blank questions or answers before they reach the model.
  • The threshold check keeps the cutoff within the valid probability range.
  • The build_model_text() call preserves the same question-answer format used during training.
  • The model.classes_ lookup finds the probability column for label 1.
  • The returned dictionary keeps the decision beside its probability and threshold.
  • Run Cell 6 using its run control.

The execution counter updates without a traceback. Your notebook now holds the callable assess_answer() function.

Cell 6 not finishing cleanly?

Confirm that the function begins at the left edge of the cell. Keep every line inside the function indented by four spaces.

If the Colab runtime restarted, rerun the earlier cells so model and build_model_text() exist again. Help me diagnose why assess_answer() is not being defined in my Colab notebook.

Test held-out and live RAG answers

A held-out row checks the gate on data excluded from training. A live answer then proves the same function can accept output from your existing RAG system.

  • Select Cell 6 again.
  • Append the held-out demo and live input flow beneath the function by pasting this code:
demo_row = test_df.iloc[0]
print("Held-out expected label:", label_names[int(demo_row["label"])])
print(assess_answer(demo_row["question"], demo_row["answer"]))

live_question = input("Paste a question from your RAG system, or press Enter to skip: ")
if live_question.strip():
    live_answer = input("Paste the RAG answer: ")
    print(assess_answer(live_question, live_answer))
else:
    print("Live RAG test skipped.")

How does the test flow work?

  • The held-out demo prints the dataset label before calling the new gate.
  • The first input prompt collects a question from your existing RAG system.
  • The second input prompt collects the candidate answer being judged.
  • Each assessment reports decision beside sufficient_probability.

Before you run Cell 6, predict whether the held-out decision will match its expected label.

  • Run Cell 6 using its run control.
  • Submit one question from your existing RAG system at the first prompt.
  • Submit its candidate answer at the second prompt.

You will see the held-out expected label followed by a dictionary containing decision, sufficient_probability, and threshold. A second dictionary shows the same fields for your live RAG answer.

Input prompt still waiting?

Colab pauses the cell while an input field is active. Submit text in the field beneath the cell so execution can continue.

If the live answer produces a surprising decision, compare its probability with the 0.5 threshold. Help me interpret my quality gate result.

Save the trusted model bundle

The fitted model depends on the same schema and label meanings used during training. The bundle stores those pieces together so a later integration can reconstruct the gate's contract.

  • Use the notebook's add-cell control to create a code cell beneath Cell 6.
  • Create Cell 7 with the model bundle and download flow by pasting this code:
# Cell 7: Save and download the trusted model bundle
model_bundle = {
    "model": model,
    "required_columns": required_columns,
    "label_map": label_map,
    "default_threshold": 0.5,
    "class_names": {0: "INSUFFICIENT", 1: "SUFFICIENT"},
}
model_filename = "rag_sufficiency_gate.pkl"
with open(model_filename, "wb") as model_file:
    pickle.dump(model_bundle, model_file)

print(f"Saved {model_filename}")
files.download(model_filename)

What does this bundle contain?

  • The model entry stores the fitted TF-IDF and logistic-regression Pipeline.
  • The schema and label entries preserve the data contract used by the notebook.
  • The default threshold records the cutoff used by assess_answer().
  • The pickle.dump() call serializes the bundle into rag_sufficiency_gate.pkl.
  • The files.download() call sends that file to your computer through the browser.

Keep the pickle trusted

A pickle can recreate Python objects when loaded. The file created by this notebook has a known source because you trained and serialized it yourself.

Files from unknown sources do not share that trust boundary. Your own downloaded bundle is the safe artifact for later integration.

Before you run Cell 7, predict which filename the notebook will confirm before the browser download begins.

  • Run Cell 7 using its run control.

You will see Saved rag_sufficiency_gate.pkl in the cell output. Your browser then starts downloading rag_sufficiency_gate.pkl.

Bundle not downloading?

Confirm that Cell 7 prints the saved filename before checking your browser's downloads. A completed save proves the file exists inside the Colab runtime.

If the runtime restarted, rerun Cells 1 through 6 before retrying Cell 7. Help me troubleshoot the model bundle download.

That is the integration layer complete. Your evaluated classifier now produces callable decisions and leaves you with a portable trusted bundle.

✔️ Awesome, I've got everything!

Great. Cells 6 and 7 now assess RAG answers before saving the fitted gate as rag_sufficiency_gate.pkl.

ⓧ I'd like to double check the full code

# Cell 1: Install pinned dependencies
%pip install -q pandas==3.1.0 scikit-learn==1.9.1

# Cell 2: Verify the environment and import the project APIs
import pickle
import sys

import pandas as pd
import sklearn
from google.colab import files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

print("Python:", sys.version.split()[0])
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)

# Cell 3: Upload one CSV and validate its schema
uploaded = files.upload()
if len(uploaded) != 1:
    raise ValueError("Upload exactly one CSV file.")

filename = next(iter(uploaded))
if not filename.lower().endswith(".csv"):
    raise ValueError("The uploaded dataset must be a CSV file.")

df = pd.read_csv(filename)
required_columns = ["question", "answer", "sufficiency_label"]
missing_columns = [column for column in required_columns if column not in df.columns]
if missing_columns:
    raise ValueError(f"Missing required columns: {missing_columns}")

label_map = {
    "0": 0,
    "0.0": 0,
    "false": 0,
    "no": 0,
    "insufficient": 0,
    "1": 1,
    "1.0": 1,
    "true": 1,
    "yes": 1,
    "sufficient": 1,
}


def normalize_label(value):
    key = str(value).strip().lower()
    if key not in label_map:
        raise ValueError(
            f"Unsupported sufficiency_label value: {value!r}. "
            f"Supported values are: {sorted(label_map)}"
        )
    return label_map[key]


df = df[required_columns].copy()
df["question"] = df["question"].fillna("").astype(str).str.strip()
df["answer"] = df["answer"].fillna("").astype(str).str.strip()
df = df[(df["question"] != "") & (df["answer"] != "")].copy()
df["label"] = df["sufficiency_label"].map(normalize_label)
df = df.drop_duplicates(subset=["question", "answer", "label"]).reset_index(drop=True)

if len(df) < 10:
    raise ValueError("At least 10 usable labeled rows are required for this prototype.")
if set(df["label"].unique()) != {0, 1}:
    raise ValueError("The cleaned dataset must contain both sufficient and insufficient examples.")
if df["label"].value_counts().min() < 2:
    raise ValueError("Each class needs at least two usable examples.")

label_counts = df["label"].value_counts().sort_index().rename(
    index={0: "INSUFFICIENT", 1: "SUFFICIENT"}
)
print(f"Cleaned rows: {len(df)}")
print("\nSample rows:")
print(df[["question", "answer", "label"]].head(3).to_string(index=False))
print("\nClass counts:")
print(label_counts.to_string())

# Cell 4: Build text pairs, split the data, train, and expose accuracy's limitation

def build_model_text(question, answer):
    return f"Question: {question}\nAnswer: {answer}"


df["model_text"] = [
    build_model_text(question, answer)
    for question, answer in zip(df["question"], df["answer"])
]

train_df, test_df = train_test_split(
    df,
    test_size=0.20,
    random_state=42,
    stratify=df["label"],
)

model = Pipeline(
    steps=[
        (
            "tfidf",
            TfidfVectorizer(
                ngram_range=(1, 2),
                min_df=1,
                max_features=20000,
                sublinear_tf=True,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                l1_ratio=0.0,
                class_weight="balanced",
                max_iter=1000,
            ),
        ),
    ]
)

model.fit(train_df["model_text"], train_df["label"])
test_predictions = model.predict(test_df["model_text"])
baseline_accuracy = accuracy_score(test_df["label"], test_predictions)

print(f"Training rows: {len(train_df)}")
print(f"Test rows: {len(test_df)}")
print(f"Baseline accuracy: {baseline_accuracy:.3f}")
print("Accuracy alone is incomplete because it does not show which class is failing.")

# Cell 5: Evaluate both classes and print a labeled confusion matrix
label_order = [0, 1]
label_names = ["INSUFFICIENT", "SUFFICIENT"]

report = classification_report(
    test_df["label"],
    test_predictions,
    labels=label_order,
    target_names=label_names,
    digits=3,
    zero_division=0,
)
print(report)

matrix = confusion_matrix(
    test_df["label"],
    test_predictions,
    labels=label_order,
)
matrix_df = pd.DataFrame(
    matrix,
    index=[f"actual_{name}" for name in label_names],
    columns=[f"predicted_{name}" for name in label_names],
)
print("Confusion matrix:")
print(matrix_df.to_string())

# Cell 6: Create the reusable RAG quality gate

def assess_answer(question, rag_answer, threshold=0.5):
    question = str(question).strip()
    rag_answer = str(rag_answer).strip()
    if not question or not rag_answer:
        raise ValueError("question and rag_answer must both contain text.")
    if not 0.0 <= threshold <= 1.0:
        raise ValueError("threshold must be between 0.0 and 1.0.")

    model_text = build_model_text(question, rag_answer)
    sufficient_index = list(model.classes_).index(1)
    sufficient_probability = float(
        model.predict_proba([model_text])[0][sufficient_index]
    )
    decision = (
        "SUFFICIENT" if sufficient_probability >= threshold else "INSUFFICIENT"
    )
    return {
        "decision": decision,
        "sufficient_probability": round(sufficient_probability, 4),
        "threshold": threshold,
    }


demo_row = test_df.iloc[0]
print("Held-out expected label:", label_names[int(demo_row["label"])])
print(assess_answer(demo_row["question"], demo_row["answer"]))

live_question = input("Paste a question from your RAG system, or press Enter to skip: ")
if live_question.strip():
    live_answer = input("Paste the RAG answer: ")
    print(assess_answer(live_question, live_answer))
else:
    print("Live RAG test skipped.")

# Cell 7: Save and download the trusted model bundle
model_bundle = {
    "model": model,
    "required_columns": required_columns,
    "label_map": label_map,
    "default_threshold": 0.5,
    "class_names": {0: "INSUFFICIENT", 1: "SUFFICIENT"},
}
model_filename = "rag_sufficiency_gate.pkl"
with open(model_filename, "wb") as model_file:
    pickle.dump(model_bundle, model_file)

print(f"Saved {model_filename}")
files.download(model_filename)

Secret mission

Add a Human Review Band

Add a review band that routes uncertain predictions to HUMAN_REVIEW. You will test the three-way gate with held-out data and at least three answers from your RAG system.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your Google Colab notebook uses a free hosted runtime with no ongoing project cost.

Resources you used:

  • A saved Google Colab notebook named rag_sufficiency_gate.ipynb containing your code and evaluation outputs.
  • A temporary hosted runtime session holding the uploaded CSV copy. The same session holds the fitted Pipeline.
  • A downloaded trusted pickle model bundle named rag_sufficiency_gate.pkl.

Keep everything running

No action is needed while you continue testing the quality gate.

  • Keep rag_sufficiency_gate.ipynb saved in its current location.
  • Keep rag_sufficiency_gate.pkl in its current download location.
  • Run the notebook cells from top to bottom when a future runtime starts.

Free Colab capacity can change between sessions. Your saved notebook remains the durable record of the project.

Pause - I'll come back to this later

Pausing ends active compute. Your saved notebook remains available with the downloaded model bundle.

  • Save rag_sufficiency_gate.ipynb from the notebook interface.
  • Stop any notebook cell that is still running.
  • Disconnect the current hosted runtime with the notebook's runtime controls.
  • Close the notebook browser tab.

Your notebook code remains available for a future session. Your downloaded model bundle remains ready for later integration.

Delete - I don't want to use this again

Deleting project artifacts can feel final. This option removes only the notebook resources created here.

Your original CSV remains untouched. Your existing RAG system remains untouched.

  • Disconnect the current hosted runtime with the notebook's runtime controls.
  • Move rag_sufficiency_gate.ipynb to trash with the notebook's file controls.
  • Empty the notebook storage trash for permanent removal.
  • Locate rag_sufficiency_gate.pkl through your browser's download history.
  • Delete rag_sufficiency_gate.pkl with your operating system's file browser.
  • Search for rag_sufficiency_gate.pkl with your operating system's file search.

You should no longer find the notebook in your notebook list. You should no longer find the model bundle in your operating system's search results.

Nice Work!

Nice Work!

You did it! Your Google Colab notebook now evaluates RAG answers with a trained TF-IDF plus logistic regression pipeline. Each prediction includes an actionable decision with its sufficient-class probability.

You've learned how to:

  • Clean and validate labeled question-answer data with visible counts for usable rows plus both classes.
  • Train a repeatable text-classification pipeline on a stratified split. Evaluate each label with precision, recall, F1, support, plus a labeled confusion matrix.
  • Run a RAG quality gate through assess_answer(). Save the fitted pipeline inside the trusted rag_sufficiency_gate.pkl bundle.
  • Secret Mission: Route uncertain predictions to HUMAN_REVIEW with an inclusive probability band.

Ready to quiz yourself?