Climate Fact-Checker with LLM Evals

Build a climate misinformation detector and evaluate it with LLM-as-judge metrics.

Introduction

30 Second Summary

Every day, climate claims flood social media feeds around the world. Telling real science from a misleading headline takes more than a gut feeling.

In this project, you will build a climate misinformation fact-checker powered by a local LLM that classifies real social media claims against scientific evidence. You will then build an evaluation pipeline that measures whether your AI system actually works.

What You'll Build

Feed a real climate claim into your terminal, watch your local AI classify it against a scientific abstract, then run an evaluation suite that shows you exactly where your system reasons well and where it falls short.

By the end of this project, you'll have:

  • A working climate fact-checker where you paste a social media claim and watch your local Qwen3 8B model classify it against peer-reviewed scientific evidence as Supports, Refutes, or Not Enough Information.
  • A multi-layered evaluation pipeline that compares exact match accuracy against LLM-as-judge scoring, exposing where simple metrics go blind and where smarter evaluation catches real nuance.
  • A measurable before-and-after improvement to your fact-checker's prompt, driven entirely by what your evaluations revealed.
  • Secret Mission: Build a custom G-Eval metric that encodes climate-specific scientific reasoning criteria into a reusable evaluation tool.

Do I need any prerequisites?

This project assumes basic Python knowledge. Before starting, make sure you have VS Code, Python, a Hugging Face account, and a Nebius account with API credits ready.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Local AI Lab

Your climate fact-checker needs two things to work: a local LLM that classifies claims, and an evaluation framework that measures how well the classifications hold up.

In this step, you will install Ollama to run an open-source LLM on your own machine, pull the Qwen3 8B model, set up your Python environment, and configure the API keys that power your evaluation pipeline.

In this step, get ready to:
  • Install Ollama and pull the Qwen3 8B model.
  • Create a Python virtual environment and install project dependencies.
  • Configure environment variables for Nebius and DeepEval.
Install Ollama and pull Qwen3 8B

Ollama is a tool that lets you run large language models locally on your machine. You will use it to run Qwen3 8B, an 8-billion-parameter open-source model that will power your fact-checker's classifications.

  • Check whether Ollama is already installed by running this command in your terminal:
ollama --version

✔️ I see a version number

Ollama is already installed. You are ready to pull the model.

ⓧ Command not found

Ollama is not installed yet. Install it now using the instructions for your operating system.

macOS

  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

After the script finishes, Ollama starts running in the background automatically.

Windows

  • Open PowerShell and install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

After installation, Ollama runs as a background app. The ollama command becomes available in any terminal window.

  • Confirm the install succeeded by running:
ollama --version

You should see a version number printed in the output.

Still seeing command not found?

Close your terminal and open a new one. The install may have updated your PATH, and the old terminal session does not pick up the change automatically.

On macOS, you can also download the installer directly from ollama.com/download/mac.

Help me troubleshoot my Ollama installation.

Now pull the Qwen3 8B model. This is a 5.2GB download, so it may take a few minutes depending on your internet speed.

  • Download the Qwen3 8B model by running:
ollama pull qwen3:8b

What is Qwen3 8B?

Qwen3 8B is an open-source model with 8.19 billion parameters, released under the Apache 2.0 license. It supports a 40K context window, which is more than enough to process a climate claim alongside a full scientific abstract.

At 5.2GB (Q4 quantization), it fits comfortably on most modern laptops. It runs entirely on your machine with no API costs.

Once the download completes, verify the model works by chatting with it.

  • Start an interactive chat with the model by running:
ollama run qwen3:8b

What should I see?

Ollama loads the model and shows a >>> prompt. Type a question like "Why is the sky blue?" and press Enter. The model generates a response right in your terminal.

The first response may take a moment while the model loads into memory. Subsequent responses are faster.

  • Type /bye and press Enter to exit the chat.

Model not responding or loading slowly?

Make sure the ollama pull qwen3:8b download finished completely. If it was interrupted, run the pull command again. Ollama resumes from where it left off.

If responses are very slow, check that no other large applications are using your RAM. Qwen3 8B needs several gigabytes of memory to run.

Help me troubleshoot my Qwen3 8B model in Ollama.

Create a Python virtual environment and install dependencies

Your project needs several Python packages that should live in an isolated environment. A virtual environment keeps these packages separate from your system Python.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir climate-fact-checker
cd climate-fact-checker
  • Create a Python virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

  • Install the project dependencies by running:
pip install deepeval datasets openai ollama

What are these packages?

  • deepeval is the evaluation framework you will use to measure your fact-checker with both deterministic and LLM-as-judge metrics.
  • datasets is the Hugging Face library for loading the ClimateCheck dataset.
  • openai provides the client library that DeepEval uses to talk to OpenAI-compatible APIs like Nebius.
  • ollama is the Python client for sending prompts to your local Ollama server.

The install may take a minute. Once it finishes, verify that DeepEval is working.

  • Confirm DeepEval installed correctly by running:
python -c "from deepeval.metrics import GEval; print('DeepEval ready')"

You should see DeepEval ready printed in your terminal.

Seeing an import error?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, run the activate command again.

If the install failed partway through, run pip install deepeval again on its own to see the full error output.

Help me fix my DeepEval installation.

Configure environment variables for Nebius and DeepEval

Your evaluation pipeline uses Nebius Token Factory as the judge model. Nebius provides access to larger models like Llama-3.3-70B-Instruct at very low cost. Using a more capable model as the judge produces more reliable evaluations than having your 8B model judge itself.

Why use Nebius instead of Ollama as the judge?

Using the same 8B model to both generate answers and judge those answers produces unreliable results. The model has the same blind spots in both roles.

Nebius gives you access to meta-llama/Llama-3.3-70B-Instruct as a judge. At $0.13 per million input tokens, the total cost for this project is roughly $0.05 and is covered by Nebius free credits.

You need to set five environment variables. Four tell DeepEval where to send its judge requests. One provides your API key.

  • Set the environment variables by running these commands in the same terminal where your virtual environment is active:

macOS

export NEBIUS_API_KEY=your-nebius-api-key-here
export USE_LOCAL_MODEL=1
export LOCAL_MODEL_BASE_URL=https://api.tokenfactory.nebius.com/v1/
export LOCAL_MODEL_API_KEY=$NEBIUS_API_KEY
export LOCAL_MODEL_NAME=meta-llama/Llama-3.3-70B-Instruct

Windows

$env:NEBIUS_API_KEY="your-nebius-api-key-here"
$env:USE_LOCAL_MODEL="1"
$env:LOCAL_MODEL_BASE_URL="https://api.tokenfactory.nebius.com/v1/"
$env:LOCAL_MODEL_API_KEY=$env:NEBIUS_API_KEY
$env:LOCAL_MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
  • Replace your-nebius-api-key-here with your actual Nebius Token Factory API key. You can find this in your Nebius account dashboard.

What do these variables do?

  • NEBIUS_API_KEY stores your Nebius authentication token.
  • USE_LOCAL_MODEL=1 tells DeepEval to use a custom model endpoint instead of the default OpenAI API.
  • LOCAL_MODEL_BASE_URL points DeepEval to the Nebius API, which is OpenAI-compatible.
  • LOCAL_MODEL_API_KEY passes your Nebius key to DeepEval for authentication.
  • LOCAL_MODEL_NAME specifies which model DeepEval uses as the judge.

These variables are session-only

Environment variables set with export (macOS) or $env: (Windows) only last for the current terminal session. If you close your terminal and come back later, you will need to set them again before running evaluations.

Before you move on, predict: do you think the DeepEval verification command from earlier will still print the same output, or will setting these environment variables change anything?

  • Verify your full setup by running:
python -c "from deepeval.metrics import GEval; print('DeepEval ready')"

You should see DeepEval ready printed again. The environment variables are picked up at evaluation time, not at import time, so the output stays the same.

Your local AI lab is ready. You have a local LLM that can classify text, a Python environment loaded with evaluation tools, and a connection to a cloud judge model. Next up, you will build the climate fact-checker that puts all of this to work.

Build the Climate Fact-Checker

Your local AI lab is set up. Ollama is running with Qwen3 8B, your Python environment is active, and your evaluation tools are configured.

Now it is time to build the thing you will be evaluating: a climate misinformation fact-checker. You will load real social media claims about climate change from the ClimateCheck dataset on Hugging Face and classify each one against a scientific abstract using your local LLM.

In this step, get ready to:
  • Load real climate misinformation data from the ClimateCheck dataset on Hugging Face.
  • Classify each claim against its paired scientific abstract using Qwen3 8B.
  • Run the fact-checker and see predictions printed alongside gold labels.
Load the ClimateCheck dataset from Hugging Face

The ClimateCheck dataset contains real social media claims about climate change paired with scientific abstracts. Each sample includes a claim, an abstract, and a gold annotation label: Supports, Refutes, or Not Enough Information.

What is the ClimateCheck dataset?

ClimateCheck is an active shared task hosted at ACL 2025 and LREC 2026. It contains 3,023 expert-annotated claim-abstract pairs across 782 unique claims.

Each sample pairs a social media claim (like "CO2 is actually beneficial for plants") with a scientific abstract that either supports, refutes, or does not address the claim.

  • In VS Code, create a new file called fact_checker.py in your project folder.
  • Add the imports and data-loading function by pasting the following code:
from ollama import chat
from datasets import load_dataset


def load_climate_data(split="train", num_samples=20):
    """Load ClimateCheck dataset from Hugging Face."""
    dataset = load_dataset("rabuahmad/climatecheck", split=split)
    samples = [dataset[i] for i in range(min(num_samples, len(dataset)))]
    return samples

What does this code do?

  • from ollama import chat imports the Ollama Python client you will use to talk to Qwen3 8B.
  • from datasets import load_dataset imports the Hugging Face function that downloads and caches datasets.
  • load_climate_data fetches the ClimateCheck dataset from Hugging Face using the identifier rabuahmad/climatecheck and returns a list of samples from the requested split.
  • Save fact_checker.py.
  • Test that the dataset loads correctly by running this command in your terminal:
python -c "from fact_checker import load_climate_data; samples = load_climate_data(num_samples=2); print(f'Loaded {len(samples)} samples'); print(f'First claim: {samples[0][chr(99)+chr(108)+chr(97)+chr(105)+chr(109)][:80]}...')"

The first time this runs, Hugging Face downloads and caches the dataset. You will see progress bars in your terminal. After that, you should see Loaded 2 samples followed by the first 80 characters of a real climate claim.

Seeing an error when loading the dataset?

  • If you see a ModuleNotFoundError for datasets or ollama, make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
  • If the download hangs or fails, check your internet connection. The dataset is hosted on Hugging Face and needs to be fetched once before it is cached locally.

Help me debug a dataset loading error in my climate fact-checker project.

Classify claims with your local LLM

With the dataset loading, you need a function that sends each claim-abstract pair to Qwen3 8B and gets back one of three labels: Supports, Refutes, or Not Enough Information.

  • In fact_checker.py, add the classify_claim function below the load_climate_data function:
def classify_claim(claim, abstract):
    """Classify whether a scientific abstract Supports, Refutes, or provides
    Not Enough Information about a climate change claim."""
    response = chat(
        model="qwen3:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "/no_think You are a scientific fact-checker. "
                    "Given a social media claim about climate change and a scientific abstract, "
                    "classify the relationship as one of: Supports, Refutes, or Not Enough Information. "
                    "Respond with ONLY the classification label, nothing else."
                ),
            },
            {
                "role": "user",
                "content": f"Claim: {claim}\n\nScientific Abstract: {abstract}",
            },
        ],
        options={"temperature": 0.0},
    )
    return response.message.content.strip()

What does this code do?

  • chat() sends a message to your locally running Qwen3 8B model via Ollama. The response comes back through response.message.content.
  • The system prompt starts with /no_think, a Qwen3 directive that disables step-by-step reasoning. This produces faster, more concise responses since you only need a classification label.
  • options={"temperature": 0.0} makes the model deterministic so you get consistent classifications across runs.
  • Save fact_checker.py.
Run the fact-checker on real climate claims

Time to bring everything together. You will add a main block that loads 5 training samples, classifies each one, and prints the results side by side with the expert-annotated gold labels.

  • At the bottom of fact_checker.py, add this main block:
if __name__ == "__main__":
    samples = load_climate_data(split="train", num_samples=5)

    print("Climate Misinformation Fact-Checker")
    print("=" * 50)

    for i, sample in enumerate(samples):
        prediction = classify_claim(sample["claim"], sample["abstract"])
        gold = sample["annotation"]
        match = prediction.lower().strip() == gold.lower().strip()

        print(f"\nSample {i + 1}:")
        print(f"  Claim: {sample['claim'][:100]}...")
        print(f"  Prediction: {prediction}")
        print(f"  Gold Label: {gold}")
        print(f"  Match: {'Yes' if match else 'No'}")

What does this code do?

  • if __name__ == "__main__": ensures this code only runs when you execute the file directly, not when another file imports it.
  • For each of the 5 samples, it sends the claim and abstract to classify_claim and compares the model's prediction against the expert gold label.
  • The match check uses .lower().strip() to handle minor formatting differences between the model's output and the gold label.
  • Save fact_checker.py.

Before you run this, think about what you expect to see. Do you think the model will get all 5 correct, or will some predictions disagree with the gold labels?

  • Run the fact-checker by executing this command:
python fact_checker.py

Each sample takes a few seconds as Qwen3 8B processes the claim-abstract pair locally. You should see 5 samples printed, each showing the claim, the model's prediction, the gold label, and whether they match.

What should I see?

Your output will look something like this (the exact claims depend on the dataset ordering):

Climate Misinformation Fact-Checker
==================================================

Sample 1:
  Claim: <a real climate claim>...
  Prediction: Supports
  Gold Label: Supports
  Match: Yes

Sample 2:
  Claim: <a different claim>...
  Prediction: Refutes
  Gold Label: Not Enough Information
  Match: No

Some predictions will match the gold label and some will not. That is expected. The mismatches are exactly what you will measure and improve in the next steps.

Not seeing any output or getting an error?

  • If you see a connection error, make sure Ollama is still running. Open a separate terminal and run ollama run qwen3:8b to verify the model responds.
  • If the script hangs for a long time on a single sample, the model is processing. Qwen3 8B running locally can take 5-15 seconds per classification depending on your hardware.
  • If predictions look unusual (long paragraphs instead of a single label), check that the system prompt in classify_claim starts with /no_think and includes the instruction to respond with only the classification label.

Help me debug my climate fact-checker script.

✔️ Awesome, I've got everything!

Great. Double check you have saved fact_checker.py.

ⓧ I'd like to double check the full code

Scroll up through the code chunks you added in this step and confirm each one is present in fact_checker.py, in this order:

  • The two import lines at the top (from ollama import chat and from datasets import load_dataset).
  • The load_climate_data function that calls load_dataset("rabuahmad/climatecheck", split=split).
  • The classify_claim function that calls chat() with the /no_think system prompt.
  • The if __name__ == "__main__": block that loads 5 samples, classifies each, and prints the results.

Your fact-checker is live. You can see Qwen3 8B classifying real climate claims against scientific evidence. But how good is it, really? Some predictions matched the gold labels and some did not. In the next step, you will build a deterministic evaluation that measures exactly how often the model gets it right.

Measure with Deterministic Evaluation

Your fact-checker is alive. It loads real climate claims from the ClimateCheck dataset and classifies them using Qwen3 8B. Now the critical question: how accurate is it?

Before reaching for complex evaluation tools, try the simplest approach first: does the model's output exactly match the expert-annotated gold label? The answer reveals raw accuracy and surfaces a limitation you would miss otherwise.

In this step, get ready to:
  • Build a deterministic evaluation script that compares model predictions to gold labels.
  • Run the evaluation on 20 test samples and observe the accuracy.
  • Analyze specific failure cases to discover the blind spot of exact match evaluation.
Build the exact match evaluation script

LLMs are unpredictable with formatting. Your model might return Supports, supports, or even SUPPORTS. A label normalizer handles these variations so your evaluation compares meanings, not capitalisation.

  • Create a new file called run_eval.py in your project folder and add the imports and normalize_label function:
import sys
import os
from fact_checker import classify_claim, load_climate_data


def normalize_label(label):
    """Normalize a classification label for comparison."""
    label = label.lower().strip()
    if "support" in label:
        return "supports"
    elif "refute" in label:
        return "refutes"
    elif "not enough" in label:
        return "not enough information"
    return label

What does this code do?

The normalize_label function converts any casing or phrasing variation into a consistent lowercase label. It uses substring matching ("support" in label) so that "Supports", "SUPPORTS", or even "supports the claim" all map to "supports".

The imports bring in classify_claim and load_climate_data from your fact-checker so this script can reuse the functions you already built.

  • Save run_eval.py and verify the function works by running this quick test:
python -c "from run_eval import normalize_label; print(normalize_label('SUPPORTS'))"

You should see supports printed in your terminal.

Seeing an ImportError?

Make sure your virtual environment is active and you saved run_eval.py in the same folder as fact_checker.py. Python needs both files in the same directory to resolve the import.

Help me debug this ImportError

  • Now add the run_exact_match function below normalize_label in run_eval.py. Start with the evaluation loop that classifies each sample and collects results:
def run_exact_match(classify_fn, num_samples=20):
    """Run exact match evaluation on ClimateCheck test samples."""
    samples = load_climate_data(split="test", num_samples=num_samples)

    correct = 0
    results = []

    for sample in samples:
        prediction = classify_fn(sample["claim"], sample["abstract"])
        gold = sample["annotation"]

        pred_norm = normalize_label(prediction)
        gold_norm = normalize_label(gold)
        is_correct = pred_norm == gold_norm

        if is_correct:
            correct += 1

        results.append({
            "claim": sample["claim"],
            "abstract": sample["abstract"][:200],
            "prediction": prediction,
            "gold": gold,
            "correct": is_correct,
        })

What does this code do?

This loop loads 20 samples from the ClimateCheck test split (not the training split you used in Step 2). For each sample, it calls your classify_claim function, normalizes both the prediction and the gold label, and checks if they match.

Each result is stored in a dictionary with the claim, a truncated abstract, the raw prediction, the gold label, and whether they matched. The classify_fn parameter lets you swap in different classifiers later without changing this function.

  • Still inside run_eval.py, add the reporting code to complete run_exact_match and add the __main__ block below the function:
    accuracy = correct / len(results) * 100

    print(f"\nAccuracy: {correct}/{len(results)} ({accuracy:.1f}%)\n")
    print("Results:")
    for r in results:
        status = "PASS" if r["correct"] else "FAIL"
        print(f"  [{status}] Pred: {r['prediction']:25s} | Gold: {r['gold']:25s}")

    return results, accuracy


if __name__ == "__main__":
    print("Climate Fact-Checker Evaluation")
    print("=" * 60)
    print("Mode: Deterministic (Exact Match)")
    run_exact_match(classify_claim, num_samples=20)

What does this code do?

The reporting section calculates accuracy as a percentage and prints each result with a [PASS] or [FAIL] tag. The :25s format pads each label to 25 characters so the table aligns neatly.

The __main__ block runs the evaluation when you execute the script directly. It passes your classify_claim function and evaluates 20 test samples.

  • Save run_eval.py.
Run the evaluation and observe your accuracy

Before you run this, what accuracy do you think Qwen3 8B will score on 20 climate claim classifications? Will it get most of them right, or will it struggle with ambiguous scientific evidence?

  • Run your evaluation script:
python run_eval.py

The script classifies all 20 test samples one by one, so it takes a minute or two. You should see an accuracy percentage followed by a table of PASS/FAIL results.

What should I see?

Your accuracy will likely land somewhere around 40-60%. The exact number depends on which samples the dataset returns, but the key observation is the FAIL rows. Some predictions disagree with the gold label.

Look at the FAIL results. Each one shows the model's prediction next to the expert's gold label. Keep those mismatches in mind. The next substep digs into whether they are genuine mistakes or something more interesting.

Script hanging or very slow?

Each sample requires a round trip to Qwen3 8B via Ollama. With 20 samples, expect 1-3 minutes total depending on your hardware. If nothing appears after 5 minutes, check that Ollama is still running by opening a new terminal and running ollama ps.

Help me troubleshoot slow Ollama responses

Dig into the failures

You have an accuracy number, but a number alone does not tell you what went wrong. Did the model hallucinate, or did it make a defensible call on ambiguous evidence that the expert simply labelled differently?

  • In run_eval.py, find the return results, accuracy line inside run_exact_match.
  • Add this failure analysis code directly above that line:
    # Failure analysis
    failures = [r for r in results if not r["correct"]]
    if failures:
        print(f"\n--- Failure Analysis ({len(failures)} mismatches) ---")
        for i, f in enumerate(failures[:3]):
            print(f"\nCase {i + 1}:")
            print(f"  Claim: {f['claim'][:120]}...")
            print(f"  Abstract: {f['abstract']}...")
            print(f"  Model said: {f['prediction']}")
            print(f"  Gold label: {f['gold']}")

What does this code do?

The failure analysis filters for mismatches and prints the details of up to 3 failures. For each one, you see the original claim, the beginning of the scientific abstract, what the model predicted, and what the expert labelled it.

This gives you the raw ingredients to judge for yourself whether each mismatch is a genuine error or an ambiguous case where both labels could be defensible.

  • Save run_eval.py.

Before you re-run the script, consider the failures you saw in the PASS/FAIL table. Do you think the model was genuinely wrong in every case, or could some of those disagreements be defensible?

  • Run the evaluation again to see the failure details:
python run_eval.py

Scroll down to the Failure Analysis section. Read through 2-3 failure cases and pay attention to the claim, the abstract excerpt, and both labels.

The blind spot of exact match

Some of those "failures" are genuine mistakes where the model hallucinated or ignored the evidence. But others are defensible calls on genuinely ambiguous evidence. The abstract might use qualifiers like "may" or "under certain conditions" that make both Supports and Not Enough Information reasonable.

Exact match evaluation treats all disagreements as equal failures. It cannot distinguish between a model that reasons well on ambiguous evidence and a model that fabricates its answer. This is the blind spot you will fix in the next step.

✔️ Awesome, I've got everything!

Make sure you have saved run_eval.py with all four sections: imports, normalize_label, run_exact_match (including failure analysis), and the __main__ block.

ⓧ I'd like to double check the full code

Your run_eval.py should contain the following sections in order. Scroll up and confirm each one is present:

  • Imports: import sys, import os, and from fact_checker import classify_claim, load_climate_data.
  • The normalize_label function that converts labels to lowercase with substring matching.
  • The run_exact_match function containing the evaluation loop, accuracy printing, results table, failure analysis, and return results, accuracy.
  • The if __name__ == "__main__": block that prints the header and calls run_exact_match(classify_claim, num_samples=20).

Your evaluation exposed the blind spot: exact match treats every disagreement as a failure, even when the model's reasoning is sound. Next up, you will bring in a smarter evaluator that can tell the difference.

Evaluate with LLM-as-Judge

Your exact match evaluation from the last step gave you a hard accuracy number. You saw specific cases where the model disagreed with the gold label. But here is the problem: exact match treats every disagreement as an equal failure.

Some of those "wrong" answers might actually demonstrate sound scientific reasoning on ambiguous evidence. Others might be genuine hallucinations. In this step, you will bring in a smarter evaluator that can tell the difference: an LLM-as-judge powered by Nebius.

In this step, get ready to:
  • Build DeepEval test cases from your ClimateCheck samples.
  • Run three LLM-as-judge metrics against your fact-checker.
  • Compare LLM-judge results against your exact match results and find where they disagree.
Build DeepEval test cases from ClimateCheck samples

Before DeepEval can evaluate your fact-checker, it needs each sample wrapped in a structured LLMTestCase object. Each test case maps your data into the fields that DeepEval metrics understand.

  • In run_eval.py, find the existing imports at the top of the file:
import sys
import os
from fact_checker import classify_claim, load_climate_data
  • Add these new imports directly below the existing ones:
from deepeval import evaluate
from deepeval.metrics import GEval, FaithfulnessMetric, HallucinationMetric
from deepeval.test_case import LLMTestCase, SingleTurnParams
  • Scroll down past the run_exact_match function. Add the start of a new run_llm_judge function below it by copying the following code:
def run_llm_judge(classify_fn, num_samples=10):
    """Run LLM-as-judge evaluation using Nebius via DeepEval."""
    samples = load_climate_data(split="test", num_samples=num_samples)

    test_cases = []
    for sample in samples:
        prediction = classify_fn(sample["claim"], sample["abstract"])

        test_case = LLMTestCase(
            input=(
                "Classify the relationship between this climate claim "
                "and scientific abstract as: Supports, Refutes, "
                "or Not Enough Information.\n\n"
                f"Claim: {sample['claim']}\n\n"
                f"Abstract: {sample['abstract']}"
            ),
            actual_output=prediction,
            expected_output=sample["annotation"],
            retrieval_context=[sample["abstract"]],
            context=[sample["abstract"]],
        )
        test_cases.append(test_case)

What does this code do?

Each LLMTestCase bundles everything DeepEval needs to judge one classification. The input is the full task prompt (claim + abstract). The actual_output is your model's prediction.

The expected_output is the gold label from the dataset. The retrieval_context and context both point to the abstract, which is the evidence source. DeepEval uses context for the HallucinationMetric and retrieval_context for the FaithfulnessMetric.

The function loads from the test split so you evaluate on the same data as your exact match run.

  • Save run_eval.py.
  • Confirm the new imports work by running:
python -c "from run_eval import run_llm_judge; print('LLM judge function loaded')"

You should see LLM judge function loaded printed with no errors.

Seeing an ImportError?

  • Make sure you added the new import lines below the existing ones, not replacing them.
  • Check that your virtual environment is active. The deepeval package was installed in Step 1.
  • Verify there are no typos in SingleTurnParams. It is imported from deepeval.test_case, not deepeval.metrics.

Still stuck? Help me debug my DeepEval import error in run_eval.py

Create evaluation metrics and run the judge

The function currently builds test cases but does nothing with them. Now you will define three metrics that evaluate your fact-checker from different angles, then run the evaluation.

  • In run_eval.py, add the following code directly below the test_cases.append(test_case) line inside run_llm_judge, at the same indentation level as the for loop:
    correctness_metric = GEval(
        name="Classification Correctness",
        evaluation_steps=[
            "Check if the actual output correctly identifies the relationship "
            "between the climate claim and the scientific evidence in the abstract.",
            "Consider that 'Not Enough Information' is valid when evidence is "
            "ambiguous or only partially related to the claim.",
            "Penalize clearly wrong classifications but accept borderline cases "
            "where the evidence could reasonably support multiple interpretations.",
        ],
        evaluation_params=[
            SingleTurnParams.INPUT,
            SingleTurnParams.ACTUAL_OUTPUT,
            SingleTurnParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )

    faithfulness_metric = FaithfulnessMetric(threshold=0.5)
    hallucination_metric = HallucinationMetric(threshold=0.5)

    evaluate(
        test_cases=test_cases,
        metrics=[correctness_metric, faithfulness_metric, hallucination_metric],
    )

What do these three metrics measure?

  • Classification Correctness is a custom GEval metric. Instead of checking for an exact string match, it uses evaluation_steps written in plain English to guide the judge LLM. The steps tell it to accept borderline cases where the evidence is genuinely ambiguous.
  • FaithfulnessMetric checks whether the classification is grounded in the abstract. A model that says "Supports" without any actual supporting evidence in the abstract would fail this.
  • HallucinationMetric catches fabricated reasoning. If the model invents facts that are not in the abstract to justify its answer, this metric flags it.

All three metrics score from 0 to 1. A score at or above the threshold of 0.5 counts as a pass.

Next, you need to update the main block so you can run the judge mode from the command line.

  • In run_eval.py, scroll down to the if __name__ == "__main__": block at the bottom of the file.
  • Find this code:
if __name__ == "__main__":
    print("Climate Fact-Checker Evaluation")
    print("=" * 60)
    print("Mode: Deterministic (Exact Match)")
    run_exact_match(classify_claim, num_samples=20)
  • Replace the entire if __name__ block with this updated version:
if __name__ == "__main__":
    use_judge = "--judge" in sys.argv

    print("Climate Fact-Checker Evaluation")
    print("=" * 60)

    if use_judge:
        print("Mode: LLM-as-Judge (Nebius)")
        run_llm_judge(classify_claim, num_samples=10)
    else:
        print("Mode: Deterministic (Exact Match)")
        run_exact_match(classify_claim, num_samples=20)

What changed in the main block?

The script now checks for a --judge flag in the command line arguments. If present, it runs the LLM-as-judge evaluation on 10 test samples. Without the flag, it still runs exact match on 20 samples, so your earlier evaluation mode is preserved.

  • Save run_eval.py.

Before you run the evaluation, think about this: you already know exact match gave you roughly 40-60% accuracy. Do you think the LLM judge will agree with exact match on every case, or will it score some "wrong" answers differently?

  • Run the LLM-as-judge evaluation:
python run_eval.py --judge

This will take a few minutes. DeepEval sends each test case to Nebius for evaluation across all three metrics. You will see a progress indicator followed by a results table with scores and reasons for each test case.

What should I see?

DeepEval prints a formatted table showing each test case with its metric scores (0 to 1) and pass/fail status. Each metric also provides a reason explaining why it scored the way it did. Look for cases where the Classification Correctness metric passes (score >= 0.5) even though the model's answer differs from the gold label.

Seeing an authentication or connection error?

  • Verify your NEBIUS_API_KEY environment variable is still set in this terminal session. If you opened a new terminal, you need to re-export it.
  • Check that USE_LOCAL_MODEL, LOCAL_MODEL_BASE_URL, LOCAL_MODEL_API_KEY, and LOCAL_MODEL_NAME are all set. DeepEval uses these to route judge calls to Nebius instead of OpenAI.
  • If you see rate limiting errors, wait a minute and try again. Nebius free credits have usage limits.

Still stuck? Help me debug my DeepEval LLM-as-judge connection to Nebius

✔️ Awesome, I've got everything!

Great. Double check you have saved run_eval.py with all your changes.

ⓧ I'd like to double check the full code

Your run_eval.py is now quite long, so scroll through and confirm these sections are present in this order:

  • Imports at the top: sys, os, fact_checker imports, then deepeval imports (evaluate, GEval, FaithfulnessMetric, HallucinationMetric, LLMTestCase, SingleTurnParams).
  • The normalize_label function.
  • The run_exact_match function (unchanged from Step 3).
  • The new run_llm_judge function with test case building, three metrics, and the evaluate() call.
  • The if __name__ == "__main__": block with the --judge flag check.

The fact_checker.py file should be unchanged from Step 2.

Compare LLM-judge results against exact match

Now that you have results from both evaluation approaches, compare them side by side. This is where the real insight emerges.

  • Look through the DeepEval output for test cases where the Classification Correctness metric scored above 0.5 (a pass). Cross-reference these against the PASS/FAIL table from your earlier exact match run.
  • Find at least one case where exact match said FAIL but the LLM judge passed. Read the judge's reason. Does the model's answer demonstrate sound reasoning even though it picked a different label than the gold standard?
  • Look for the reverse: a case where exact match said PASS but the Faithfulness or Hallucination metric flagged a problem. The model's label happened to match, but the judge noticed the reasoning was not grounded in the abstract.

Why do the two evaluations disagree?

Exact match is binary: either the label matches or it does not. It cannot tell you whether a mismatch is a defensible interpretation or a complete hallucination.

The LLM judge reads the claim, the abstract, and the prediction, then reasons about whether the classification makes sense given the evidence. On ambiguous climate claims, the same evidence can legitimately support more than one label. The judge can recognize this nuance. That is the gap exact match cannot bridge.

You now have two evaluation lenses on the same fact-checker. In the next step, you will use the failure patterns from both to actually improve your prompt and measure the difference.

Improve Your Fact-Checker with Eval Feedback

Your LLM-as-judge evaluation just revealed something powerful: not all "wrong" answers are equally wrong. Some mismatches were defensible reasoning on ambiguous evidence, while others were genuine hallucinations.

Evaluation is not just measurement. It is a feedback loop. In this step, you will read the specific failure reasons from your judge output, improve your fact-checker's prompt to address those weaknesses, and re-run both evaluation modes to measure the difference.

In this step, get ready to:
  • Review the LLM-judge failure reasons and identify the most common failure patterns.
  • Create an improved classify_claim_v2 function that addresses those weaknesses.
  • Re-run both evaluation modes on the improved version and compare before-and-after scores.
Review the LLM-judge failure patterns

The DeepEval output from your LLM-as-judge run included per-test-case scores and reasons for each metric. Those reasons are the feedback you need.

  • Scroll back through your terminal output from the python run_eval.py --judge run in the previous step.
  • Look at the test cases where the Classification Correctness metric gave a low score or where FaithfulnessMetric or HallucinationMetric flagged problems.
  • Read the reason text for each low-scoring case. Note which failure patterns repeat.

What patterns should I look for?

Three failure patterns tend to dominate in climate fact-checking with a basic prompt:

  • Ignoring scientific qualifiers: the model treats "may increase" or "under certain conditions" as definitive proof, when the evidence is actually inconclusive.
  • Treating correlation as causation: the abstract shows two things happened together, and the model concludes one caused the other.
  • Defaulting to "Supports" on ambiguous evidence: when the abstract only partially relates to the claim, the model picks "Supports" instead of "Not Enough Information."
Create the improved classifier function

Each failure pattern maps to a concrete instruction you can add to the system prompt. Instead of editing the original function, you will create a new classify_claim_v2 so you can compare both versions side by side.

  • In fact_checker.py, find the end of the classify_claim function (the return response.message.content.strip() line).
  • Add the following classify_claim_v2 function below it, leaving a blank line between the two functions:
def classify_claim_v2(claim, abstract):
    """Improved fact-checker with detailed scientific reasoning instructions."""
    response = chat(
        model="qwen3:8b",
        messages=[
            {
                "role": "system",
                "content": (
                    "/no_think You are an expert climate scientist and fact-checker. "
                    "Given a social media claim about climate change and a scientific abstract, "
                    "classify the relationship between them.\n\n"
                    "IMPORTANT GUIDELINES:\n"
                    "- 'Supports' means the abstract provides DIRECT evidence FOR the claim\n"
                    "- 'Refutes' means the abstract provides DIRECT evidence AGAINST the claim\n"
                    "- 'Not Enough Information' means the abstract does not directly address "
                    "the claim, or the evidence is inconclusive\n"
                    "- Pay close attention to scientific qualifiers like 'may', 'could', "
                    "'under certain conditions', 'in some regions'\n"
                    "- Do NOT assume causation from correlation\n"
                    "- If the abstract only partially relates to the claim, choose "
                    "'Not Enough Information'\n\n"
                    "Respond with ONLY the classification label: Supports, Refutes, "
                    "or Not Enough Information."
                ),
            },
            {
                "role": "user",
                "content": f"Claim: {claim}\n\nScientific Abstract: {abstract}",
            },
        ],
        options={"temperature": 0.0},
    )
    return response.message.content.strip()

What changed in the v2 prompt?

The original prompt was four lines: "You are a scientific fact-checker" with a generic instruction. The v2 prompt addresses each failure pattern directly:

  • It defines each label precisely, requiring DIRECT evidence rather than loose association.
  • It explicitly calls out scientific qualifiers like "may" and "could" that the v1 prompt ignored.
  • It tells the model to default to "Not Enough Information" on partial evidence, fixing the "Supports" bias.
  • Save fact_checker.py.
  • Verify the new function loads by running this quick test in your terminal:
python -c "from fact_checker import classify_claim_v2; print('classify_claim_v2 loaded')"

You should see classify_claim_v2 loaded printed with no errors.

Seeing an ImportError?

  • Make sure you saved fact_checker.py after adding the new function.
  • Check that the function name is exactly classify_claim_v2 with an underscore before v2.
  • Confirm the new function is placed after classify_claim and before the if __name__ == "__main__": block.

Help me fix my classify_claim_v2 import error.

✔️ Awesome, I've got everything!

Great. Double check you have saved fact_checker.py with both classify_claim and classify_claim_v2.

ⓧ I'd like to double check the full code

Your fact_checker.py should now contain the following sections in order. Scroll through your file and confirm each one is present:

  • Imports (from ollama import chat and from datasets import load_dataset).
  • The load_climate_data function (from Step 2).
  • The original classify_claim function (from Step 2).
  • The new classify_claim_v2 function (added in this step).
  • The if __name__ == "__main__": block at the bottom (from Step 2, unchanged).
Wire up the evaluation script and compare results

Your improved function exists, but run_eval.py does not know about it yet. You need to import it and add a --v2 flag so you can switch between the original and improved classifiers.

  • In run_eval.py, find this import line near the top of the file:
from fact_checker import classify_claim, load_climate_data
  • Add classify_claim_v2 to the import, so the line now looks like this:
from fact_checker import classify_claim, classify_claim_v2, load_climate_data

Next, the __main__ block needs a --v2 flag to choose which classifier to run.

  • Replace the entire if __name__ == "__main__": block at the bottom of run_eval.py with the following code:
if __name__ == "__main__":
    use_judge = "--judge" in sys.argv
    use_v2 = "--v2" in sys.argv

    classify_fn = classify_claim_v2 if use_v2 else classify_claim
    version_label = "v2 (improved)" if use_v2 else "v1 (original)"

    print(f"Climate Fact-Checker Evaluation - {version_label}")
    print("=" * 60)

    if use_judge:
        print("Mode: LLM-as-Judge (Nebius)")
        run_llm_judge(classify_fn, num_samples=10)
    else:
        print("Mode: Deterministic (Exact Match)")
        run_exact_match(classify_fn, num_samples=20)

What does this code do?

  • use_v2 = "--v2" in sys.argv checks whether you passed the --v2 flag when running the script.
  • classify_fn picks the right function based on the flag. Both run_exact_match and run_llm_judge already accept a classify_fn parameter, so no changes are needed inside those functions.
  • version_label prints which version is running so you can tell the outputs apart.
  • Save run_eval.py.

Now run the exact match evaluation with the improved v2 classifier by running this command:

  • Run the v2 exact match evaluation by running this command:
python run_eval.py --v2

You should see the header print v2 (improved) followed by the accuracy percentage and PASS/FAIL table. Compare this accuracy against the v1 results you saw in Step 3.

Seeing the same accuracy as v1?

Exact match is a blunt instrument. The improved prompt may show only a small accuracy change here because exact match cannot capture the quality of reasoning. The LLM-as-judge evaluation in the next run will reveal the deeper improvement.

If you see an error instead of results, make sure the header prints v2 (improved). If it prints v1 (original), the --v2 flag is not being recognized. Check that your __main__ block matches the code above.

Help me debug why my v2 evaluation is not working.

Now for the real test. Before you run the LLM-as-judge evaluation on v2, think about this: which metric do you expect to improve the most? The improved prompt adds explicit definitions and handles ambiguous evidence differently. Do you think it will score higher on Classification Correctness, Faithfulness, or Hallucination?

  • Run the LLM-as-judge evaluation on v2 by running this command:
python run_eval.py --judge --v2

DeepEval prints per-test-case scores and reasons for each metric. Compare these scores against the v1 LLM-judge results you saw in Step 4. You should see measurable improvement in at least one metric.

What should I compare?

Look at the per-test-case pass/fail counts for each metric across v1 and v2. The v2 prompt typically improves Classification Correctness on ambiguous cases because it handles scientific qualifiers and defaults to "Not Enough Information" more appropriately.

This is the eval-driven development cycle in action: run, measure, read the failure reasons, improve the prompt, measure again. Professionals repeat this loop until the metrics stabilize.

✔️ Awesome, I've got everything!

Great. Double check you have saved run_eval.py with the updated import and __main__ block.

ⓧ I'd like to double check the full code

Your run_eval.py should now contain the following sections in order. Scroll through your file and confirm each one is present:

  • Imports: sys, os, and the updated import line that includes both classify_claim and classify_claim_v2 from fact_checker.
  • The normalize_label function (from Step 3, unchanged).
  • The run_exact_match function (from Step 3, unchanged).
  • The run_llm_judge function (from Step 4, unchanged).
  • The updated if __name__ == "__main__": block with use_v2, classify_fn, and version_label (updated in this step).

You just closed the eval-driven development loop. You measured, identified weaknesses, improved the prompt, and proved the improvement with concrete metrics. Next up, you will take this even further by building a custom evaluation metric tailored specifically to climate science reasoning.

Secret mission

Secret Mission: Build a Custom Climate Reasoning Metric

The pre-built metrics you used in Step 4 evaluate general quality. But climate science has its own reasoning traps: cherry-picked findings, correlation mistaken for causation, scientific qualifiers like "may" or "under certain conditions" that completely change the strength of evidence. Can you build a custom G-Eval metric that encodes these domain-specific criteria and catches mistakes the generic metrics miss?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with no ongoing costs, though the Qwen3 8B model takes up 5.2GB of disk space.

Resources you used:

  • Qwen3 8B model stored by Ollama (5.2GB on disk).
  • Nebius Token Factory API for LLM-as-judge evaluations (estimated ~$0.05 total, covered by free credits).
  • Python virtual environment (venv) with installed packages.
  • Project files: fact_checker.py, run_eval.py, and eval_custom_metric.py.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with LLM evaluation or building on your fact-checker.

  • Ollama and the Qwen3 8B model stay on your machine for future projects. There is no ongoing cost.
  • Your Python virtual environment and project files remain ready for more experiments.
  • Ollama stays installed as a background service. You can use it with other models any time.

Pause - I'll come back to this later

Free up memory by stopping Ollama, but keep your model and files on disk so you can pick up where you left off.

  • Stop Ollama by quitting the Ollama application from your menu bar (macOS) or system tray (Windows). Your downloaded model stays on disk.
  • Deactivate your virtual environment by running:
deactivate
  • When you come back, start Ollama again and reactivate your virtual environment with source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).

Delete - I don't want to use this again

Remove all project resources and reclaim your disk space.

  • Delete the Qwen3 8B model to reclaim 5.2GB of disk space by running:
ollama rm qwen3:8b
  • Deactivate and delete your Python virtual environment by running:

macOS

deactivate
rm -rf venv

Windows

deactivate
rmdir /s /q venv
  • Delete the project files by running:

macOS

rm fact_checker.py run_eval.py eval_custom_metric.py

Windows

del fact_checker.py run_eval.py eval_custom_metric.py

What about Nebius?

Nebius Token Factory charges were incurred during your evaluation runs and are already complete. There is nothing to delete or stop on the Nebius side. Your API key environment variable was set for the terminal session and will be cleared when you close the terminal.

Nice Work!

Nice Work!

Nice work! You just built a climate misinformation fact-checker powered by a local open-source LLM and paired it with a professional-grade evaluation pipeline that proves your system actually works.

You've learned how to:

  • Built a climate misinformation fact-checker using Qwen3 8B running locally through Ollama, classifying real social media claims against scientific abstracts from the ClimateCheck shared task dataset.
  • Evaluated your system with deterministic exact match metrics, discovered their blind spot, then deployed LLM-as-judge evaluation through Nebius to distinguish defensible scientific reasoning from genuine hallucinations.
  • Closed the eval-driven development loop by using evaluation feedback to improve your prompt, then measured the before-and-after difference with concrete metric scores using DeepEval.
  • Secret Mission: Built a custom G-Eval metric encoding domain-specific climate science reasoning criteria, creating a reusable evaluation tool that goes beyond generic quality checks.

Ready to quiz yourself?