LLM Evaluation Pipeline with LangSmith

Build a traced Q&A assistant with automated LLM evaluation in Python.

Introduction

30 Second Summary

Every time you change a prompt in an LLM application, you are flying blind unless you can measure the impact. Professional AI teams never ship prompt changes without automated scoring, the same way software teams never ship code without tests.

In this project, you will build a full evaluation pipeline that traces every LLM call, scores outputs automatically against a test dataset, and lets you compare different prompts side-by-side with real metrics.

What You'll Build

A Python Q&A assistant powered by Groq's Llama 3.3 70B model, instrumented with full LangSmith tracing and a professional evaluation pipeline.

By the end of this project, you'll have:

  • A fully traced LLM application where every call is visible in the LangSmith dashboard with inputs, outputs, latency, and token counts.
  • A working evaluation pipeline that automatically scores LLM outputs against a test dataset using custom evaluators.
  • Comparative experiment results showing how different prompts perform against each other on the same dataset.
  • Secret Mission: Build an automated quality gate that fails if evaluation scores drop below a threshold.

Are there any prerequisites?

You should be comfortable with basic Python (functions, dictionaries) and have some familiarity with calling LLM APIs. No prior experience with LangSmith or evaluation frameworks is needed.

Before We Start

Before you can evaluate an LLM, you need a way to see what it is actually doing under the hood. LangSmith gives you full visibility into every LLM call your application makes, including inputs, outputs, latency, and token counts.

In this step, you will set up your project environment, create free accounts for Groq and LangSmith, and verify that tracing works end-to-end.

In this step, get ready to:
  • Create a project directory with a Python virtual environment and install packages.
  • Create free Groq and LangSmith accounts and generate API keys.
  • Configure environment variables and verify that tracing is working.
Create your project directory and install packages

Every Python project needs its own isolated environment so package versions do not conflict with other projects on your machine.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Check your Python version by running this command:
python3 --version

✔️ I see version 3.10 or higher

Your Python version is compatible. Continue to the next instruction.

ⓧ I see a version lower than 3.10

This project requires Python 3.10 or higher. Visit python.org/downloads to download and install the latest version.

On macOS, the system Python can be outdated. After installing a newer version, use the full path (e.g. python3.12) or install via a version manager like pyenv.

ⓧ Command not found

Python is not installed or not in your PATH. Visit python.org/downloads to download and install Python 3.10 or higher.

After installing, close and reopen your terminal, then run python3 --version again to confirm.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir llm-eval-pipeline
cd llm-eval-pipeline
  • Create a Python virtual environment by running this command:
python3 -m venv .venv

What is a virtual environment?

A virtual environment is an isolated copy of Python that keeps your project's packages separate from your system Python. This prevents version conflicts between different projects.

  • Activate the virtual environment by running this command:

macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

You should see (.venv) appear at the beginning of your terminal prompt.

  • Install the required packages by running this command:
pip install groq langsmith openai

Why three packages?

The groq package connects to Groq's LLM API. The langsmith package provides tracing and evaluation tools. The openai package is used because Groq exposes an OpenAI-compatible interface, and LangSmith's wrap_openai utility wraps the OpenAI client to enable automatic tracing.

  • Verify the installation by running:
python -c "import groq; import langsmith; print('OK')"

You should see OK printed in your terminal.

Seeing an import error?

Make sure your virtual environment is active. You should see (.venv) in your prompt. If not, re-run the activate command from above.

If you still see errors, try running pip install groq langsmith openai again. Help me debug my package installation

Create your Groq and LangSmith accounts

You need two free accounts to power this project. Groq provides the LLM that answers questions. LangSmith records every call so you can evaluate quality.

  • Open console.groq.com in your browser.
  • Sign up for a free account (no credit card required).
  • Once logged in, navigate to API Keys in the left sidebar.
  • Click Create API Key.
  • Copy the key and save it somewhere safe. You will only see it once.

Why Groq?

Groq offers a completely free tier with no credit card required. It provides an OpenAI-compatible API endpoint, which means the same tracing tools that work with OpenAI also work with Groq out of the box.

  • Open smith.langchain.com in your browser.
  • Sign up for a free account using Google, GitHub, or email (no credit card required).
  • During signup, select the US data region when prompted.
  • Once logged in, click the Settings icon in the bottom left of the sidebar.
  • Select API Keys.
  • Click Create API Key.
  • Copy the key and save it somewhere safe. LangSmith only shows the key once.

What is LangSmith?

LangSmith is a platform for observing and evaluating LLM applications. It records every LLM call (a "trace") so you can inspect inputs, outputs, latency, and token usage after the fact. The Developer plan gives you 5,000 traces per month for free.

Configure environment variables and verify your setup

Your scripts need access to both API keys. You will store them in a .env file for reference and export them into your terminal session.

  • Open VS Code. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the llm-eval-pipeline folder on your Desktop, then click Open.
  • Create a new file by clicking File then New File.
  • Save it immediately as .env by pressing Cmd+S (macOS) or Ctrl+S (Windows) and typing .env as the filename.
  • Paste the following into your .env file, replacing the placeholder values with your actual keys:
GROQ_API_KEY=your-groq-api-key-here
LANGSMITH_API_KEY=your-langsmith-api-key-here
LANGSMITH_TRACING=true

What do these variables do?

GROQ_API_KEY authenticates your requests to Groq's LLM. LANGSMITH_API_KEY authenticates your requests to LangSmith's tracing server. LANGSMITH_TRACING=true tells the LangSmith SDK to automatically send traces for every LLM call.

  • Save .env.
  • Back in your terminal (with (.venv) active), export the variables into your current session by running:

macOS

export GROQ_API_KEY=your-groq-api-key-here
export LANGSMITH_API_KEY=your-langsmith-api-key-here
export LANGSMITH_TRACING=true

Windows

$env:GROQ_API_KEY="your-groq-api-key-here"
$env:LANGSMITH_API_KEY="your-langsmith-api-key-here"
$env:LANGSMITH_TRACING="true"

Replace the placeholder values

Make sure to replace your-groq-api-key-here and your-langsmith-api-key-here with the actual API keys you copied earlier.

  • Verify that everything works by running this test command:
python -c "
import os
from langsmith import wrappers
from openai import OpenAI

client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get('GROQ_API_KEY'),
        base_url='https://api.groq.com/openai/v1',
    )
)

response = client.chat.completions.create(
    model='llama-3.3-70b-versatile',
    messages=[{'role': 'user', 'content': 'Say hello in one sentence.'}],
)
print(response.choices[0].message.content)
"

You should see a short greeting printed in your terminal.

  • Open smith.langchain.com in your browser and look at the Traces section in the left sidebar.
  • You should see a new trace showing the LLM call you just made, including the input prompt, output response, latency, and token count.

Not seeing a trace in LangSmith?

Double check that you exported LANGSMITH_TRACING=true and LANGSMITH_API_KEY in the same terminal session where you ran the test. Environment variables do not persist between terminal windows.

If you selected a non-US region during LangSmith signup, you may also need to set LANGSMITH_ENDPOINT to your region's API URL. Help me troubleshoot my LangSmith tracing setup

✔️ Awesome, I've got everything!

Great. Double check you have saved your .env file.

ⓧ I'd like to double check the full code

Here is what your .env file should look like (with your own keys substituted):

GROQ_API_KEY=your-groq-api-key-here
LANGSMITH_API_KEY=your-langsmith-api-key-here
LANGSMITH_TRACING=true

Your environment is fully configured. Next up, you will build the traced Q&A assistant that answers Python questions.

Build the Traced Q&A Assistant

Your environment is ready with all the packages installed and API keys configured. Now it's time to build the actual application that your evaluation pipeline will test.

In this step, you'll create a Python Q&A assistant that answers technical questions using Groq's Llama model. Every call will be fully traced in LangSmith, giving you complete visibility into inputs, outputs, latency, and token usage.

In this step, get ready to:
  • Create a traced Q&A assistant that calls Groq through the OpenAI-compatible interface.
  • Verify nested traces appear in the LangSmith dashboard.
Create app.py with the wrapped OpenAI client

Groq exposes an OpenAI-compatible API endpoint. This means you can use the standard openai Python package pointed at Groq's servers. LangSmith's wrap_openai function wraps this client so every API call is automatically traced.

  • Create a new file called app.py in your project folder.
  • Add the imports, wrapped client, and system prompts by pasting this code:
import os

from langsmith import traceable, wrappers
from openai import OpenAI

# Create an OpenAI client pointed at Groq, wrapped for LangSmith tracing
oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)

# Two system prompts we'll compare later in evaluation
SYSTEM_PROMPT_V1 = "You are a helpful Python tutor. Answer the question clearly."

SYSTEM_PROMPT_V2 = (
    "You are a helpful Python tutor. Answer the question in 1-2 sentences. "
    "Be precise and technically accurate. Do not include examples unless asked."
)

What does this code do?

wrappers.wrap_openai() intercepts every call the OpenAI client makes and sends the details (input messages, output response, latency, token count) to LangSmith as a traced span. You don't need to change how you call the API at all.

base_url is set to https://api.groq.com/openai/v1 so the OpenAI SDK sends requests to Groq instead of OpenAI's servers. Groq processes them with its Llama model and returns responses in the same format.

SYSTEM_PROMPT_V1 and SYSTEM_PROMPT_V2 are two different instruction prompts you'll compare against each other in a later evaluation step.

Add the traceable Q&A function

The @traceable decorator from LangSmith creates a parent span around your entire function. When the wrapped OpenAI client makes its API call inside this function, that call appears as a nested child span. This gives you a two-level trace: function call containing the LLM call.

  • Below the system prompts in app.py, add the qa_assistant function and a test block:
@traceable
def qa_assistant(question: str, system_prompt: str = SYSTEM_PROMPT_V1) -> str:
    # Call the Groq LLM through the wrapped OpenAI client
    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
        temperature=0,
        max_tokens=256,
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    answer = qa_assistant("What is a list comprehension in Python?")
    print(answer)

What does this code do?

@traceable turns qa_assistant into a traced run. LangSmith records the function's input (the question) and output (the answer) as a parent span. The LLM call inside it becomes a child span automatically because the client is wrapped.

temperature=0 makes responses deterministic, which is important for evaluation. If the model gave different answers each time, your scores would be inconsistent.

The if __name__ == "__main__" block lets you run app.py directly to test with a sample question.

  • Save app.py.
  • Run the script to test your assistant:
python app.py

You should see a Python explanation printed in your terminal about list comprehensions.

Seeing an error?

If you see a connection or authentication error, make sure your environment variables are exported in this terminal session. Run echo $GROQ_API_KEY to confirm it's set.

If you see a module import error, confirm your virtual environment is active (you should see its name in your terminal prompt).

help me debug my app.py script

  • Now test with two more questions to generate additional traces. Change the question string in the if __name__ block to "What is the difference between a tuple and a list?" and run python app.py again.
  • Change the question to "What does the 'self' keyword mean in Python?" and run python app.py one more time.
  • Switch to your browser and open the LangSmith dashboard.
  • Click into your project's traces. You should see 3 traces listed, one for each question you ran.
  • Click on any trace. You should see a parent qa_assistant span with a nested child span for the LLM completion call inside it.

What should I see in LangSmith?

Each trace shows a tree structure. The outer span is qa_assistant (created by @traceable). Inside it, you'll see the LLM completion span (created by wrap_openai). Click on the LLM span to see the full messages array, the model's response, token count, and latency.

✔️ Awesome, I've got everything!

Great. Double check you've saved app.py and confirmed 3 traces are visible in your LangSmith dashboard.

ⓧ I'd like to double check the full code

Here's the complete app.py file:

import os

from langsmith import traceable, wrappers
from openai import OpenAI

oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)

SYSTEM_PROMPT_V1 = "You are a helpful Python tutor. Answer the question clearly."

SYSTEM_PROMPT_V2 = (
    "You are a helpful Python tutor. Answer the question in 1-2 sentences. "
    "Be precise and technically accurate. Do not include examples unless asked."
)


@traceable
def qa_assistant(question: str, system_prompt: str = SYSTEM_PROMPT_V1) -> str:
    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
        temperature=0,
        max_tokens=256,
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    answer = qa_assistant("What is a list comprehension in Python?")
    print(answer)

Your traced assistant is working. Every question you ask is now fully visible in LangSmith with inputs, outputs, latency, and token counts. Next up, you'll create an evaluation dataset to test this assistant systematically.

Create an Evaluation Dataset

Your Q&A assistant is now fully traced in LangSmith. Every question you ask shows up with inputs, outputs, latency, and token counts.

To properly evaluate your assistant, you need a consistent set of test questions with known-good answers. Running random questions by hand tells you nothing about whether a prompt change made things better or worse. An evaluation dataset gives you a fixed benchmark to measure against.

In this step, get ready to:
  • Create a Python script that uploads 8 Q&A examples to LangSmith.
  • Verify the dataset appears in the LangSmith UI with all examples visible.
Create the dataset script

Each example in a LangSmith dataset has two parts: the input (the question you send to your assistant) and the expected output (the reference answer you compare against). When you run evaluations later, LangSmith passes these reference answers to your evaluator functions so they can score how close the assistant's actual response is.

  • Create a new file called dataset.py in your project folder.
  • Add the following code to define and upload your evaluation dataset:
from langsmith import Client


def main():
    client = Client()

    dataset = client.create_dataset(
        dataset_name="Python QA",
        description="Python technical Q&A for evaluating our assistant",
    )

    examples = [
        {
            "inputs": {"question": "What is a list comprehension in Python?"},
            "outputs": {"answer": "A list comprehension is a concise way to create lists using a single line of code with the syntax [expression for item in iterable if condition]."},
        },
        {
            "inputs": {"question": "What is the difference between a tuple and a list?"},
            "outputs": {"answer": "Lists are mutable and use square brackets, while tuples are immutable and use parentheses."},
        },
        {
            "inputs": {"question": "What does the 'self' keyword mean in Python?"},
            "outputs": {"answer": "self refers to the current instance of a class and is used to access instance attributes and methods."},
        },
        {
            "inputs": {"question": "What is a decorator in Python?"},
            "outputs": {"answer": "A decorator is a function that takes another function as input and extends its behavior without modifying it, using the @syntax."},
        },
        {
            "inputs": {"question": "What is the GIL in Python?"},
            "outputs": {"answer": "The Global Interpreter Lock is a mutex that allows only one thread to execute Python bytecode at a time, limiting true parallelism in CPython."},
        },
        {
            "inputs": {"question": "What is a generator in Python?"},
            "outputs": {"answer": "A generator is a function that yields values one at a time using the yield keyword, allowing lazy evaluation and memory-efficient iteration."},
        },
        {
            "inputs": {"question": "What is the difference between '==' and 'is' in Python?"},
            "outputs": {"answer": "'==' checks value equality while 'is' checks identity, meaning whether two variables point to the same object in memory."},
        },
        {
            "inputs": {"question": "What are *args and **kwargs?"},
            "outputs": {"answer": "*args collects positional arguments into a tuple and **kwargs collects keyword arguments into a dictionary, allowing functions to accept variable numbers of arguments."},
        },
    ]

    client.create_examples(
        dataset_id=dataset.id,
        examples=examples,
    )

    print(f"Dataset '{dataset.name}' created with {len(examples)} examples.")


if __name__ == "__main__":
    main()

What does this code do?

Client() connects to LangSmith using the LANGSMITH_API_KEY environment variable you set earlier. No need to pass it explicitly.

create_dataset() creates an empty dataset container in LangSmith with the name "Python QA". create_examples() then populates it with 8 question/answer pairs that cover core Python topics.

Each example uses inputs (the question your assistant receives) and outputs (the reference answer evaluators compare against). LangSmith passes these as reference_outputs to your evaluator functions during evaluation runs.

  • Save dataset.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved dataset.py before moving on.

ⓧ I'd like to double check the full code

from langsmith import Client


def main():
    client = Client()

    dataset = client.create_dataset(
        dataset_name="Python QA",
        description="Python technical Q&A for evaluating our assistant",
    )

    examples = [
        {
            "inputs": {"question": "What is a list comprehension in Python?"},
            "outputs": {"answer": "A list comprehension is a concise way to create lists using a single line of code with the syntax [expression for item in iterable if condition]."},
        },
        {
            "inputs": {"question": "What is the difference between a tuple and a list?"},
            "outputs": {"answer": "Lists are mutable and use square brackets, while tuples are immutable and use parentheses."},
        },
        {
            "inputs": {"question": "What does the 'self' keyword mean in Python?"},
            "outputs": {"answer": "self refers to the current instance of a class and is used to access instance attributes and methods."},
        },
        {
            "inputs": {"question": "What is a decorator in Python?"},
            "outputs": {"answer": "A decorator is a function that takes another function as input and extends its behavior without modifying it, using the @syntax."},
        },
        {
            "inputs": {"question": "What is the GIL in Python?"},
            "outputs": {"answer": "The Global Interpreter Lock is a mutex that allows only one thread to execute Python bytecode at a time, limiting true parallelism in CPython."},
        },
        {
            "inputs": {"question": "What is a generator in Python?"},
            "outputs": {"answer": "A generator is a function that yields values one at a time using the yield keyword, allowing lazy evaluation and memory-efficient iteration."},
        },
        {
            "inputs": {"question": "What is the difference between '==' and 'is' in Python?"},
            "outputs": {"answer": "'==' checks value equality while 'is' checks identity, meaning whether two variables point to the same object in memory."},
        },
        {
            "inputs": {"question": "What are *args and **kwargs?"},
            "outputs": {"answer": "*args collects positional arguments into a tuple and **kwargs collects keyword arguments into a dictionary, allowing functions to accept variable numbers of arguments."},
        },
    ]

    client.create_examples(
        dataset_id=dataset.id,
        examples=examples,
    )

    print(f"Dataset '{dataset.name}' created with {len(examples)} examples.")


if __name__ == "__main__":
    main()
Upload the dataset to LangSmith

Now that the script is ready, run it to push your evaluation dataset to LangSmith. This makes the dataset available for automated evaluation runs in the next step.

  • Run the dataset script by executing this command in your terminal:
python dataset.py
  • You should see this output in your terminal:
Dataset 'Python QA' created with 8 examples.
  • Switch to the LangSmith dashboard in your browser.
  • Click Datasets & Experiments in the left-hand menu.
  • Click on the Python QA dataset to open it.
  • Verify you can see all 8 examples listed, each showing the input question and reference answer.

What should I see?

The dataset page shows an Examples tab listing all 8 rows. Each row displays the question under inputs and the expected answer under outputs. This is the data your evaluators will score against.

Seeing an error when you run the script?

If you see an authentication error, make sure your LANGSMITH_API_KEY environment variable is exported in your current terminal session. Re-run source .env or export the variable again.

If you see a "dataset already exists" error, the script ran successfully before. You can view the existing dataset in the LangSmith UI or delete it from the Datasets & Experiments page and re-run the script.

Help me debug my dataset upload error.

Your evaluation dataset is locked and loaded. Next up, you'll write custom evaluator functions that automatically score your assistant's answers against these reference examples.

Write Custom Evaluators and Run Your First Evaluation

You have a dataset of 8 Python Q&A examples sitting in LangSmith, ready to be tested against. Now you need a way to automatically score your assistant's answers.

In this step, you will write two custom evaluator functions and wire them into an evaluation pipeline that runs your assistant against every example in the dataset. One evaluator checks conciseness with simple logic. The other uses the LLM itself as a judge to assess factual correctness.

In this step, get ready to:
  • Write a conciseness evaluator and an LLM-as-a-judge correctness evaluator.
  • Wire up target functions and run your first evaluation experiment.
  • View per-example scores and aggregate metrics in the LangSmith dashboard.
Create the evaluator functions

Evaluators are functions that receive the assistant's output and the reference answer, then return a score. You will build two types: a deterministic length check and an LLM-powered correctness judge.

  • Create a new file called eval.py in your project folder.
  • Add the imports, the Groq client setup, and the conciseness evaluator by pasting this code:
import os

from langsmith import Client, traceable, wrappers
from openai import OpenAI

from app import qa_assistant, SYSTEM_PROMPT_V1, SYSTEM_PROMPT_V2

# Create a separate Groq client for the evaluator's LLM judge calls
oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)


def conciseness(outputs: dict, reference_outputs: dict) -> bool:
    """Check if the answer is reasonably concise compared to the reference."""
    return len(outputs["answer"]) < 3 * len(reference_outputs["answer"])

What does this code do?

The imports pull in your qa_assistant function and both system prompts from app.py. This keeps eval.py focused on evaluation logic only.

The conciseness evaluator receives outputs (what your assistant produced) and reference_outputs (the expected answer from the dataset). It returns True if the answer is less than 3 times the reference length. This is a simple heuristic that catches overly verbose responses.

  • Below the conciseness function, add the correctness evaluator:

@traceable(name="correctness_judge")
def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    """Use an LLM to judge if the answer is factually correct."""
    judge_prompt = f"""You are an evaluation judge. Compare the actual answer to the reference answer.
Question: {inputs["question"]}
Reference answer: {reference_outputs["answer"]}
Actual answer: {outputs["answer"]}

Is the actual answer factually correct and aligned with the reference? Reply with only 'true' or 'false'."""

    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0,
        max_tokens=10,
    )
    result = response.choices[0].message.content.strip().lower()
    return result == "true"

What is LLM-as-a-judge?

Instead of writing complex string-matching rules, you use another LLM call to judge whether the assistant's answer is factually aligned with the reference. The @traceable decorator means this judge call also appears in LangSmith as a traced span, so you can inspect exactly what the judge saw and decided.

The evaluator accepts three parameters: inputs (the original question), outputs (the assistant's answer), and reference_outputs (the expected answer). LangSmith automatically passes these from your dataset.

  • Save eval.py.

Seeing an import error?

If Python cannot find from app import qa_assistant, make sure app.py is in the same folder as eval.py. Both files must live in your project directory.

help me debug an import error in my eval.py file

Wire up the evaluation pipeline

Evaluators score individual outputs, but you also need target functions that bridge your dataset to your assistant. A target function receives a dataset example's inputs and returns the assistant's output in the format the evaluators expect.

  • Below the correctness function in eval.py, add the target functions:

def target_v1(inputs: dict) -> dict:
    """Baseline target using simple system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V1)
    return {"answer": answer}


def target_v2(inputs: dict) -> dict:
    """Improved target using detailed system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V2)
    return {"answer": answer}

Why two target functions?

Each target wraps qa_assistant with a different system prompt. This lets you run separate experiments against the same dataset and compare which prompt performs better. You will use target_v1 for your baseline experiment now, and target_v2 in the next step.

  • Add the main() function that runs both experiments:

def main():
    client = Client()

    print("Running baseline experiment...")
    results_v1 = client.evaluate(
        target_v1,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="baseline",
        max_concurrency=2,
    )
    print(results_v1)

    print("\nRunning detailed-prompt experiment...")
    results_v2 = client.evaluate(
        target_v2,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="detailed-prompt",
        max_concurrency=2,
    )
    print(results_v2)


if __name__ == "__main__":
    main()

What does client.evaluate() do?

The client.evaluate() function runs your target against every example in the named dataset, applies each evaluator to score the outputs, and uploads all results as an experiment to LangSmith. The experiment_prefix labels the experiment so you can identify it in the dashboard.

Setting max_concurrency=2 limits how many examples run in parallel. This keeps you well within Groq's free-tier rate limits.

  • Save eval.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved eval.py with all the code above.

ⓧ I'd like to double check the full code

import os

from langsmith import Client, traceable, wrappers
from openai import OpenAI

from app import qa_assistant, SYSTEM_PROMPT_V1, SYSTEM_PROMPT_V2

oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)


def conciseness(outputs: dict, reference_outputs: dict) -> bool:
    """Check if the answer is reasonably concise compared to the reference."""
    return len(outputs["answer"]) < 3 * len(reference_outputs["answer"])


@traceable(name="correctness_judge")
def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    """Use an LLM to judge if the answer is factually correct."""
    judge_prompt = f"""You are an evaluation judge. Compare the actual answer to the reference answer.
Question: {inputs["question"]}
Reference answer: {reference_outputs["answer"]}
Actual answer: {outputs["answer"]}

Is the actual answer factually correct and aligned with the reference? Reply with only 'true' or 'false'."""

    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0,
        max_tokens=10,
    )
    result = response.choices[0].message.content.strip().lower()
    return result == "true"


def target_v1(inputs: dict) -> dict:
    """Baseline target using simple system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V1)
    return {"answer": answer}


def target_v2(inputs: dict) -> dict:
    """Improved target using detailed system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V2)
    return {"answer": answer}


def main():
    client = Client()

    print("Running baseline experiment...")
    results_v1 = client.evaluate(
        target_v1,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="baseline",
        max_concurrency=2,
    )
    print(results_v1)

    print("\nRunning detailed-prompt experiment...")
    results_v2 = client.evaluate(
        target_v2,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="detailed-prompt",
        max_concurrency=2,
    )
    print(results_v2)


if __name__ == "__main__":
    main()
Run the evaluation and view results

With your evaluators and target functions wired up, you can now run the full evaluation pipeline. This will send each dataset question through your assistant, score the responses, and upload everything to LangSmith.

  • Run the evaluation script by executing this command:
python eval.py

What should I see?

The script runs 8 examples through your assistant twice (once per experiment), scoring each with both evaluators. You will see progress output in your terminal as each example is processed. This may take 30-60 seconds depending on Groq's response times.

When it finishes, the terminal prints a summary URL for each experiment. You can click these to go directly to the results in LangSmith.

  • Open the LangSmith dashboard at smith.langchain.com.
  • Click Datasets & Experiments in the left sidebar.
  • Click the Python QA dataset to open it.
  • Select the Experiments tab. You should see your baseline and detailed-prompt experiments listed.
  • Click the baseline experiment row to see per-example scores for conciseness and correctness.

You should see a table with each of your 8 dataset examples as rows, and columns showing the input, output, and feedback scores from both evaluators.

Seeing rate limit errors?

Groq's free tier has rate limits. If you see 429 errors, wait 60 seconds and run the script again. The max_concurrency=2 setting should prevent this in most cases, but heavy traffic on the free tier can still trigger limits.

help me troubleshoot rate limit errors when running my LangSmith evaluation with Groq

Your evaluation pipeline is live. Next up, you will compare the two experiments side-by-side to see which prompt strategy produces better answers.

Compare Prompts with a Second Experiment

A single experiment tells you how one prompt performs in isolation. The real insight comes from running the same dataset against a different prompt and comparing the results side-by-side.

Your eval.py already contains a second target function that uses SYSTEM_PROMPT_V2. In this step, you'll run both experiments and use the LangSmith comparison view to identify which prompt strategy performs better.

In this step, get ready to:
  • Run both experiments against the Python QA dataset.
  • Compare baseline and detailed-prompt results in the LangSmith dashboard.
Review your second target function

Before running the evaluation, take a moment to understand how your two target functions differ. The key change lives in app.py where SYSTEM_PROMPT_V2 adds explicit constraints.

  • In VS Code, open app.py and find the two system prompt constants near the top of the file:
SYSTEM_PROMPT_V1 = "You are a helpful Python tutor. Answer the question clearly."

SYSTEM_PROMPT_V2 = (
    "You are a helpful Python tutor. Answer the question in 1-2 sentences. "
    "Be precise and technically accurate. Do not include examples unless asked."
)

What is the difference between these prompts?

SYSTEM_PROMPT_V1 gives the model a simple instruction to answer clearly. It doesn't constrain length or format.

SYSTEM_PROMPT_V2 adds three specific constraints: answer in 1-2 sentences, be technically accurate, and skip examples unless asked. This should produce shorter, more focused answers that score higher on conciseness.

  • Now open eval.py and find the target_v2 function:
def target_v2(inputs: dict) -> dict:
    """Improved target using detailed system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V2)
    return {"answer": answer}

How does target_v2 use the new prompt?

target_v2 passes SYSTEM_PROMPT_V2 as the system_prompt argument to qa_assistant. This overrides the default SYSTEM_PROMPT_V1 so the model receives the more constrained instructions.

Run both experiments

Your eval.py script's main() function runs both experiments sequentially. It first evaluates target_v1 (baseline) and then evaluates target_v2 (detailed-prompt).

  • Run the evaluation script by running this command:
python eval.py

The script will take a minute or two to complete because it runs all 8 examples through the LLM twice (once per experiment), plus 8 additional LLM calls for the correctness judge in each experiment.

  • Wait for the script to finish. You should see output for both experiments printed to your terminal, including links to view each experiment in LangSmith.

Seeing rate limit errors?

Groq's free tier has rate limits. If you see 429 errors, wait 60 seconds and run the script again. The max_concurrency=2 setting in your code keeps requests low, but heavy usage on your account can still trigger limits.

help me fix rate limit errors when running eval.py

Compare experiments in the LangSmith dashboard

Now that both experiments have run, you can compare them side-by-side in the LangSmith UI to see which prompt strategy performed better.

  • In your browser, navigate to the LangSmith dashboard at smith.langchain.com.
  • Click Datasets & Experiments in the left-hand menu.
  • Click on the Python QA dataset to open it. This will open the Experiments tab.
  • You should see both experiments listed: one starting with baseline and one starting with detailed-prompt.
  • Select both experiments using the checkboxes on the left, then click Compare.

What am I looking at in the comparison view?

The comparison view shows each dataset example as a row. For each row, you can see the output from both experiments side by side, along with their evaluator scores. Look at the conciseness and correctness columns to see where the detailed prompt improved (or regressed) compared to the baseline.

You can toggle between Compact, Full, and Diff views using the icons in the top right of the comparison view. The Diff view highlights text differences between the two experiment outputs.

  • Click on any row to expand the details panel. Use the Details toggle in the panel to see feedback scores and metrics for that specific example.
  • Compare the aggregate scores across both evaluators. The detailed-prompt experiment should show improved conciseness scores because SYSTEM_PROMPT_V2 explicitly asks for 1-2 sentence answers.

Why might correctness scores differ?

The detailed prompt constrains the model to be brief. Shorter answers sometimes omit nuance that the correctness judge expects. This is the core tradeoff in prompt engineering: optimizing for one metric (conciseness) can affect another (correctness). The comparison view makes this tradeoff visible.

Not seeing both experiments?

Make sure eval.py ran to completion without errors. Both experiments need to finish for them to appear. Check your terminal output for any Python tracebacks or Groq API errors.

help me find my experiments in the LangSmith dashboard

✔️ Awesome, I've got everything!

Great. You can now see exactly how your two prompt strategies compare across the same dataset. This is the professional workflow for iterating on prompts with confidence.

ⓧ I'd like to double check the full code

Here is the complete eval.py that runs both experiments:

import os

from langsmith import Client, traceable, wrappers
from openai import OpenAI

from app import qa_assistant, SYSTEM_PROMPT_V1, SYSTEM_PROMPT_V2

oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)


def conciseness(outputs: dict, reference_outputs: dict) -> bool:
    """Check if the answer is reasonably concise compared to the reference."""
    return len(outputs["answer"]) < 3 * len(reference_outputs["answer"])


@traceable(name="correctness_judge")
def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    """Use an LLM to judge if the answer is factually correct."""
    judge_prompt = f"""You are an evaluation judge. Compare the actual answer to the reference answer.
Question: {inputs["question"]}
Reference answer: {reference_outputs["answer"]}
Actual answer: {outputs["answer"]}

Is the actual answer factually correct and aligned with the reference? Reply with only 'true' or 'false'."""

    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0,
        max_tokens=10,
    )
    result = response.choices[0].message.content.strip().lower()
    return result == "true"


def target_v1(inputs: dict) -> dict:
    """Baseline target using simple system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V1)
    return {"answer": answer}


def target_v2(inputs: dict) -> dict:
    """Improved target using detailed system prompt."""
    answer = qa_assistant(inputs["question"], system_prompt=SYSTEM_PROMPT_V2)
    return {"answer": answer}


def main():
    client = Client()

    print("Running baseline experiment...")
    results_v1 = client.evaluate(
        target_v1,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="baseline",
        max_concurrency=2,
    )
    print(results_v1)

    print("\nRunning detailed-prompt experiment...")
    results_v2 = client.evaluate(
        target_v2,
        data="Python QA",
        evaluators=[conciseness, correctness],
        experiment_prefix="detailed-prompt",
        max_concurrency=2,
    )
    print(results_v2)


if __name__ == "__main__":
    main()

And the complete app.py with both prompt versions:

import os

from langsmith import traceable, wrappers
from openai import OpenAI

oai_client = wrappers.wrap_openai(
    OpenAI(
        api_key=os.environ.get("GROQ_API_KEY"),
        base_url="https://api.groq.com/openai/v1",
    )
)

SYSTEM_PROMPT_V1 = "You are a helpful Python tutor. Answer the question clearly."

SYSTEM_PROMPT_V2 = (
    "You are a helpful Python tutor. Answer the question in 1-2 sentences. "
    "Be precise and technically accurate. Do not include examples unless asked."
)


@traceable
def qa_assistant(question: str, system_prompt: str = SYSTEM_PROMPT_V1) -> str:
    response = oai_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
        temperature=0,
        max_tokens=256,
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    answer = qa_assistant("What is a list comprehension in Python?")
    print(answer)

Secret mission

Build an Automated Quality Gate

Your evaluation pipeline scores prompts, but what if you could make it automatically reject a prompt that drops below a quality threshold? Build a gate that programmatically passes or fails based on evaluation scores, the same pattern professional AI teams use in CI/CD pipelines to block bad prompts from reaching production.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on free-tier services with no ongoing costs. Both Groq and LangSmith offer free plans that do not require a credit card, so nothing will charge you if left as-is.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Project folder on your Desktop containing app.py, dataset.py, eval.py, eval_with_gate.py, and .env.
  • Python virtual environment (inside the project folder).
  • LangSmith "Python QA" dataset with baseline, detailed-prompt, and quality-gate experiment traces.
  • Groq API key (free tier, no charges).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with different prompts or evaluators.

  • Your LangSmith data is retained for 14 days on the free Developer plan. After that, traces expire automatically.
  • Groq's free tier has no charges. Your API key remains active with rate limits only.
  • Your project files stay on your Desktop until you decide to remove them.

Pause - I'll come back to this later

Free up terminal resources while keeping your files and accounts intact.

  • Deactivate your virtual environment by running:
  • Close any open terminal sessions for this project.
  • Your LangSmith dataset and experiments will remain accessible when you return (within 14 days for traces).
deactivate

When you come back, reactivate your virtual environment and pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh. Since everything is free, this is purely about tidiness.

  • Delete the LangSmith dataset by running this Python command from your project folder:
python -c "from langsmith import Client; Client().delete_dataset(dataset_name='Python QA')"

What does this command do?

This uses the LangSmith Python SDK to delete the "Python QA" dataset. Deleting the dataset also removes the associated experiment runs (baseline, detailed-prompt, and quality-gate).

  • Revoke your Groq API key to prevent any future use:
  • Go to console.groq.com/keys.
  • Find the API key you created for this project.
  • Delete or revoke it from the key management table.
  • Delete your project folder from the Desktop. On macOS or Linux, run:
cd ~/Desktop
rm -rf llm-eval-pipeline

On Windows, run:

cd ~\Desktop
Remove-Item -Recurse -Force llm-eval-pipeline

This removes all Python files, the virtual environment, and the .env file containing your API keys.

Nice Work!

Nice Work!

Nice work! You just built a production-grade LLM evaluation pipeline that traces, scores, and compares prompt strategies using Python, Groq, and LangSmith.

You've learned how to:

  • Instrument an LLM application with full LangSmith tracing and build custom evaluators including an LLM-as-a-judge correctness scorer.
  • Create an evaluation dataset and run automated experiments that score LLM outputs for conciseness and correctness.
  • Compare two prompt strategies side-by-side in the LangSmith dashboard to identify which system prompt performs better.
  • Secret Mission: Implemented an automated quality gate that programmatically passes or fails based on evaluation scores.

Ready to quiz yourself?