Grade Agent Runs with LangSmith

Use LangSmith and Jev to compare agent prompts with repeatable evals.

Introduction

30 Second Summary

A promising answer can make a wording change feel successful. A few lucky examples can hide a weaker result across the questions that matter.

In this project, you will build a small AI agent that records its runs in LangSmith. You will use Jev to judge two system prompts against the same dataset.

What You'll Build

You'll open matching score tables in LangSmith to see whether one prompt change improved the agent across the same ten questions.

By the end of this project, you'll have:

  • A trace view that lets you inspect the steps behind one agent answer.
  • A ten-run dataset that keeps both prompt versions on the same test.
  • A saved Jev evaluation that displays two score tables for the same dataset.
  • Secret Mission: Add edge-case questions to the dataset to test whether the winning prompt still leads.

Before We Start

Automatic grading only helps when your agent can send traces to LangSmith. This setup gives those traces a secure destination.

Your local Python environment also needs access to the model that answers each question. This walkthrough uses OpenAI as the model provider.

In this step, get ready to:
  • Prepare your LangSmith access for tracing.
  • Create an isolated Python environment.
  • Configure LangSmith tracing with model credentials.
Create your account and keys

LangSmith receives your traces through an API key. This project also runs evaluations, so your key needs permission to work with evaluation resources.

  • Visit the LangSmith sign-in page.
  • Create your free account with Google, GitHub, or email.
  • Complete the sign-in flow to reach your LangSmith workspace.
  • Select Settings.
  • Select API Keys.
  • Select the Personal key option.
  • Set an expiration that covers the time you plan to keep this project.
  • Click Create API Key.

Why use a personal key?

A tracing key can submit traces. A Personal Access Token can also work with the evaluation resources you create later.

The full key appears once, so take a moment to store it before closing the dialog.

  • Copy the LangSmith key into your password manager.
  • Close the key dialog after the key is stored.
  • Visit the OpenAI API Keys page.
  • Sign in to the OpenAI Platform.
  • Create a secret key for this project with the page's key creation control.
  • Copy the OpenAI key into your password manager.

Keep keys private

Never paste either key into a screenshot or a chat message. You only need the two saved values when you configure the terminal later in this step.

Prepare your Python environment

A virtual environment keeps this project's packages separate from the rest of your computer. First, confirm that Python is available from your terminal.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Terminal on macOS or PowerShell on Windows.
  • Press Enter to open the terminal.

✔️ I see a Python version

macOS

  • Check your installed Python version by running this command:
python3 --version
What does this check do?

The command asks your Python installation to print its version. A version number confirms that the interpreter is available from Terminal.

You should see a Python version number in the output.

Windows

  • Check your installed Python version by running this command:
python --version
What does this check do?

The command asks your Python installation to print its version. A version number confirms that the interpreter is available from PowerShell.

You should see a Python version number in the output.

ⓧ Command not found

macOS

  • Visit the official Python downloads for macOS.
  • Download the latest supported macOS installer.
  • Run the downloaded installer package.
  • Complete the standard installation steps.
  • Reopen Terminal after the installation finishes.
  • Check Python again by running this command:
python3 --version
What should you see?

The command should now print a Python version number. This confirms that the installer added Python to your shell path.

Still missing Python?

Close every Terminal window before opening a fresh one. If the command still fails, help me fix Python on macOS.

Windows

  • Visit the official Python downloads for Windows.
  • Download the Python Install Manager.
  • Run the downloaded installer.
  • Click Install when the installer asks for confirmation.
  • Reopen PowerShell after the installation finishes.
  • Check Python again by running this command:
python --version
What should you see?

The command should now print a Python version number. This confirms that the Python Install Manager made the interpreter available to PowerShell.

Still missing Python?

Close every PowerShell window before opening a fresh one. If the command still fails, help me fix Python on Windows.

Your Python interpreter is available. Now you can isolate the packages used by this project.

macOS

  • Move to your Desktop by running this command:
cd ~/Desktop

What does this command do?

The command moves Terminal to your Desktop. Your project folder will be easy to find there.

  • Create the ls-quickstart folder by running these commands:
mkdir ls-quickstart
cd ls-quickstart

What do these commands do?

  • The first command creates the ls-quickstart folder.
  • The second command moves Terminal into that folder.
  • Confirm that your Terminal prompt now ends inside ls-quickstart.
  • Create the virtual environment with venv by running these commands:
python3 -m venv .venv
source .venv/bin/activate

What do these commands do?

  • The first command creates an isolated Python environment in .venv.
  • The second command activates that environment for the current Terminal session.
  • Check your Terminal prompt for (.venv) at the beginning.

Virtual environment not active?

Confirm that Terminal is inside ls-quickstart before running the activation command. If the prompt still does not change, help me activate the environment.

  • Install the LangSmith SDK with the OpenAI client by running this command:
pip install -U langsmith openai

What does this command do?

The command installs the two Python packages inside your active virtual environment. The upgrade flag requests the latest versions available to your environment.

  • Confirm that the installation output finishes without a package installation failure.

Package installation failed?

Confirm that your prompt starts with (.venv). For a network or package error, help me install the project packages.

Windows

  • Move to your Desktop by running this command:
Set-Location "$env:USERPROFILE\Desktop"

What does this command do?

The command moves PowerShell to the Desktop folder for your Windows account. Your project folder will be easy to find there.

  • Create the ls-quickstart folder by running these commands:
mkdir ls-quickstart
cd ls-quickstart

What do these commands do?

  • The first command creates the ls-quickstart folder.
  • The second command moves PowerShell into that folder.
  • Confirm that your PowerShell prompt now ends inside ls-quickstart.
  • Create the virtual environment with venv by running these commands:
python -m venv .venv
.\.venv\Scripts\Activate.ps1

What do these commands do?

  • The first command creates an isolated Python environment in .venv.
  • The second command activates that environment for the current PowerShell session.
  • Check your PowerShell prompt for (.venv) at the beginning.

Virtual environment not active?

PowerShell may block local activation scripts. If that happens, help me activate the environment.

  • Install the LangSmith SDK with the OpenAI client by running this command:
pip install -U langsmith openai

What does this command do?

The command installs the two Python packages inside your active virtual environment. The upgrade flag requests the latest versions available to your environment.

  • Confirm that the installation output finishes without a package installation failure.

Package installation failed?

Confirm that your prompt starts with (.venv). For a network or package error, help me install the project packages.

Connect tracing to your workspace

Environment variables give your Python process access to tracing settings without placing credentials in application code. These values stay in the current terminal session.

macOS

  • Replace both secret placeholders with the matching keys before running these commands:
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langsmith-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

What do these settings control?

The tracing setting tells LangSmith integrations to record runs. The two key variables authenticate your tracing requests plus your model requests.

Before you run the check, predict whether all three values are available to Python.

  • Verify the three variables without printing either secret by running this command:
python -c "import os; print(all(os.getenv(name) for name in ('LANGSMITH_TRACING', 'LANGSMITH_API_KEY', 'OPENAI_API_KEY')))"

What should you see?

You should see True. The check only reports whether every value exists.

Seeing False?

One of the three environment variables is empty or missing. Run the export commands again after replacing both placeholders, or help me find the missing setting.

Windows

  • Replace both secret placeholders with the matching keys before running these commands:
$Env:LANGSMITH_TRACING = "true"
$Env:LANGSMITH_API_KEY = "<your-langsmith-api-key>"
$Env:OPENAI_API_KEY = "<your-openai-api-key>"

What do these settings control?

The tracing setting tells LangSmith integrations to record runs. The two key variables authenticate your tracing requests plus your model requests.

Before you run the check, predict whether all three values are available to Python.

  • Verify the three variables without printing either secret by running this command:
python -c "import os; print(all(os.getenv(name) for name in ('LANGSMITH_TRACING', 'LANGSMITH_API_KEY', 'OPENAI_API_KEY')))"

What should you see?

You should see True. The check only reports whether every value exists.

Seeing False?

One of the three environment variables is empty or missing. Run the PowerShell assignments again after replacing both placeholders, or help me find the missing setting.

That is the setup locked in. Your active terminal can now authenticate with LangSmith plus OpenAI.

  • Keep the active terminal open for the next step.
  • Return to your signed-in LangSmith browser tab.
  • Use the top-left control to confirm that your workspace is available.
  • Select Tracing from the LangSmith interface.
  • Confirm that the tracing workspace loads without another sign-in prompt.

You can now reach the workspace where your agent's runs will appear. Your first trace creates the default tracing project during the next step.

Your local environment is ready. Next up, you will write the small agent that starts filling LangSmith with traces.

Build and Trace Your Agent

Your local environment is ready. A prompt change can feel better while still performing worse.

Tracing with LangSmith turns each answer into evidence. In this step, you will build a small agent that creates a trace for you to inspect.

In this step, get ready to:
  • Build a small question-answering agent around one initial system prompt.
  • Run one question through the agent to create a trace.
  • Inspect the completed trace in LangSmith.
Create the baseline agent

The first version needs to stay simple because it becomes your baseline. A small Python question-answering agent gives later prompt versions a fair starting point.

What must the baseline agent do?

  • It uses one initial system prompt for every question.
  • It accepts one user question at a time.
  • It sends the question to the LLM provider from your prepared environment.
  • It prints the model response in your local output.
  • It sends the run data to a named LangSmith tracing project.
  • Choose a name for the LangSmith tracing project that receives this baseline run: your tracing project name.
  • Write one Python file inside the local project location from your prepared environment to satisfy the five baseline requirements above.
  • Save the Python file.
Generate the first traced run

A local answer proves that the provider call works. A traced run proves that the measurement layer can capture the agent's activity.

  • Choose one short factual question for the agent.
  • Enter the question through the agent's input method.

Before you run the agent, make a prediction about whether one local answer also produces a remote trace.

  • Run the agent using the run method from your prepared Python environment.

Your local output should show one complete answer to the question. That first result gives your future evaluations a working baseline.

No answer from the agent?

  • Confirm that the configured provider key is available to the same Python process.
  • Check that the initial system prompt reaches the model.
  • Check that the user question reaches the model.

If the run still fails, help me diagnose why my agent did not return an answer.

Inspect the completed trace

A trace records one run from input to output. It lets you inspect the steps that produced the final response.

Before you return to LangSmith, make a prediction about whether the local answer is already waiting in the tracing project.

  • Return to the LangSmith project workspace from earlier.
  • Open your tracing project name.

You should see a completed trace from the question-answering agent.

  • Open the newest completed trace.
  • Inspect the recorded initial system prompt.
  • Inspect the recorded user question.
  • Inspect the recorded model response.
  • Review each recorded step inside the trace.

The opened trace should show the same question from your local run. It should also show the answer that appeared in your local output.

You now have your first measurable baseline. The trace shows exactly what the agent received before it produced its answer.

Trace missing from LangSmith?

  • Confirm that the agent's tracing destination matches your tracing project name.
  • Confirm that tracing remains enabled in the prepared environment.
  • Run the agent once more.
  • Refresh the tracing project.

If the trace is still missing, help me diagnose why my LangSmith trace is missing.

Your agent now leaves a trace every time it answers a question. Next up, you will collect more runs to create a reusable dataset.

Create a Dataset from Agent Runs

Your question-answering agent can now answer a prompt. LangSmith records each answer as a trace.

One trace reveals how the agent handled one question. In this step, you will capture ten runs inside a reusable dataset.

In this step, get ready to:
  • Run the agent against ten fixed question inputs.
  • Confirm that LangSmith received ten new completed runs.
  • Create a LangSmith dataset that pairs every question with its agent output.
Run the agent ten times

A useful baseline covers several types of question. Keeping this set fixed gives every future prompt the same test.

Why use fixed questions?

A fixed input set holds the questions constant. Later score changes can be connected to the prompt version under test.

  • Question 1: Why is the sky blue?.
  • Question 2: What causes ocean tides?.
  • Question 3: What is photosynthesis?.
  • Question 4: Explain recursion in one paragraph..
  • Question 5: What does HTTP status code 404 mean?.
  • Question 6: Convert 25 degrees Celsius to Fahrenheit..
  • Question 7: Who wrote Pride and Prejudice?.
  • Question 8: Name two benefits of unit tests..
  • Question 9: What is the largest planet in the solar system?.
  • Question 10: Summarize the water cycle in three steps..
  • Switch back to the local agent interface from earlier.
  • Leave the initial system prompt unchanged.
  • Submit every question from the list through the agent.
  • Wait for each answer before sending the next question.

Good progress. Your agent has now answered the full ten-question batch with one consistent system prompt.

Confirm the ten traced runs

Each completed answer should produce a new traced run. Checking the batch now prevents a missing example from weakening the dataset.

  • Return to the LangSmith workspace from earlier.
  • Select Tracing in the left sidebar.
  • Select the tracing project used by your agent.
  • Select the Runs tab.

Before you refresh, ask yourself this: how many new completed runs should the batch produce? Hold that number in mind.

  • Refresh the runs table.
  • Count the ten newest completed runs.

You should count exactly ten new completed runs. Each run represents one question from your fixed set.

Missing One of the Ten Runs?

  • Refresh the Runs tab after the final agent response completes.
  • Confirm that you returned to the same tracing project from earlier.
  • Check that LangSmith tracing remains enabled in your local environment.

Help me diagnose why one of my ten agent runs is missing from LangSmith.

Create the dataset

Converting runs into examples preserves the input with its corresponding output. The resulting dataset becomes the shared baseline for later evaluations.

  • Select the checkbox beside each of the ten newest completed runs.
  • Click Add to Dataset at the bottom of the page.
  • Click New Dataset in the dataset picker.
  • Enter Agent Evaluation Dataset in the Name field.
  • Click Create to create the dataset from the selected runs.

Your ten runs now belong to one reusable dataset. The final check confirms that every selected run became an example.

Before you inspect the dataset, ask yourself this: how many examples should appear? Keep your prediction in mind.

  • Select Datasets & Experiments in the left sidebar.
  • Select Agent Evaluation Dataset from the dataset list.
  • Select the Examples tab.

You should see ten examples in the dataset. This count matches the ten runs you selected.

  • Select one example from the table.

You should see one stored agent input with its corresponding agent output. Strong work. Your evaluation baseline is ready for repeatable testing.

Dataset Count Below Ten?

  • Return to the Runs tab in your tracing project.
  • Select the missing completed run.
  • Click Add to Dataset.
  • Choose Agent Evaluation Dataset from the dataset picker.

Help me fix a LangSmith dataset that contains fewer examples than the selected runs.

Your ten-example baseline is fixed. Next up, you will add a judge that scores every run against this same test set.

Evaluate Runs with Jev

Your ten traced answers now form a fixed test set in LangSmith. Reading them by eye still leaves answer quality as a feeling.

Jev supplies a consistent judge through a typed score for each response. You will use those scores to establish the initial prompt's baseline.

In this step, get ready to:
  • Connect Jev to LangSmith.
  • Define a rubric that grades answer quality.
  • Produce a score table for the ten dataset examples.
Connect Jev to LangSmith

Jev is a System One model from TypeSafe. LangSmith calls it through a TypeSafe API key stored as a workspace secret.

Protect the TypeSafe key

This key is a live credential that stays inside LangSmith's secret store. Keep it out of screenshots.

  • Visit the TypeSafe console to access your account.
  • Sign in to your TypeSafe account.
  • Create an API key from the console's key area.
  • Copy the key while it is visible.
  • Switch back to LangSmith.
  • Click Settings in the sidebar.
  • Select Integrations.
  • Open Provider secrets.

You are now in the workspace area that keeps provider credentials away from traces.

  • Click + Secret.
  • Select TypeSafe as the provider.
  • Paste the key into the TYPESAFE_API_KEY field.
  • Submit the form to store the secret.

That connection is done. LangSmith can now call Jev without exposing the key in your agent.

  • Return to the ten-example dataset from earlier.
  • Select the Evaluators tab.
  • Click + Evaluator.
  • Select LLM-as-a-Judge Evaluator under Create from scratch.
  • Enter Initial Answer Quality in the Name field.
Configure the judging rubric

Every judge needs context plus a clear scoring question. State holds the material Jev evaluates. Questions define the criterion it applies.

  • Open Prompt & Model.
  • Open Model Configuration.
  • Select TypeSafe in Provider.
  • Select jev-latest in Model.
  • Click Apply.

What does Jev receive?

The state contains the dataset question plus the current agent response. Jev reads this context before assigning a score.

Each question defines one evaluation criterion. Its name becomes a feedback key in the experiment table.

  • Map the dataset's question input into State.
  • Map the experiment's current agent output into State.
  • Add one question under Feedback Configuration.
  • Enter answer_quality in Name.
  • Select Score in Type.
  • Enter How fully does the response answer the user's question correctly? in Instructions.

Why use concrete levels?

A vague scale can make different responses look identical. Concrete levels give Jev a visible boundary between partial answers and complete answers.

  • Define the first level as Does not answer the question correctly.
  • Define the second level as Answers part of the question correctly.
  • Define the third level as Answers most of the question correctly.
  • Define the fourth level as Answers the full question correctly.
  • Click Create to save the evaluator.

You have built the judge that every prompt version will face. Its shared rubric makes future comparisons meaningful.

Jev missing from the model list?

Confirm that the TypeSafe secret is stored in the same LangSmith workspace as the dataset. Check that TypeSafe remains selected in Model Configuration.

Help me diagnose why Jev is unavailable in my LangSmith evaluator.

Run and review the first evaluation

An evaluator attached to a dataset runs on new experiments. Your first baseline therefore needs a fresh experiment with the initial prompt.

Keep the baseline small

This batch uses only the ten examples you already prepared. Check your current provider pricing before expanding the dataset.

Before you launch the batch, do you expect the ten responses to earn similar scores?

  • In your local Python environment from earlier, start a LangSmith experiment that runs the existing initial-prompt agent over the ten-example dataset.
  • Return to the dataset in LangSmith.
  • Select the Experiments tab.
  • Watch Progress until all ten runs have completed evaluation.

You will see a completed experiment with ten evaluated runs. The answer_quality feedback appears with the results.

No completed experiment?

Confirm that the experiment uses the ten-example dataset. Make sure the Jev evaluator was created before you started the experiment.

Help me find why my LangSmith experiment is missing runs or Jev scores.

  • Click the completed experiment row to open its results.
  • Select the Compact table view.
  • Click the Columns icon.
  • Enable the answer_quality feedback column if it is hidden.
  • Review the score recorded for each of the ten responses.
  • Open two experiment rows to compare each response with its score.

Your initial prompt now has a measurable baseline. You can point to ten scored runs instead of relying on a few memorable answers.

Spot-check the judge

A judge score is evidence that benefits from human spot-checking. A few row-level checks reveal whether the rubric matches your own reading of the responses.

Your baseline is locked in. Next, you will change one prompt variable and run the same test again.

Compare the Prompt Versions

Your first evaluation in LangSmith gave you a baseline for the initial prompt. Now you will test a revised prompt against that baseline.

The comparison is trustworthy only when the dataset stays fixed. Jev must stay fixed too.

In this step, get ready to:
  • Revise the agent's system prompt to address one weakness from the first evaluation.
  • Score the revised agent against the same ten examples with Jev.
  • Compare the first score table with the second score table side by side.
Revise the system prompt

The first score table shows where the initial system prompt fell short. Your revision should target one visible weakness so the experiment keeps one clear variable.

  • Switch back to the local editor that contains your agent from earlier.
  • Locate the initial system prompt in the agent code.
  • Replace the initial prompt with revised wording that targets one weakness from the first score table.
  • Save the file that contains the agent.

What stays fixed?

  • Keep the agent logic unchanged.
  • Keep the model settings unchanged.
  • Leave the LangSmith tracing configuration unchanged.
  • Retain the same ten-example dataset.
Run the second evaluation

Scoring the revised agent against the same examples creates a controlled comparison. Each new result reflects the prompt revision instead of a different test set.

  • Return to the LangSmith browser tab with the first score table from earlier.
  • Start a second evaluation through the same workflow you used for the first evaluation.
  • Use the same agent entry point that produced the first evaluation.
  • Select the existing dataset that contains the ten question examples.
  • Keep Jev selected as the judge evaluator.

Before you start the second evaluation, do you expect the revised prompt to improve every example equally?

  • Start the second evaluation.

When the evaluation finishes, you will see a second score table containing results for the same ten examples.

That completes the controlled test. Your revised prompt now has evidence built from the same questions as the initial prompt.

Second evaluation did not complete?

  • Confirm that the existing ten-example dataset is selected.
  • Check that the LangSmith API key remains configured in your local environment.
  • Check that the LLM provider API key remains configured in your local environment.

If the run still stops, help me diagnose the second evaluation.

Compare the score tables

The overall scores show which prompt performed better across the dataset. The row-level results reveal where that difference came from.

How to read the difference

A higher overall score points toward the stronger prompt version. Matching rows show which questions improved or regressed.

A stronger average can still hide a weak example. Spot-checking individual judgments keeps the final decision grounded in the agent's actual answers.

  • Keep the second score table open in the current browser window.
  • Open the first evaluation's score table in a separate browser window.
  • Resize both browser windows so the score tables sit side by side.
  • Align the tables using the matching dataset examples.

Before you compare the scores, which prompt version do you expect to perform better across the ten examples?

  • Compare the overall score from the initial-prompt evaluation with the overall score from the revised-prompt evaluation.
  • Compare every matching example across the two score tables.
  • Open three judged examples to spot-check whether Jev's reasoning matches the agent outputs.

You should now see two score tables covering the same ten-example dataset. The system prompt is the only intentional change between them.

You have turned a prompt change into measurable evidence. Next, you will keep the stronger version and save this evaluation as a regression check.

Save Your Best Evaluation

Your two Jev evaluation runs now show how each system prompt performed on the same ten examples in LangSmith. That evidence gives you a clear winner.

A stronger score matters when it shapes the version you keep. This step turns that result into a reusable regression check for your next prompt change.

In this step, get ready to:
  • Identify the better-performing evaluation.
  • Keep its system prompt in your agent.
  • Set its experiment as the baseline for future regression checks.
Choose the winning evaluation

A fair comparison uses the same ten-example dataset for both prompt versions. Your side-by-side table already isolates the system prompt as the variable that changed.

  • Return to the side-by-side comparison from earlier.
  • Identify the experiment with the stronger Jev result across the ten examples.
  • Record the exact experiment name here: name of your better-performing experiment.

You now have an evidence-backed winner. That result decides which prompt your agent keeps.

Keep the winning prompt

The agent currently contains the revised system prompt. The winning version determines whether that prompt stays or changes.

Revised prompt scored better

Evaluation Run 2 already matches the winning result.

  • Switch back to the Python agent file from earlier.
  • Find the revised system prompt in the agent file.
  • Leave the revised system prompt unchanged.
  • Confirm LangSmith tracing remains enabled for the same tracing project.

Your agent already uses the winning prompt. Its tracing setup remains ready for future runs.

Initial prompt scored better

Evaluation Run 1 contains the prompt your agent needs to restore.

  • Return to the comparison view from earlier.
  • Select a result row from Evaluation Run 1.
  • Switch the details panel to Traces.
  • Select the model call that contains the initial system prompt.
  • Copy the exact initial system prompt from the model input.

You now have the exact prompt that produced the stronger result.

  • Switch back to the Python agent file from earlier.
  • Replace the revised system prompt with the copied initial prompt.
  • Leave the LangSmith tracing configuration unchanged.
  • Save the agent file.

Your saved agent now uses the prompt that achieved the stronger Jev result. Its future runs continue to reach the same tracing project.

Set the winning experiment as your baseline

A baseline experiment anchors future comparisons. LangSmith measures later experiments against this saved reference.

  • Return to LangSmith in your browser.
  • Select Datasets & Experiments in the left menu.
  • Select the same ten-example dataset from earlier.
  • Select the Experiments tab.
  • Locate name of your better-performing experiment.

The baseline control is easy to miss because it appears only when you hover over an experiment row.

Before you set the baseline, which experiment do you expect LangSmith to pin at the top?

  • Hover over the row for name of your better-performing experiment.
  • Click Set baseline at the right end of the row.

Strong finish. Your winning experiment now sits at the top of the table with the Baseline tag.

How does the baseline become a regression check?

LangSmith uses the baseline as the source experiment for future comparisons. New score deltas reveal improvements or regressions against your winner.

Both completed experiments stay in the dataset. Their score tables remain available for review.

Jev remains attached to the same dataset. Each future experiment can use the same judge.

  • Confirm the winning experiment appears first in the Experiments tab.
  • Confirm the Baseline tag appears next to its name.
  • Confirm Evaluation Run 1 remains visible.
  • Confirm Evaluation Run 2 remains visible.
  • Confirm the Jev scores still display for both evaluation runs.

Your agent now uses the better-performing prompt. The next prompt change starts from a benchmark you can beat.

Secret mission

Audit Your Jev Grades

Manually audit three Jev grades from your better-performing evaluation. Check whether your saved regression check rewards the answer quality you actually want to preserve.

Clean Up Your Resources

Clean Up Your Resources

Your completed evaluation evidence now lives in LangSmith without launching another Jev judge run. Check current pricing before running larger batches.

Resources you used:

  • The LangSmith dataset containing the same ten examples.
  • The LangSmith tracing project containing runs from both prompt versions.
  • The first Jev evaluation run with its score table.
  • The second Jev evaluation run with its score table.
  • The saved regression check that uses the better-performing evaluation configuration.
  • The local question-answering agent in your Python development environment.

Keep everything running

No action is needed. Choose this option if you plan to test more prompt changes against the same baseline.

  • Reuse the same ten examples so future score changes remain comparable.
  • Run the saved regression check before keeping another prompt change.
  • Spot-check a few graded runs because a judge model can still make mistakes.
  • Keep both score tables so you can trace why one prompt version won.

Pause - I'll come back to this later

LangSmith retains the evaluation evidence for your next session. No new judge run starts while the project is paused.

  • Leave the ten-example dataset unchanged.
  • Leave both evaluation runs in the evaluation history.
  • Stop the local agent if it is still running by pressing Ctrl+C in its terminal.
  • Return later by running the saved regression check before your next prompt change.

Delete - I don't want to use this again

Deleting local files can feel risky. The LangSmith evidence remains available because these steps leave it untouched.

  • Use Finder on macOS or File Explorer on Windows to return to the location where you saved the question-answering agent.
  • Move the question-answering agent files to Trash on macOS or Recycle Bin on Windows.
  • Confirm that the local agent no longer appears in its saved location.
  • Empty Trash on macOS or Recycle Bin on Windows when you are certain that you no longer need the local agent.
  • Keep the LangSmith dataset because it provides the fixed baseline for future comparisons.
  • Keep both evaluation runs because their score tables show whether the prompt change helped.
  • Keep the saved regression check because it makes the next prompt change measurable.

Preserve Your Regression Baseline

Your dataset is the durable baseline. The two evaluation runs preserve the prompt comparison.

Your tracing project preserves the underlying runs. The saved regression check turns that evidence into a repeatable quality gate.

That closes the disposable part of the project. Your reusable evaluation evidence remains ready in LangSmith.

Nice Work!

Nice Work!

You made prompt improvement measurable. Your question-answering agent now uses the prompt version that earned the stronger Jev evaluation score.

Your ten-example LangSmith dataset supports fair comparisons across prompt versions. The saved regression check gives your next prompt edit a repeatable test.

You've learned how to:

  • Capture agent runs with LangSmith tracing. Inspect one trace to see the steps behind an answer.
  • Build a ten-example dataset from traced runs. Use Jev to grade both prompt versions against identical inputs.
  • Compare two evaluation score tables with only the system prompt changed. Keep the better-performing prompt. Save its evaluation as a regression check.
  • Secret Mission: Take on an optional challenge to push your automated evaluation workflow further.

Ready to quiz yourself?