Benchmarking Local LLMs with Ollama
Run and benchmark open-source LLMs locally with GPU acceleration.
Introduction
30 Second Summary
Every time you ask ChatGPT or Claude a question, your words travel to a data center, get processed by a massive model, and the answer streams back over the internet. But what if you could run those same types of models right on your own machine, with no cloud dependency, no API costs, and no data leaving your hardware?
In this project, you will install Ollama on Linux, pull multiple open-source LLMs of varying sizes and architectures, explore how generation parameters control the sampling process, and build a Python benchmarking suite that systematically compares model performance.
What You'll Build
A local LLM benchmarking framework that measures inference speed, latency, and structured output compliance across 4 open-source models running on your NVIDIA GPU.
By the end of this project, you'll have:
- A running Ollama instance with GPU acceleration serving 4 open-source models of different architectures and sizes.
- A parameter exploration script using the OpenAI-compatible API pattern that demonstrates how temperature, top_p, and seed control text generation.
- A benchmarking suite that produces comparative performance data (tokens/second, time-to-first-token) and JSON format compliance scores across all models.
- Secret Mission: Connect Hermes Agent to your local Ollama instance, turning your benchmarked inference server into the brain of a self-improving AI agent.
Are there any prerequisites?
This project assumes you have a Linux machine with an NVIDIA GPU (RTX 5070 Ti or similar with 16 GB VRAM), NVIDIA drivers already installed, and Python 3.9+ with pip available. You should be comfortable working in the terminal and writing Python scripts.
Install Ollama and Set Up Your Environment
To benchmark local language models, you need an inference server that can load models into GPU memory and serve them through an API. Ollama handles this by downloading models, managing GPU resources, and exposing an API endpoint for programmatic access.
In this step, you will install Ollama, confirm your NVIDIA GPU is detected, create a Python project environment, and pull your first model to verify everything works end-to-end.
In this step, get ready to:
- Install Ollama and confirm GPU acceleration is working.
- Create a Python virtual environment with the benchmarking libraries.
- Pull and test your first local model.
Install Ollama and verify GPU detection
Ollama installs as a systemd service on Linux. The install script automatically detects your NVIDIA GPU and configures GPU-accelerated inference.
- Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | shWhat does this command do?
This downloads and runs Ollama's official install script. The script installs the ollama binary to /usr/local/bin, creates a systemd service so Ollama runs in the background, and detects your GPU hardware.
Near the end of the output, look for the line that says NVIDIA GPU installed. which confirms your RTX 5070 Ti was detected.
- Verify Ollama installed correctly by running:
ollama -vYou should see a version number printed, confirming the binary is on your PATH.
- Confirm your GPU is visible to the system by running:
nvidia-smiYou should see your RTX 5070 Ti listed with its 16 GB of VRAM and the CUDA version.
Don't see NVIDIA GPU installed in the output?
Make sure your NVIDIA drivers are installed and working. Run nvidia-smi on its own. If it reports a driver error, reinstall your NVIDIA drivers before re-running the Ollama install script.
If the install script hangs during GPU detection, try running it with debug output: curl -fsSL https://ollama.com/install.sh | sh -x to see where it stalls.
Help me troubleshoot why Ollama cannot detect my NVIDIA GPU on Linux.
Set up the Python environment
You will use two Python libraries in this project. The native ollama library gives access to timing metadata needed for benchmarking. The openai library connects via the OpenAI-compatible API pattern, which is transferable to any cloud LLM service.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir llm-benchmark
cd llm-benchmark- Confirm the folder exists by running:
lsYou should see the llm-benchmark folder listed (you are now inside it, so the output will be empty since the folder is fresh).
- Create a Python virtual environment and activate it by running:
python3 -m venv venv
source venv/bin/activateWhat is a virtual environment?
A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, preventing version conflicts between projects.
You will know it is active when your terminal prompt starts with (venv).
- Install the two libraries by running:
pip install ollama==0.6.2 openai==2.45.0- Verify both libraries installed correctly by running:
python -c "import ollama; print('ollama OK')"python -c "import openai; print('openai OK')"You should see ollama OK and openai OK printed with no errors.
Seeing a ModuleNotFoundError?
Make sure your virtual environment is activated. Your terminal prompt should start with (venv). If not, run source venv/bin/activate again from inside the llm-benchmark folder.
Help me fix a ModuleNotFoundError for ollama or openai in my Python virtual environment.
Pull the first model and test
Ollama downloads models on demand. You will start with a small 3B parameter model that downloads quickly and runs fast, giving you a baseline for comparison later.
- Pull the Llama 3.2 3B model by running:
ollama pull llama3.2:3bThis downloads approximately 2 GB. You will see a progress bar as the model layers download.
- Test the model by sending it a one-shot prompt:
ollama run llama3.2:3b "What is the GGUF format?"You should see the model generate a response about the GGUF file format. After it finishes, you are returned to your terminal.
What is GGUF?
GGUF is the file format that Ollama uses to store quantized model weights. It packages the model architecture, tokenizer, and compressed weights into a single file optimized for fast inference on consumer hardware.
- While the model is still loaded in GPU memory, check VRAM usage by running:
nvidia-smiYou should see VRAM usage has increased (the 3B model uses roughly 2 GB). Ollama keeps models loaded in memory for a few minutes after a request, so you will see the usage even after the response completes.
Interactive mode
You can also chat with models interactively by running ollama run llama3.2:3b without a prompt. This opens a chat session where you can send multiple messages. Type /bye to exit interactive mode.
Model download stalled or errored?
If the download hangs or fails, check your internet connection and retry with ollama pull llama3.2:3b. Ollama resumes partial downloads automatically.
If nvidia-smi shows 0 MB used by Ollama processes, the model may be running on CPU instead of GPU. Check the Ollama service logs with journalctl -u ollama and look for GPU detection messages.
Help me troubleshoot why my Ollama model is not using GPU memory.
Your local inference server is running and GPU-accelerated. Next up, you will pull more models of different sizes and architectures to give your benchmark something to compare.
Pull and Explore Models
Your local inference server is running and you have confirmed it works end-to-end with a 3B model. Now it is time to build your model library.
A single model tells you nothing about how architecture and size affect performance. To benchmark meaningfully, you need models from different organizations, different parameter counts, and different training approaches. In this step, you will pull three more models and learn how to read their metadata.
In this step, get ready to:
- Pull 3 additional models from different organizations and architectures.
- Inspect model metadata to understand quantization, context length, and structure.
- Learn what GGUF format, quantization levels, and architecture details mean for inference.
Pull models of varying sizes and architectures
Each model represents a different approach to language modeling. Having models from Meta, Alibaba, and Google gives you a cross-section of the open-source LLM landscape to compare.
- Pull the three remaining benchmark models by running these commands one at a time:
ollama pull llama3.1:8bollama pull qwen3:8bollama pull gemma2:9bWhat are these models?
- llama3.1:8b is Meta's dense transformer with 8 billion parameters and 128K context length.
- qwen3:8b is Alibaba's 8B model with thinking capabilities (it can reason step-by-step when asked).
- gemma2:9b is Google's 9B model trained with knowledge distillation, meaning a larger teacher model guided its training.
Each pull command shows a progress bar as it downloads. The 8B/9B models are roughly 4.5 GB each, so downloads take a few minutes depending on your internet speed.
- After all three downloads complete, confirm all four models are available by running:
ollama listYou should see four entries: llama3.2:3b, llama3.1:8b, qwen3:8b, and gemma2:9b.
Download stuck or failing?
- If a download stalls, press Ctrl+C and rerun the same ollama pull command. It resumes from where it left off.
- If you see a disk space error, each 8B model needs approximately 4.5 GB. Run df -h to check available space.
- If you see "model not found", double-check the model name matches exactly (including the colon and size tag).
Ask for help: my ollama pull command is failing or stuck, how do I fix it?
Inspect model metadata
Every model you pulled is stored on disk in GGUF format with compressed weights. The ollama show command lets you peek inside to see how a model is structured without loading it into memory.
- Inspect the 3B model's metadata by running:
ollama show llama3.2:3b- Now inspect the 8B model for comparison:
ollama show llama3.1:8bWhat should I look for in the output?
Each ollama show output contains several key pieces of information:
- Model family and parameter count (e.g., llama, 8B).
- Quantization level (e.g., Q4_K_M) which tells you how aggressively the weights are compressed.
- Context length (the maximum tokens the model can process in one conversation).
- Architecture details like the number of layers, attention heads, and embedding dimensions.
Compare the two outputs. Notice how the 8B model has more layers and wider dimensions than the 3B model, which directly affects both quality and inference speed.
- Repeat for the other two models to see their architectures:
ollama show qwen3:8bollama show gemma2:9bNow let's see how these models use your GPU when they are actually loaded into memory.
- Load a model into memory by running a quick one-shot prompt:
ollama run llama3.1:8b "Hello"- Check which models are currently loaded and their VRAM consumption:
ollama psYou should see llama3.1:8b listed with its size and the percentage of layers running on your GPU. With 16 GB VRAM, the entire model fits on your GPU with room to spare.
ollama ps shows nothing?
Ollama unloads models from memory after 5 minutes of inactivity. If you waited too long between the ollama run command and ollama ps, run the prompt again and immediately check ollama ps afterwards.
Ask for help: ollama ps is showing no results even though I just ran a prompt
Understand what you are looking at
The metadata you just inspected reveals three core concepts that determine how a model performs on your hardware.
What is quantization?
Neural network weights are originally stored in 16-bit floating point (FP16), which means an 8B parameter model would need about 16 GB just for its weights. Quantization compresses these weights to fewer bits.
The Q4_K_M label you saw means 4-bit quantization with the K-quant method (a smart compression that preserves important weights at higher precision). This shrinks that 16 GB model down to roughly 4.5 GB while preserving most of the output quality.
What is context length?
Context length is the maximum number of tokens a model can process in a single conversation. Llama 3.1 supports 128K tokens, meaning it can read and respond to very long documents in a single pass.
Longer context requires more memory during inference. A model processing a 128K context window uses significantly more VRAM than the same model processing a 2K prompt.
What do the architecture numbers mean?
The layer count, attention head count, and embedding dimensions define the model's capacity to learn patterns. More layers means the model can build more abstract representations. Wider embedding dimensions mean each token carries more information.
This is why an 8B model from one architecture can outperform a 9B model from another. The arrangement of layers and heads matters as much as raw parameter count.
You now have four models ready to go and you understand what makes each one different under the hood. Next up, you will write Python code that talks to these models programmatically and explore how sampling parameters control the text they generate.
Explore Sampling Parameters
You now have four models pulled and inspected their architecture metadata. The next question is: how does a model actually decide what text to produce?
In this step, you will write a Python script that calls your local Ollama instance through the OpenAI-compatible API pattern. By changing a few numbers, you will see how generation parameters control whether output is deterministic, creative, or reproducible.
In this step, get ready to:
- Create a parameter exploration script using the OpenAI-compatible API.
- Understand the sampling pipeline that turns model logits into text.
- Run the script and observe how temperature, top_p, and seed affect output.
Create the parameter exploration script
This script uses the same API pattern as OpenAI and Anthropic cloud services. The only difference is the base URL, which points at your local Ollama server instead of a cloud endpoint.
- Create a file called explore_params.py inside your llm-benchmark folder.
- Add the client setup and the first exploration function by pasting this code:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1/",
api_key="ollama",
)
PROMPT = "Explain what a neural network is in one sentence."
MODEL = "llama3.1:8b"
def explore_temperature():
print("=" * 60)
print("TEMPERATURE: Controls randomness of sampling")
print("=" * 60)
for temp in [0.0, 0.7, 1.5]:
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=temp,
max_tokens=100,
)
print(f"\nTemperature {temp}:")
print(f" {response.choices[0].message.content}")What does this code do?
The OpenAI client connects to your local Ollama at http://localhost:11434/v1/. The api_key is required by the client but Ollama ignores it, so any string works.
The explore_temperature() function sends the same prompt three times with different temperature values (0.0, 0.7, 1.5). Temperature controls how random the model's token selection is. Low temperature produces focused, deterministic text. High temperature produces varied, sometimes surprising text.
Why the OpenAI library for a local model?
Ollama exposes an OpenAI-compatible endpoint. This means any code you write here transfers directly to cloud services like OpenAI or Anthropic. You just swap the base URL and API key.
You won't see any output yet. The next chunk adds the remaining functions.
- Add the remaining two exploration functions and the main block below your existing code:
def explore_top_p():
print("\n" + "=" * 60)
print("TOP-P: Nucleus sampling - cumulative probability threshold")
print("=" * 60)
for top_p in [0.1, 0.5, 0.95]:
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=0.8,
top_p=top_p,
max_tokens=100,
)
print(f"\nTop-P {top_p}:")
print(f" {response.choices[0].message.content}")
def explore_seed():
print("\n" + "=" * 60)
print("SEED: Reproducible generation")
print("=" * 60)
for run in range(3):
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=0.8,
seed=42,
max_tokens=100,
)
print(f"\nRun {run + 1} (seed=42):")
print(f" {response.choices[0].message.content}")
if __name__ == "__main__":
print(f"Model: {MODEL}")
print(f"Prompt: {PROMPT}\n")
explore_temperature()
explore_top_p()
explore_seed()What do these functions do?
- explore_top_p() tests nucleus sampling. Top-p filters the model's token choices to only the most probable tokens whose cumulative probability reaches your threshold. A low value (0.1) keeps only the safest picks. A high value (0.95) allows more diversity.
- explore_seed() runs the same prompt 3 times with an identical seed value. When you fix the seed and temperature, the model produces identical output every time. This makes experiments reproducible.
- The if __name__ block prints the model and prompt, then calls all three exploration functions in sequence.
- Save explore_params.py.
- Confirm the file exists by running:
ls explore_params.pyYou should see explore_params.py listed in the output.
File not found?
Make sure you are inside the llm-benchmark directory and that your virtual environment is activated. Run pwd to check your current directory.
help me find my explore_params.py file
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before moving on.
ⓧ I'd like to double check the full code
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1/",
api_key="ollama",
)
PROMPT = "Explain what a neural network is in one sentence."
MODEL = "llama3.1:8b"
def explore_temperature():
print("=" * 60)
print("TEMPERATURE: Controls randomness of sampling")
print("=" * 60)
for temp in [0.0, 0.7, 1.5]:
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=temp,
max_tokens=100,
)
print(f"\nTemperature {temp}:")
print(f" {response.choices[0].message.content}")
def explore_top_p():
print("\n" + "=" * 60)
print("TOP-P: Nucleus sampling - cumulative probability threshold")
print("=" * 60)
for top_p in [0.1, 0.5, 0.95]:
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=0.8,
top_p=top_p,
max_tokens=100,
)
print(f"\nTop-P {top_p}:")
print(f" {response.choices[0].message.content}")
def explore_seed():
print("\n" + "=" * 60)
print("SEED: Reproducible generation")
print("=" * 60)
for run in range(3):
response = client.chat.completions.create(
messages=[{"role": "user", "content": PROMPT}],
model=MODEL,
temperature=0.8,
seed=42,
max_tokens=100,
)
print(f"\nRun {run + 1} (seed=42):")
print(f" {response.choices[0].message.content}")
if __name__ == "__main__":
print(f"Model: {MODEL}")
print(f"Prompt: {PROMPT}\n")
explore_temperature()
explore_top_p()
explore_seed()Understand the sampling pipeline
Before you run the script, it helps to know what is happening inside the model each time it generates a token. The parameters you just set control specific stages of this process.
How does a model generate text?
Every time the model produces the next word (token), it goes through these stages:
- The model produces a probability distribution over all possible next tokens. These raw scores are called logits.
- Temperature scales these probabilities. Low temperature (0.0) makes the distribution peaked so only the single most likely token gets picked. High temperature (1.5) flattens the distribution so unlikely tokens get a chance.
- Top-p (nucleus sampling) filters to only consider tokens whose cumulative probability reaches your threshold. A top_p of 0.1 means only the top 10% of probability mass is considered.
- A token is sampled from the remaining distribution.
- The process repeats until a stop condition is met (max tokens reached, end-of-sequence token, or a stop sequence).
This is the same process whether you use a local model or a cloud API. The generation parameters you set in your script control steps 2 and 3.
Run the script and observe the results
- Run your exploration script by executing this command:
python explore_params.pyThe script takes a minute or two to complete since it calls the model 9 times total. Watch the output appear in sections as each function runs.
What should I see?
- Temperature 0.0 produces a clean, focused answer. If you ran it again, it would give the exact same response (greedy decoding).
- Temperature 1.5 produces varied, sometimes unusual phrasing. Each run would be different.
- Top-P 0.1 constrains the model to only the most likely tokens, producing safe, predictable output.
- Top-P 0.95 allows more diversity since the model can sample from a wider range of probable tokens.
- Seed runs (all with seed=42 at temperature 0.8) produce identical output across all 3 runs. This proves that fixing the seed makes generation reproducible.
Seeing a connection error?
If you see a connection refused error, Ollama may not be running. Check its status with systemctl status ollama and start it with systemctl start ollama if needed.
If you see a model not found error, confirm llama3.1:8b is pulled by running ollama list.
help me debug my explore_params.py script
Extra for Experts: Why do seed runs sometimes differ?
On some GPU configurations, floating-point rounding can introduce tiny differences between runs even with the same seed. If your 3 seed runs are not perfectly identical, this is normal GPU non-determinism. The outputs should still be extremely similar.
You now have a concrete feel for how sampling parameters shape model output. Next up, you will build a full benchmarking suite that measures how fast each of your four models generates text and how well they follow structured output instructions.
Build and Run the Benchmarking Suite
In the previous step, you explored how sampling parameters shape model output. You saw temperature, top-p, and seed control the randomness of generation at the token level.
Now you will put all four models to a systematic test. You will build a benchmarking suite that measures real performance metrics across different prompt categories, giving you concrete data on how model architecture and size affect inference speed and structured output compliance.
In this step, get ready to:
- Create a standardized prompt set spanning reasoning, coding, creative, and structured output categories.
- Build a Python benchmarking script that measures tokens/second, time-to-first-token, and JSON compliance across all four models.
- Run the benchmark and interpret comparative performance data.
Create the test prompt set and benchmark script
A good benchmark needs diverse prompts that test different capabilities. You will create a prompt file with four categories: reasoning (logical thinking), coding (code generation), creative (open-ended writing), and structured (JSON format compliance).
- Create a new file called prompts.json in your llm-benchmark folder, then paste the following content:
{
"reasoning": [
"Explain step by step why the sky appears blue during the day but red at sunset.",
"A farmer has 17 sheep. All but 9 run away. How many sheep does the farmer have left? Explain your reasoning."
],
"coding": [
"Write a Python function that finds the longest palindrome substring in a given string.",
"Write a bash one-liner that finds the 10 largest files in the current directory tree."
],
"creative": [
"Write a haiku about a programmer debugging code at 3am.",
"Describe a world where gravity works in reverse, in exactly three sentences."
],
"structured": [
"List exactly 3 programming languages with their year of creation and primary use case. Respond in JSON format with a 'languages' array containing objects with 'name', 'year', and 'use_case' fields.",
"Describe the solar system's inner planets. Respond in JSON format with a 'planets' array containing objects with 'name', 'distance_from_sun_au' (number), and 'has_atmosphere' (boolean) fields."
]
}Why these four categories?
Each category tests a different model capability. Reasoning tests logical chain-of-thought. Coding tests structured problem-solving. Creative tests open-ended generation where there is no single right answer.
The structured category is special. Those prompts ask the model to produce JSON matching a specific schema. This lets you measure format compliance alongside speed.
- Save prompts.json.
- Confirm the file exists by running:
ls prompts.json- You should see prompts.json listed in the output.
File not found?
Make sure you are in the llm-benchmark directory. Run pwd to check your current location. If the file is elsewhere, move it into this directory.
help me find where I saved prompts.json
Now for the benchmark script itself. The script uses the native ollama Python library (not the OpenAI-compatible endpoint) because it exposes timing metadata that the OpenAI API does not return.
- Create a new file called benchmark.py in your llm-benchmark folder, then paste the following setup and measurement functions:
import json
import time
from datetime import datetime
import ollama
MODELS = ["llama3.2:3b", "llama3.1:8b", "qwen3:8b", "gemma2:9b"]
NUM_RUNS = 3
MAX_TOKENS = 256
STRUCTURED_SCHEMAS = {
"languages": {
"type": "object",
"properties": {
"languages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"year": {"type": "integer"},
"use_case": {"type": "string"},
},
"required": ["name", "year", "use_case"],
},
}
},
"required": ["languages"],
},
"planets": {
"type": "object",
"properties": {
"planets": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"distance_from_sun_au": {"type": "number"},
"has_atmosphere": {"type": "boolean"},
},
"required": ["name", "distance_from_sun_au", "has_atmosphere"],
},
}
},
"required": ["planets"],
},
}
def load_prompts(filepath="prompts.json"):
with open(filepath) as f:
return json.load(f)
def benchmark_single(model_name, prompt, format_schema=None):
"""Run a single prompt against a model and return timing metrics."""
kwargs = {
"model": model_name,
"messages": [{"role": "user", "content": prompt}],
"options": {"num_predict": MAX_TOKENS},
}
if format_schema:
kwargs["format"] = format_schema
response = ollama.chat(**kwargs)
eval_count = response.get("eval_count", 0)
eval_duration = response.get("eval_duration", 0)
prompt_eval_duration = response.get("prompt_eval_duration", 0)
total_duration = response.get("total_duration", 0)
tokens_per_second = (
eval_count / (eval_duration / 1e9) if eval_duration > 0 else 0
)
time_to_first_token = prompt_eval_duration / 1e9
total_time = total_duration / 1e9
return {
"tokens_generated": eval_count,
"tokens_per_second": tokens_per_second,
"time_to_first_token_s": time_to_first_token,
"total_time_s": total_time,
"output": response["message"]["content"],
}
def check_json_compliance(output):
"""Check if the output is valid JSON."""
try:
json.loads(output)
return True
except (json.JSONDecodeError, TypeError):
return False
def benchmark_model(model_name, prompt, num_runs=NUM_RUNS, format_schema=None):
"""Run multiple iterations and average the results."""
results = []
for _ in range(num_runs):
result = benchmark_single(model_name, prompt, format_schema)
results.append(result)
return resultsWhat does this code do?
- MODELS lists the four models you pulled earlier. NUM_RUNS = 3 means each prompt runs three times per model for stable averages.
- STRUCTURED_SCHEMAS defines JSON schemas that Ollama's format parameter uses to constrain model output to valid JSON matching the schema.
- benchmark_single() calls ollama.chat() and extracts timing fields from the response: eval_count (tokens generated), eval_duration (generation time in nanoseconds), and prompt_eval_duration (time to process the input prompt).
- benchmark_model() repeats benchmark_single() multiple times so you can average out variance between runs.
You won't see a change yet. The next code block adds the orchestration that ties everything together.
- Add the following code below the benchmark_model() function in benchmark.py:
def run_benchmark(prompts):
"""Run full benchmark suite across all models and prompts."""
all_results = {}
compliance_scores = {}
schema_keys = list(STRUCTURED_SCHEMAS.keys())
for model in MODELS:
print(f"\n{'=' * 60}")
print(f"Benchmarking: {model}")
print(f"{'=' * 60}")
print(f" Ensuring model is available...")
ollama.pull(model)
# Warm up
print(f" Warming up...")
ollama.chat(
model=model,
messages=[{"role": "user", "content": "Hello"}],
options={"num_predict": 1},
)
model_results = {}
model_compliance = {"total": 0, "valid": 0}
for category, prompt_list in prompts.items():
print(f" Category: {category}")
category_results = []
is_structured = category == "structured"
for i, prompt in enumerate(prompt_list):
format_schema = None
if is_structured and i < len(schema_keys):
format_schema = STRUCTURED_SCHEMAS[schema_keys[i]]
results = benchmark_model(
model, prompt, format_schema=format_schema
)
avg_tps = sum(r["tokens_per_second"] for r in results) / len(
results
)
avg_ttft = sum(r["time_to_first_token_s"] for r in results) / len(
results
)
avg_total = sum(r["total_time_s"] for r in results) / len(results)
entry = {
"prompt": (
prompt[:50] + "..." if len(prompt) > 50 else prompt
),
"avg_tokens_per_second": round(avg_tps, 2),
"avg_time_to_first_token_s": round(avg_ttft, 4),
"avg_total_time_s": round(avg_total, 4),
"sample_output": results[0]["output"][:200],
}
if is_structured:
for r in results:
model_compliance["total"] += 1
if check_json_compliance(r["output"]):
model_compliance["valid"] += 1
entry["json_valid"] = check_json_compliance(
results[0]["output"]
)
category_results.append(entry)
model_results[category] = category_results
all_results[model] = model_results
compliance_scores[model] = model_compliance
return all_results, compliance_scores
def print_summary(results, compliance_scores):
"""Print a formatted comparison table."""
print(f"\n{'=' * 90}")
print("BENCHMARK SUMMARY")
print(f"{'=' * 90}")
print(
f"{'Model':<20} {'Avg tok/s':<12} {'Avg TTFT (s)':<15} "
f"{'JSON Compliance':<18} {'Prompts'}"
)
print(f"{'-' * 90}")
for model, categories in results.items():
all_tps = []
all_ttft = []
for category, prompt_results in categories.items():
for result in prompt_results:
all_tps.append(result["avg_tokens_per_second"])
all_ttft.append(result["avg_time_to_first_token_s"])
avg_tps = sum(all_tps) / len(all_tps) if all_tps else 0
avg_ttft = sum(all_ttft) / len(all_ttft) if all_ttft else 0
total_prompts = sum(len(v) for v in categories.values())
comp = compliance_scores.get(model, {"valid": 0, "total": 0})
if comp["total"] > 0:
comp_pct = f"{comp['valid']}/{comp['total']} ({100 * comp['valid'] // comp['total']}%)"
else:
comp_pct = "N/A"
print(
f"{model:<20} {avg_tps:<12.2f} {avg_ttft:<15.4f} "
f"{comp_pct:<18} {total_prompts}"
)
print(f"{'=' * 90}")
if __name__ == "__main__":
print(f"Local LLM Benchmark - {datetime.now().strftime('%Y-%m-%d %H:%M')}")
print(f"Models: {', '.join(MODELS)}")
print(f"Runs per prompt: {NUM_RUNS}")
prompts = load_prompts()
results, compliance_scores = run_benchmark(prompts)
print_summary(results, compliance_scores)
output_file = "benchmark_results.json"
with open(output_file, "w") as f:
json.dump(
{"results": results, "compliance": compliance_scores}, f, indent=2
)
print(f"\nFull results saved to {output_file}")What does this code do?
- run_benchmark() orchestrates the full suite. For each model, it first calls ollama.pull() to confirm availability, then warms up the model with a single-token generation so the GPU memory is primed.
- For structured prompts, it passes the matching JSON schema via the format parameter. This constrains the model to output valid JSON, and the script scores compliance by attempting to parse each output.
- print_summary() aggregates all results into a comparison table showing average tokens/second, time-to-first-token, and JSON compliance per model.
- The __main__ block ties it all together and saves full results to benchmark_results.json for later analysis.
- Save benchmark.py.
✔️ Awesome, I've got everything!
Great. Double check you have saved both prompts.json and benchmark.py in your llm-benchmark directory.
ⓧ I'd like to double check the full code
Here is the full benchmark.py for comparison:
import json
import time
from datetime import datetime
import ollama
MODELS = ["llama3.2:3b", "llama3.1:8b", "qwen3:8b", "gemma2:9b"]
NUM_RUNS = 3
MAX_TOKENS = 256
STRUCTURED_SCHEMAS = {
"languages": {
"type": "object",
"properties": {
"languages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"year": {"type": "integer"},
"use_case": {"type": "string"},
},
"required": ["name", "year", "use_case"],
},
}
},
"required": ["languages"],
},
"planets": {
"type": "object",
"properties": {
"planets": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"distance_from_sun_au": {"type": "number"},
"has_atmosphere": {"type": "boolean"},
},
"required": ["name", "distance_from_sun_au", "has_atmosphere"],
},
}
},
"required": ["planets"],
},
}
def load_prompts(filepath="prompts.json"):
with open(filepath) as f:
return json.load(f)
def benchmark_single(model_name, prompt, format_schema=None):
"""Run a single prompt against a model and return timing metrics."""
kwargs = {
"model": model_name,
"messages": [{"role": "user", "content": prompt}],
"options": {"num_predict": MAX_TOKENS},
}
if format_schema:
kwargs["format"] = format_schema
response = ollama.chat(**kwargs)
eval_count = response.get("eval_count", 0)
eval_duration = response.get("eval_duration", 0)
prompt_eval_duration = response.get("prompt_eval_duration", 0)
total_duration = response.get("total_duration", 0)
tokens_per_second = (
eval_count / (eval_duration / 1e9) if eval_duration > 0 else 0
)
time_to_first_token = prompt_eval_duration / 1e9
total_time = total_duration / 1e9
return {
"tokens_generated": eval_count,
"tokens_per_second": tokens_per_second,
"time_to_first_token_s": time_to_first_token,
"total_time_s": total_time,
"output": response["message"]["content"],
}
def check_json_compliance(output):
"""Check if the output is valid JSON."""
try:
json.loads(output)
return True
except (json.JSONDecodeError, TypeError):
return False
def benchmark_model(model_name, prompt, num_runs=NUM_RUNS, format_schema=None):
"""Run multiple iterations and average the results."""
results = []
for _ in range(num_runs):
result = benchmark_single(model_name, prompt, format_schema)
results.append(result)
return results
def run_benchmark(prompts):
"""Run full benchmark suite across all models and prompts."""
all_results = {}
compliance_scores = {}
schema_keys = list(STRUCTURED_SCHEMAS.keys())
for model in MODELS:
print(f"\n{'=' * 60}")
print(f"Benchmarking: {model}")
print(f"{'=' * 60}")
print(f" Ensuring model is available...")
ollama.pull(model)
# Warm up
print(f" Warming up...")
ollama.chat(
model=model,
messages=[{"role": "user", "content": "Hello"}],
options={"num_predict": 1},
)
model_results = {}
model_compliance = {"total": 0, "valid": 0}
for category, prompt_list in prompts.items():
print(f" Category: {category}")
category_results = []
is_structured = category == "structured"
for i, prompt in enumerate(prompt_list):
format_schema = None
if is_structured and i < len(schema_keys):
format_schema = STRUCTURED_SCHEMAS[schema_keys[i]]
results = benchmark_model(
model, prompt, format_schema=format_schema
)
avg_tps = sum(r["tokens_per_second"] for r in results) / len(
results
)
avg_ttft = sum(r["time_to_first_token_s"] for r in results) / len(
results
)
avg_total = sum(r["total_time_s"] for r in results) / len(results)
entry = {
"prompt": (
prompt[:50] + "..." if len(prompt) > 50 else prompt
),
"avg_tokens_per_second": round(avg_tps, 2),
"avg_time_to_first_token_s": round(avg_ttft, 4),
"avg_total_time_s": round(avg_total, 4),
"sample_output": results[0]["output"][:200],
}
if is_structured:
for r in results:
model_compliance["total"] += 1
if check_json_compliance(r["output"]):
model_compliance["valid"] += 1
entry["json_valid"] = check_json_compliance(
results[0]["output"]
)
category_results.append(entry)
model_results[category] = category_results
all_results[model] = model_results
compliance_scores[model] = model_compliance
return all_results, compliance_scores
def print_summary(results, compliance_scores):
"""Print a formatted comparison table."""
print(f"\n{'=' * 90}")
print("BENCHMARK SUMMARY")
print(f"{'=' * 90}")
print(
f"{'Model':<20} {'Avg tok/s':<12} {'Avg TTFT (s)':<15} "
f"{'JSON Compliance':<18} {'Prompts'}"
)
print(f"{'-' * 90}")
for model, categories in results.items():
all_tps = []
all_ttft = []
for category, prompt_results in categories.items():
for result in prompt_results:
all_tps.append(result["avg_tokens_per_second"])
all_ttft.append(result["avg_time_to_first_token_s"])
avg_tps = sum(all_tps) / len(all_tps) if all_tps else 0
avg_ttft = sum(all_ttft) / len(all_ttft) if all_ttft else 0
total_prompts = sum(len(v) for v in categories.values())
comp = compliance_scores.get(model, {"valid": 0, "total": 0})
if comp["total"] > 0:
comp_pct = f"{comp['valid']}/{comp['total']} ({100 * comp['valid'] // comp['total']}%)"
else:
comp_pct = "N/A"
print(
f"{model:<20} {avg_tps:<12.2f} {avg_ttft:<15.4f} "
f"{comp_pct:<18} {total_prompts}"
)
print(f"{'=' * 90}")
if __name__ == "__main__":
print(f"Local LLM Benchmark - {datetime.now().strftime('%Y-%m-%d %H:%M')}")
print(f"Models: {', '.join(MODELS)}")
print(f"Runs per prompt: {NUM_RUNS}")
prompts = load_prompts()
results, compliance_scores = run_benchmark(prompts)
print_summary(results, compliance_scores)
output_file = "benchmark_results.json"
with open(output_file, "w") as f:
json.dump(
{"results": results, "compliance": compliance_scores}, f, indent=2
)
print(f"\nFull results saved to {output_file}")Run the benchmark and analyze
The benchmark runs each of the 8 prompts against all 4 models, 3 times each. That is 96 total inference calls. Expect this to take several minutes depending on your GPU speed.
- Run the benchmark by executing:
python benchmark.pyWhat should I see?
The script prints progress as it goes: which model is being benchmarked, the warm-up step, and each prompt category. At the end, you will see a summary table with columns for Model, Avg tok/s, Avg TTFT (s), JSON Compliance, and Prompts.
The full detailed results are saved to benchmark_results.json in your project folder.
- Wait for the benchmark to complete. You should see a BENCHMARK SUMMARY table printed in your terminal with a row for each model.
- Confirm the results file was created by running:
ls benchmark_results.json- You should see benchmark_results.json listed.
Benchmark failing or hanging?
- If you see a connection error, make sure Ollama is still running. Check with ollama -v and restart if needed.
- If it hangs on one model, that model may not be fully downloaded. Let the ollama.pull() step complete before the benchmark continues.
- If you see a FileNotFoundError for prompts.json, make sure both files are in the same directory and you are running the command from inside llm-benchmark.
help me debug my benchmark script
With the summary table in front of you, look for these patterns in the data:
- The 3B model (llama3.2:3b) should show significantly higher tokens/second than the 8B and 9B models. Fewer parameters means less computation per token.
- Compare same-size models: llama3.1:8b vs qwen3:8b vs gemma2:9b. Different architectures produce different speed profiles even at similar parameter counts.
- Time-to-first-token (TTFT) measures how long prompt processing takes. Longer prompts take longer to process before the first output token appears.
- JSON compliance should be high across all models when using Ollama's format parameter for constrained generation. The interesting comparison is whether structured output noticeably slows down generation.
Interpret the results
Raw numbers are useful, but the real value comes from what they tell you about model selection. Use your benchmark data to answer these questions.
- Which model gives the best throughput for its parameter class? A smaller model that is 3x faster might be the right choice for latency-sensitive applications.
- Does structured output (JSON mode) measurably slow down inference compared to unconstrained generation? Compare the average total time for structured prompts vs creative or reasoning prompts on the same model.
- Which architecture handles format constraints most naturally? Some models may produce valid JSON faster than others because their training data included more structured content.
- At what point does model size stop providing meaningful quality gains for your prompts? Read the sample outputs in benchmark_results.json to compare quality alongside the speed metrics.
Why warm up matters
The script sends a single-token warm-up call before each model's real benchmark runs. This ensures the model is loaded into GPU memory before timing starts. Without this, the first run would include model load time and skew your results.
Extra for Experts: Why 3 runs per prompt?
GPU inference has minor variance between runs due to scheduling, thermal throttling, and memory allocation. Running each prompt 3 times and averaging smooths out these fluctuations. For more rigorous benchmarking, you could increase NUM_RUNS to 5 or 10, though this proportionally increases total benchmark time.
Secret mission
Connect Hermes Agent to Local Ollama
You have a benchmarked, GPU-accelerated inference server sitting idle between benchmarks. What if you connected it to an autonomous AI agent that can use tools, remember across sessions, and improve itself over time?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing cloud costs. Ollama does run as a background service and the downloaded models consume disk space (roughly 15 GB total across all four models).
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Ollama systemd service (runs automatically on boot).
- Downloaded models: llama3.2:3b, llama3.1:8b, qwen3:8b, gemma2:9b (roughly 15 GB total disk space).
- Project directory llm-benchmark/ with Python scripts, virtual environment, and benchmark results.
Keep everything running
No action needed. Choose this if you plan to keep experimenting with local models or want Ollama available for other projects.
- Ollama continues running as a background service. It uses minimal resources when no model is actively loaded.
- Models stay on disk in /usr/share/ollama and are available immediately when you need them.
- Your benchmarking scripts in llm-benchmark/ remain ready to rerun or extend.
- Hermes Agent configuration and session data remain stored locally on your machine.
Pause - I'll come back to this later
Free up memory and GPU resources by stopping the Ollama service, while keeping all your models and project files on disk.
- Stop the Ollama service so it no longer runs in the background:
sudo systemctl stop ollama
sudo systemctl disable ollama- Your models remain on disk. When you want to resume, re-enable and start the service:
sudo systemctl enable ollama
sudo systemctl start ollama- Your llm-benchmark/ project directory stays untouched.
Delete - I don't want to use this again
Remove all project resources completely. This frees up disk space and removes the Ollama service from your system.
- Remove individual models first (while Ollama is still running) to cleanly free model storage:
ollama rm llama3.2:3b
ollama rm llama3.1:8b
ollama rm qwen3:8b
ollama rm gemma2:9b- Stop and disable the Ollama service, then remove the systemd unit file:
sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo systemctl daemon-reload- Remove the Ollama binary:
sudo rm $(which ollama)- Remove the Ollama libraries:
sudo rm -rf /usr/local/lib/ollama- Remove the Ollama service user, group, and model data directory:
sudo userdel ollama
sudo groupdel ollama
sudo rm -rf /usr/share/ollama- Remove any user-level model cache:
rm -rf ~/.ollama- Delete the project directory and all its contents:
cd ~/Desktop
rm -rf llm-benchmarkThat's a Wrap!
That's a Wrap!
Nice work! You have built a complete local LLM benchmarking framework from scratch, proving that you understand how open-source language models perform under real workloads.
You've learned how to:
- Install and configure Ollama with NVIDIA GPU acceleration, pull models of varying architectures and sizes, and explore their metadata to understand quantization and context length.
- Build a parameter exploration script using the OpenAI-compatible API pattern to demonstrate how temperature, top_p, and seed control the sampling pipeline.
- Build and run a benchmarking suite that systematically compares 4 models on throughput, time-to-first-token, and structured output compliance, producing comparative data in JSON format.
- (Secret Mission) Connected Hermes Agent to your local Ollama instance, turning your benchmarked inference server into the backend for an autonomous AI agent with tool-calling capabilities.
Ready to quiz yourself?