Local AI: Model Selection and Quantization

Build a Python tool that calculates whether any AI model fits your hardware.

Introduction

30 Second Summary

Every local AI guide says "just download the model and run it." But what happens when a 7 billion parameter model needs more memory than your laptop has? You get crashes, frozen terminals, and swap thrashing that makes your system unusable.

In this project, you will build a Python command-line tool that calculates whether any AI model will run on any hardware. Along the way, you will learn to read model architecture details, calculate memory requirements from first principles, and optimize Ollama for CPU-only inference.

What You'll Build

A "Local AI Hardware Advisor" that takes hardware specs and model parameters, then outputs a feasibility assessment with recommendations for running models locally.

By the end of this project, you'll have:

  • A working mental framework for calculating whether any model fits on any hardware, whether GPU or CPU-only.
  • Hands-on experience comparing quantization levels and observing their impact on quality, speed, and memory.
  • A Python script (model_advisor.py) that produces accurate feasibility assessments for any hardware configuration.
  • Secret Mission: Extend your advisor to calculate fine-tuning VRAM requirements and recommend the cheapest cloud GPU for the job.

Are there any prerequisites?

This project assumes you have Ollama installed and working, basic Python knowledge, and terminal familiarity. You will also need a text editor like VS Code.

The project is designed for a Linux system with 16 GB RAM and no GPU, but the concepts apply to any hardware configuration.

Inspect Your System and Understand Model Anatomy

Before you can decide which model to run, you need to know your hardware limits. Your system's available RAM is the hard ceiling that determines which models fit, how much context you can afford, and whether inference will be fast or painfully slow.

In this step, you'll confirm your system resources, pull two quantization variants of the same model, and learn to read the architecture details that feed into memory calculations.

In this step, get ready to:
  • Check your system RAM and confirm Ollama is running.
  • Pull two quantization variants of the same model for size comparison.
  • Inspect model architecture details via CLI and REST API.
Check your system resources and verify Ollama is running

Every model you run competes with the operating system and other processes for RAM. Knowing your total and available memory tells you exactly how much budget you have for model weights and the KV cache.

  • Check your total and available RAM by running this command:
free -h

What does this output tell you?

The total column shows your physical RAM (you should see around 16 GB). The available column shows how much is free for new processes.

For Ollama, roughly 75% of your total RAM is usable for model inference. On a 16 GB system, that gives you about 12 GB of budget for model weights, KV cache, and overhead.

  • Confirm Ollama is operational and see any currently downloaded models by running:
ollama list

You should see a list of models (possibly empty if you haven't pulled any yet) with no error messages.

Seeing "connection refused"?

The Ollama server might not be running. Start it with ollama serve in a separate terminal window, then retry the command.

Help me troubleshoot

  • Pull the default 3B model (Q4_K_M quantization) by running:
ollama pull qwen2.5:3b
  • Pull the same model at Q8_0 quantization by running:
ollama pull qwen2.5:3b-instruct-q8_0

Why two versions of the same model?

Both of these are the same 3-billion-parameter Qwen 2.5 model. The difference is how aggressively the weights are compressed. qwen2.5:3b uses Q4_K_M (roughly 4.9 bits per weight, about 1.9 GB on disk). qwen2.5:3b-instruct-q8_0 uses Q8_0 (8.5 bits per weight, about 3.3 GB on disk).

By comparing them, you'll see exactly how quantization trades file size and RAM for response quality.

  • Verify both models are downloaded by running:
ollama list

You should see both qwen2.5:3b and qwen2.5:3b-instruct-q8_0 in the output with their respective sizes.

Download stuck or failing?

Model pulls require a stable internet connection. If the download stalls, press Ctrl+C and retry the pull command. Ollama resumes from where it left off.

Help me fix my download

Inspect model details with ollama show

Now that you have two models, you can read their specs directly from the CLI. The ollama show command displays the architecture details you'll need for memory calculations.

  • Inspect the default Q4_K_M model by running:
ollama show qwen2.5:3b

What do these fields mean?

  • architecture tells you the model family (e.g. qwen2). This determines the internal structure.
  • parameters is the total number of learnable weights in the model (e.g. 3.09B). More parameters generally means more capable but more RAM.
  • context length is the maximum number of tokens the model can process at once (e.g. 32768). This is the upper bound, not what you'll always use.
  • embedding length is the dimensionality of the model's internal representations. You'll use this when calculating attention head dimensions.
  • quantization shows the compression level applied to the weights (e.g. Q4_K_M). Lower numbers mean more compression, smaller size, and slightly reduced quality.
  • Now inspect the Q8_0 variant for comparison by running:
ollama show qwen2.5:3b-instruct-q8_0

Notice that the parameter count and context length are identical. Only the quantization level and the resulting file size differ.

Extract architecture details via the Ollama API

The CLI output is human-friendly, but your Python tool will need these values programmatically. The Ollama REST API exposes a richer set of architecture details including the layer count, attention heads, and KV heads that feed directly into the memory formulas.

  • Query the API for full model architecture details by running:
curl -s http://localhost:11434/api/show -d '{"model": "qwen2.5:3b"}' | python3 -m json.tool

What does this JSON response contain?

The response includes a model_info object with architecture-specific keys. The ones you'll use for memory math are:

  • qwen2.block_count is the number of transformer layers (36 for Qwen 2.5 3B).
  • qwen2.attention.head_count is the total number of attention heads (16 for this model).
  • qwen2.attention.head_count_kv is the number of key-value heads (2 for this model). This is the number that matters for KV cache calculations.
  • qwen2.embedding_length is the hidden size. Divide this by the total head count to get the head dimension (128 in this case).

These are the exact values your Model Advisor tool will use in Step 5 to calculate whether a model fits on any given hardware.

  • Look specifically at the details section for the quick summary. It includes parameter_size and quantization_level which confirm what ollama show displayed in the CLI.

Pipe to grep for specific fields

If the JSON output is overwhelming, try piping it through grep to find specific keys like block_count or head_count.

Seeing a parse error or empty response?

Make sure the JSON in the -d flag uses single quotes around the entire string and double quotes for the JSON keys. Also confirm Ollama is still running with ollama list.

Help me fix the curl command

You now have a concrete mental model of what lives inside a GGUF model file. Next up, you'll turn these architecture numbers into actual memory formulas.

Learn the Memory Math

In Step 1, you pulled models and read their specs. You now know that a model's architecture has layers, KV heads, and a head dimension. But knowing the specs is only half the picture.

The real question is: does this model fit in my RAM? In this step, you will learn three formulas that answer that question for any model on any hardware. These formulas are pure arithmetic. Once you know them, you will never again be surprised by a crashed session or swap thrashing.

In this step, get ready to:
  • Calculate model weight memory from parameter count and quantization level.
  • Calculate KV cache size from architecture details and context length.
  • Combine both formulas to determine which models fit on your 16 GB system.
Understand the weight memory formula

Model weights are the learned parameters stored on disk and loaded into RAM during inference. The memory they consume depends on two values: how many parameters the model has, and how many bits are used to store each one.

The formula is:

weight_memory_gb = params_billions * (bits_per_weight / 8)

What does this formula mean?

Each parameter is stored using a certain number of bits. Dividing by 8 converts bits to bytes, and multiplying by billions of parameters gives you the total in GB.

The "bits per weight" value depends on the quantization level. FP16 uses 16 bits. Q8_0 uses approximately 8.5 bits. Q4_K_M uses approximately 4.89 bits. These are measured averages, not exact round numbers, because GGUF quantization uses a mixed-precision block structure where some weights get higher precision than others.

Work through these concrete examples for an 8B parameter model:

  • FP16: 8 * (16 / 8) = 16 GB. Full precision. Too large for your 16 GB system (the OS and other processes need RAM too).
  • Q8_0: 8 * (8.5 / 8) = ~8.5 GB. Near-lossless quality. Weights alone use over half your usable RAM.
  • Q4_K_M: 8 * (4.89 / 8) = ~4.9 GB. The standard quantization for most hardware. Leaves plenty of room for the KV cache and system overhead.

Why is Q4_K_M ~4.9 bits and not exactly 4?

The "4" in Q4_K_M describes the target precision, but the K-quant format uses a mixed-precision block structure. Some weights within each block are stored at higher precision to preserve quality on the most important parameters. The measured average across the file lands at approximately 4.89 bits per weight.

Understand the KV cache formula

Weights are only part of the memory story. During inference, the model stores intermediate computations called the KV cache (Key-Value cache). This cache grows with the number of tokens in the conversation. The longer your context window, the more RAM the KV cache consumes.

The formula is:

kv_cache_bytes = 2 * num_layers * num_kv_heads * head_dim * context_length * bytes_per_element

What does each part mean?

  • 2: there are two tensors stored per layer. One for Keys and one for Values.
  • num_layers: how many transformer layers the model has. Each layer maintains its own KV cache.
  • num_kv_heads: the number of attention heads used for Key/Value (often fewer than the total attention heads due to Grouped Query Attention).
  • head_dim: the size of each attention head's internal representation (typically 128).
  • context_length: how many tokens the model is tracking. This is the variable that makes KV cache grow.
  • bytes_per_element: 2 for FP16, 1 for INT8. Most Ollama models use FP16 for KV cache by default.

Now calculate it for a real model. Llama 3.1 8B has 32 layers, 8 KV heads, and a head dimension of 128. Use FP16 (2 bytes per element) for the KV cache.

At 4096 tokens (Ollama's default context length):

kv_cache = 2 * 32 * 8 * 128 * 4096 * 2
         = 536,870,912 bytes
         = ~0.5 GB

That is small. At the default context length, the KV cache barely registers compared to the model weights.

At 32,768 tokens (32K context):

kv_cache = 2 * 32 * 8 * 128 * 32768 * 2
         = 4,294,967,296 bytes
         = ~4 GB

At 32K tokens, the KV cache alone takes 4 GB. For a 16 GB system, this is massive. The KV cache scales linearly with context length, so doubling the context doubles the cache.

Why is the KV cache the hidden cost?

Model weights are fixed. Once loaded, they don't grow. The KV cache is the variable that catches people off guard. A model that fits comfortably at 4K context can crash your system at 32K context because the KV cache grew by 8x.

Put it all together into the total memory formula

The total memory a model needs is the sum of its weights, its KV cache, and a fixed overhead for the inference engine (Ollama's runtime buffers, memory-mapped I/O, and scratch space).

total_memory = weight_memory + kv_cache + overhead (~1.5 GB)

On your 16 GB system, approximately 75% of your RAM is usable for Ollama. The OS and background processes take the rest. That gives you roughly 12 GB to work with.

Apply this to several model configurations and see which ones fit:

8B model at Q4_K_M, 4096 context:

weight:    8 * (4.89 / 8)  = 4.89 GB
kv_cache:  ~0.5 GB (calculated above)
overhead:  1.5 GB
total:     ~6.9 GB

Available: 12 GB
Headroom:  ~5.1 GB  --> FITS comfortably

8B model at Q4_K_M, 32K context:

weight:    4.89 GB
kv_cache:  ~4.0 GB
overhead:  1.5 GB
total:     ~10.4 GB

Available: 12 GB
Headroom:  ~1.6 GB  --> FITS, but tight

8B model at Q8_0, 4096 context:

weight:    8 * (8.5 / 8) = 8.5 GB
kv_cache:  ~0.5 GB
overhead:  1.5 GB
total:     ~10.5 GB

Available: 12 GB
Headroom:  ~1.5 GB  --> FITS, barely

8B model at Q8_0, 32K context:

weight:    8.5 GB
kv_cache:  ~4.0 GB
overhead:  1.5 GB
total:     ~14.0 GB

Available: 12 GB
Headroom:  -2.0 GB  --> DOES NOT FIT

What does "does not fit" actually mean?

When total memory exceeds your available RAM, the system starts swapping to disk. On an SSD this means inference slows to a crawl (10-100x slower). On a hard drive, it becomes completely unusable. The model won't crash immediately. It will just grind to a halt.

Your decision framework

For your 16 GB CPU-only system, the sweet spot is an 8B model at Q4_K_M with 4096 context. This gives you over 5 GB of headroom for the OS and other tasks. If you need higher quality responses, Q8_0 works at short context lengths. If you need longer conversations, Q4_K_M handles up to 32K tokens with tight but viable margins.

You now have the mental math to evaluate any model before you pull it. Next up, you will put these formulas to the test by benchmarking real quantization levels on your hardware and measuring the actual speed, quality, and memory tradeoffs.

Benchmark Quantization Levels

You now have the formulas to predict memory usage on paper. The real test is whether those predictions hold up when you run actual models on your hardware.

In this step, you will run the same prompt through two different quantization levels of the same model and measure the differences in speed, memory, and quality. You will also push the context window to see how the KV cache formula plays out in reality.

In this step, get ready to:
  • Run benchmarks comparing Q4_K_M and Q8_0 quantizations of the same model.
  • Monitor actual memory usage during inference and compare to your calculations.
  • Test context window impact at different num_ctx values.
Run the same prompt through both quantization levels

The --verbose flag tells Ollama to print performance metrics after every response. This gives you tokens per second, prompt evaluation speed, and total duration for each run.

  • Before running the models, record your baseline memory with this command:
free -h

What does this show you?

The free -h command shows total, used, and available system memory in human-readable format. Note the "available" column value. This is your pre-inference baseline.

  • Run the Q4_K_M variant with the verbose flag by running this command:
ollama run qwen2.5:3b --verbose
  • Once the interactive session loads, type this exact prompt and press Enter:
Explain the difference between compiled and interpreted programming languages. Give two examples of each.

Why use the same prompt?

Using an identical prompt across both quantization levels ensures a fair comparison. Any difference in speed or quality comes from the quantization, not from the prompt difficulty.

After the model responds, you should see a performance summary below the response. It looks something like this:

total duration:       3.2s
load duration:        42ms
prompt eval count:    25 token(s)
prompt eval duration: 180ms
prompt eval rate:     138.89 tokens/s
eval count:           150 token(s)
eval duration:        2.9s
eval rate:            51.72 tokens/s

What do these metrics mean?

  • prompt eval rate is how fast the model processes your input (higher is better).
  • eval rate is how fast the model generates output tokens. This is the number you feel as a user.
  • total duration is the full wall-clock time from prompt to final token.
  • Write down the eval rate (tokens/s) from this run.
  • Exit the interactive session by typing:
/bye
  • Now run the Q8_0 variant with the same verbose flag:
ollama run qwen2.5:3b:q8_0 --verbose
  • Type the same prompt and press Enter:
Explain the difference between compiled and interpreted programming languages. Give two examples of each.
  • Write down the eval rate from this run.
  • Compare the two numbers. Q4_K_M should be noticeably faster than Q8_0 because smaller weights mean less data for the CPU to process per token.

What should I expect?

On a 16 GB CPU-only system, Q4_K_M typically generates 30-60% more tokens per second than Q8_0 for the same model. The quality difference is subtle for a 3B model. Both produce coherent answers.

Not seeing verbose output?

Make sure --verbose is on the ollama run line itself, not typed inside the interactive session. If you started the session without it, type /bye and re-run with the flag.

Help me debug missing verbose output in Ollama.

Monitor actual memory usage during inference

Your formulas predicted specific memory usage for each quantization level. Now you will check those predictions against what your system actually reports.

  • Keep the Q8_0 session running (do not exit yet).
  • Open a second terminal window by pressing Ctrl+Alt+T or opening a new tab in your terminal.
  • Check what Ollama has loaded by running:
ollama ps

You should see a table with columns for NAME, ID, SIZE, PROCESSOR, and UNTIL.

What does ollama ps tell you?

  • SIZE shows how much memory the loaded model occupies. Compare this to your weight memory calculation.
  • PROCESSOR shows whether the model is running on CPU or GPU. On your system, this shows 100% CPU.
  • UNTIL shows how long before the model auto-unloads from memory due to inactivity.
  • Check system memory while the model is loaded by running:
free -h
  • Compare the "available" value now to your baseline from before you ran the model.
  • Calculate the difference. This is the actual memory consumed by inference.

Now compare your prediction to reality. For qwen2.5:3b at Q8_0 (3B parameters, ~8.5 bits per weight): your formula predicts about 3.19 GB for weights alone. The actual SIZE in ollama ps should be close to that value.

  • Exit the Q8_0 session by typing /bye in the first terminal.
  • Run the Q4_K_M variant:
ollama run qwen2.5:3b --verbose
  • In your second terminal, run ollama ps again and note the SIZE column.
  • For Q4_K_M (3B parameters, ~4.89 bits per weight): your formula predicts about 1.83 GB for weights. The SIZE value should be noticeably smaller than the Q8_0 value.

Why might the numbers not match exactly?

Ollama's reported SIZE includes more than just raw weights. It bundles the KV cache allocation and runtime buffers into that number. Your prediction covers weights only. Expect the reported value to be somewhat higher than your weight-only calculation.

ollama ps shows nothing?

Models auto-unload after a few minutes of inactivity. If you waited too long between running the model and checking ollama ps, the model may have unloaded. Send another prompt in the interactive session first, then immediately check ollama ps in your second terminal.

Help me see loaded models with ollama ps.

Test context window impact on memory and speed

The KV cache is the hidden memory cost that grows with context length. You will now observe this directly by running the same model at three different num_ctx values.

  • If you exited the Q4_K_M session, start it again:
ollama run qwen2.5:3b --verbose
  • Set the context window to 2048 tokens by typing this inside the interactive session:
/set parameter num_ctx 2048
  • Run the same benchmark prompt:
Explain the difference between compiled and interpreted programming languages. Give two examples of each.
  • Note the eval rate (tokens/s).
  • In your second terminal, run free -h and note the available memory.
  • Now increase the context window to 4096 tokens:
/set parameter num_ctx 4096
  • Run the same prompt again and record the eval rate and memory usage.
  • Increase context to 8192 tokens:
/set parameter num_ctx 8192
  • Run the prompt one more time and record eval rate and memory.
  • Compare your three sets of numbers. You should see memory increase with each context jump, and speed may decrease slightly at higher context values.

What is happening to memory?

Each time you increase num_ctx, Ollama allocates a larger KV cache buffer. For qwen2.5:3b (36 layers, 2 KV heads, 128 head_dim), your formula predicts:

  • At 2048 tokens: ~0.018 GB KV cache.
  • At 4096 tokens: ~0.036 GB KV cache.
  • At 8192 tokens: ~0.072 GB KV cache.

For this small model, the KV cache is tiny even at 8192 tokens. For larger models like Llama 3.1 8B (32 layers, 8 KV heads), the cache grows 8x faster per context token. That is where context length becomes a real constraint on 16 GB systems.

  • Exit the session by typing /bye.

Not seeing memory differences between context values?

For the 3B model, the KV cache differences are small (tens of megabytes). The changes may be hard to spot in free -h output because other processes also fluctuate. The key takeaway is understanding the formula. The dramatic differences show up on larger models with more KV heads.

Help me understand why I'm not seeing memory changes between context lengths.

You now have real data showing how quantization and context length affect performance on your hardware. Next, you will use these insights to configure Ollama for optimal CPU-only inference speed.

Optimize Ollama for CPU-Only Inference

In the last step, you collected hard data on how different quantization levels and context lengths affect memory and speed on your system. Now you can use that data to make deliberate configuration choices.

Ollama ships with defaults that work across many hardware profiles. For a CPU-only system with 16 GB of RAM, those defaults leave performance on the table. In this step, you will create a custom Modelfile that locks in optimal settings and configure the Ollama server itself for maximum CPU throughput.

In this step, get ready to:
  • Create a custom Modelfile tuned for your 16 GB CPU-only system.
  • Configure Ollama server environment variables for CPU performance.
  • Benchmark the optimized configuration against defaults.
Create a custom Modelfile for your system

A Modelfile is Ollama's equivalent of a Dockerfile. It defines a base model and overrides runtime parameters like context window size and thread count.

Why a Modelfile instead of CLI flags?

CLI flags apply once per session and are easy to forget. A Modelfile bakes your optimal settings into a named model you can run anytime with zero extra typing.

It also documents your reasoning for future reference.

Before writing the Modelfile, you need to know how many CPU cores your system has. Ollama uses this to parallelize computation across threads.

  • Check your CPU core count by running this command:
nproc

What does this command do?

The nproc command prints the number of processing units (CPU cores) available on your system. You will use this number for the num_thread parameter in your Modelfile.

  • Note the number that appears. This is your core count.
  • Create a file called Modelfile.optimized by running this command:
cat > Modelfile.optimized << 'EOF'
FROM qwen2.5:3b
PARAMETER num_ctx 4096
PARAMETER num_thread [[CPU_CORES="your core count from nproc"]]
EOF

What does this Modelfile do?

  • FROM qwen2.5:3b sets the base model to the Q4_K_M variant you already pulled.
  • PARAMETER num_ctx 4096 sets the context window to 4096 tokens. From your benchmarks, this uses a small amount of KV cache while giving you enough context for most tasks.
  • PARAMETER num_thread tells Ollama how many CPU threads to use during inference. Matching this to your core count ensures full utilization without oversubscription.

Choosing num_ctx for your RAM budget

With 12 GB usable and a 3B Q4_K_M model weighing roughly 1.8 GB, you have significant headroom after overhead. Setting num_ctx to 4096 keeps response times snappy for interactive use while leaving plenty of room for the OS and other processes.

  • Build the optimized model by running this command:
ollama create optimized-3b -f Modelfile.optimized

What does this command do?

The ollama create command reads your Modelfile and registers a new model named optimized-3b with those parameters baked in. It does not re-download weights.

It creates a reference that points to the same base model with your custom configuration applied.

  • Verify your new model appears by running:
ollama list
  • You should see optimized-3b alongside your existing qwen2.5:3b entries in the output.

Don't see optimized-3b in the list?

  • Check that Modelfile.optimized exists in your current directory. Run ls Modelfile.optimized to confirm.
  • Make sure you used num_thread (singular, no trailing 's'). Using num_threads gives an unknown parameter error.
  • Confirm the FROM line references a model you have pulled (qwen2.5:3b).

Help me troubleshoot my Ollama Modelfile creation error.

Configure Ollama server environment variables

The Modelfile controls per-model settings. Server-level environment variables control how Ollama itself behaves, regardless of which model you load.

On Linux, Ollama runs as a systemd service. You configure its environment variables through the service override file.

  • Open the Ollama service editor by running this command:
sudo systemctl edit ollama.service

What does this command do?

This opens a text editor for the systemd override file. Any settings you add here persist across Ollama updates, unlike editing the main service file directly.

  • In the editor that opens, paste the following content between the comment lines:
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=4096"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_FLASH_ATTENTION=1"

What do these variables control?

  • OLLAMA_CONTEXT_LENGTH=4096 sets the server-wide default context window. Any model loaded without an explicit num_ctx will use 4096 tokens.
  • OLLAMA_NUM_PARALLEL=1 limits concurrent request handling to one. RAM scales by OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH, so keeping this at 1 on a 16 GB system prevents memory exhaustion.
  • OLLAMA_FLASH_ATTENTION=1 enables an optimized attention computation path. This reduces memory usage and can improve inference speed even on CPU.
  • Save and exit the editor.
  • Reload the systemd daemon and restart Ollama by running these commands:
sudo systemctl daemon-reload
sudo systemctl restart ollama
  • Verify Ollama restarted successfully by running:
ollama list
  • You should see your models listed, confirming the server is back up.

Ollama not responding after restart?

  • Check the service status with sudo systemctl status ollama to see if there are error messages.
  • Make sure your override file has the [Service] header. Without it, systemd ignores the Environment lines.
  • Verify you wrapped each value in quotes: Environment="VARNAME=value".

Help me fix my Ollama systemd environment variable configuration.

Test your optimized configuration against defaults

Configuration is only useful if you can measure the improvement. You will run the same prompt through both the default and optimized models, capturing tokens-per-second to see the difference.

  • Run a prompt through the default model with verbose output by running:
ollama run qwen2.5:3b --verbose "Explain the difference between TCP and UDP in exactly three sentences."
  • Note the tokens/second from the output (look for the eval rate line).
  • Now run the same prompt through your optimized model:
ollama run optimized-3b --verbose "Explain the difference between TCP and UDP in exactly three sentences."

What should I compare?

Focus on three metrics between the two runs:

  • Tokens per second (eval rate) tells you raw generation speed.
  • Response quality should be identical since both use the same base weights.
  • Memory usage (check with ollama ps while a model is loaded) shows the effect of your context window and flash attention settings.
  • Check memory usage while the optimized model is loaded by running:
ollama ps
  • Compare the Size column output to what you saw during your Step 3 benchmarks.
  • Also check system RAM by running:
free -h
  • Document the results. Write down the tokens/second for default vs optimized, and the RAM usage for each.

Which setting makes the biggest difference?

On CPU-only systems, num_thread matching your core count typically provides the largest speed improvement. Flash attention reduces memory overhead.

Context length mainly affects how much RAM is reserved, not raw speed, unless you were previously overflowing into swap.

Not seeing a speed improvement?

  • Ensure you ran the optimized model (optimized-3b), not the default qwen2.5:3b again.
  • Confirm your num_thread value matches nproc output. A mismatch leaves cores idle.
  • Check that the Ollama service actually restarted with your new environment variables. Run sudo systemctl show ollama | grep -i environment to verify they are loaded.

Help me understand why my optimized Ollama config is not faster than the default.

You now have a fully tuned Ollama setup that squeezes the best performance out of your 16 GB CPU-only system. Next up, you will take everything you have learned about memory math, architecture inspection, and configuration and build it into a reusable Python tool.

Build the Model Advisor Tool

You now have the memory formulas and real benchmark data to back them up. The missing piece is a tool that puts it all together so you never have to do the math by hand again.

In this step, you will build a Python command-line tool that takes any model's specs and any hardware configuration, then instantly tells you whether the model fits, how much headroom you have, and what your maximum context length is.

In this step, get ready to:
  • Implement the core memory calculation functions in Python.
  • Add feasibility assessment and maximum context length logic.
  • Build a recommendation engine and formatted report output.
Write the core calculation engine

The foundation of your advisor tool is a set of reference data and calculation functions that implement the memory formulas you learned in Step 2. You will encode the effective bits-per-weight for each GGUF quantization level and store common model architectures so the tool can look them up instantly.

  • Create a new file called model_advisor.py in your project directory by running:
touch model_advisor.py
  • Open model_advisor.py in your text editor and add the following reference data:
#!/usr/bin/env python3
"""Local AI Model Advisor - Calculate whether a model fits your hardware."""

import json
import sys

# Effective bits-per-weight for common GGUF quantization levels
# These are measured averages, not theoretical minimums
QUANT_BPW = {
    "Q2_K": 2.97,
    "Q3_K_S": 3.50,
    "Q3_K_M": 3.91,
    "Q4_K_S": 4.67,
    "Q4_K_M": 4.89,
    "Q5_K_S": 5.54,
    "Q5_K_M": 5.70,
    "Q6_K": 6.57,
    "Q8_0": 8.50,
    "FP16": 16.00,
}

# Common model architectures (layers, kv_heads, head_dim)
MODEL_ARCHITECTURES = {
    "llama-3.1-8b": {"layers": 32, "kv_heads": 8, "head_dim": 128, "params_b": 8.0},
    "llama-3.1-70b": {"layers": 80, "kv_heads": 8, "head_dim": 128, "params_b": 70.0},
    "qwen2.5-7b": {"layers": 28, "kv_heads": 4, "head_dim": 128, "params_b": 7.0},
    "qwen2.5-3b": {"layers": 36, "kv_heads": 2, "head_dim": 128, "params_b": 3.0},
    "mistral-7b": {"layers": 32, "kv_heads": 8, "head_dim": 128, "params_b": 7.0},
    "gemma-2-9b": {"layers": 42, "kv_heads": 8, "head_dim": 256, "params_b": 9.0},
}

What does this code do?

QUANT_BPW maps each quantization level name to its measured effective bits-per-weight. These are averages, not exact values, because K-quant formats use mixed precision across blocks. For example, Q4_K_M averages about 4.89 bits per weight, not exactly 4.

MODEL_ARCHITECTURES stores the values you extracted with ollama show and the API in earlier steps: layer count, KV head count, head dimension, and parameter count. These feed directly into the memory formulas.

  • Save model_advisor.py.
  • You won't see output yet. The next chunk adds the functions that use this data.

Now you need the three functions that implement the memory formulas you learned in Step 2. Each function handles one component: weight memory, KV cache memory, and total memory.

  • In model_advisor.py, add the following three functions below the MODEL_ARCHITECTURES dictionary:

def calculate_weight_memory_gb(params_billions: float, quant: str) -> float:
    """Calculate model weight memory in GB."""
    if quant not in QUANT_BPW:
        raise ValueError(f"Unknown quantization: {quant}. Options: {list(QUANT_BPW.keys())}")
    bpw = QUANT_BPW[quant]
    return params_billions * (bpw / 8)


def calculate_kv_cache_gb(
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    context_length: int,
    bytes_per_element: float = 2.0,
) -> float:
    """Calculate KV cache memory in GB.

    Formula: 2 * layers * kv_heads * head_dim * context_length * bytes_per_element
    The leading 2 accounts for separate Key and Value tensors.
    """
    kv_bytes = 2 * num_layers * num_kv_heads * head_dim * context_length * bytes_per_element
    return kv_bytes / (1024**3)


def calculate_total_memory_gb(
    params_billions: float,
    quant: str,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    context_length: int,
    overhead_gb: float = 1.5,
) -> dict:
    """Calculate total memory required and return breakdown."""
    weight_mem = calculate_weight_memory_gb(params_billions, quant)
    kv_cache = calculate_kv_cache_gb(num_layers, num_kv_heads, head_dim, context_length)
    total = weight_mem + kv_cache + overhead_gb

    return {
        "weight_memory_gb": round(weight_mem, 2),
        "kv_cache_gb": round(kv_cache, 2),
        "overhead_gb": overhead_gb,
        "total_gb": round(total, 2),
    }

What does this code do?

  • calculate_weight_memory_gb() implements the weight formula: parameters * (bits_per_weight / 8). It looks up the effective bpw from your QUANT_BPW dict and raises an error if the quantization is unknown.
  • calculate_kv_cache_gb() implements the KV cache formula. The leading 2 in the formula accounts for both the Key tensor and the Value tensor stored at each layer.
  • calculate_total_memory_gb() combines both formulas and adds a 1.5 GB overhead estimate. It returns a dictionary breaking down each component so you can see exactly where memory goes.
  • Save model_advisor.py.
  • You won't see output yet. The next substep adds the assessment logic that uses these functions.

Seeing a syntax error?

Make sure there is a blank line between each function definition. Python requires consistent indentation (4 spaces per level). Check that the closing parentheses on calculate_kv_cache_gb and calculate_total_memory_gb have the correct multi-line parameter formatting.

Help me fix a syntax error in my model_advisor.py calculation functions

Add the feasibility assessment logic

Knowing a model's total memory requirement is only half the answer. You also need to compare it against your actual hardware and figure out the maximum context length your system can support.

  • In model_advisor.py, add these two functions below calculate_total_memory_gb():

def assess_feasibility(
    total_memory_gb: float,
    system_ram_gb: float,
    usable_fraction: float = 0.75,
    gpu_vram_gb: float = 0.0,
) -> dict:
    """Assess whether a model fits on the given hardware."""
    if gpu_vram_gb > 0:
        available = gpu_vram_gb * 0.9
        target = "GPU"
    else:
        available = system_ram_gb * usable_fraction
        target = "CPU (system RAM)"

    fits = total_memory_gb <= available
    headroom = available - total_memory_gb

    return {
        "fits": fits,
        "target": target,
        "available_gb": round(available, 2),
        "required_gb": round(total_memory_gb, 2),
        "headroom_gb": round(headroom, 2),
    }


def find_max_context(
    params_billions: float,
    quant: str,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    available_gb: float,
    overhead_gb: float = 1.5,
) -> int:
    """Find the maximum context length that fits in available memory."""
    weight_mem = calculate_weight_memory_gb(params_billions, quant)
    remaining = available_gb - weight_mem - overhead_gb

    if remaining <= 0:
        return 0

    # Solve for context_length from remaining GB
    # remaining_bytes = 2 * layers * kv_heads * head_dim * context * bytes_per_elem
    bytes_per_element = 2.0
    remaining_bytes = remaining * (1024**3)
    context = remaining_bytes / (2 * num_layers * num_kv_heads * head_dim * bytes_per_element)

    return int(context)

What does this code do?

  • assess_feasibility() compares the total memory against what your hardware actually offers. For CPU-only systems, it uses 75% of RAM (the rest is reserved for the OS and other processes). For GPU systems, it uses 90% of VRAM.
  • find_max_context() works backwards from the available memory. It subtracts the weight memory and overhead, then solves the KV cache formula in reverse to find the largest context length that fits in the remaining space.
  • Save model_advisor.py.
  • You won't see output yet. The final substep adds the recommendation engine and report output that ties everything together.
Add practical recommendations and run the tool

The final piece is a recommendation engine that scans all known models and quantizations to find what fits your hardware. It also includes a formatted report printer and a main() function that demonstrates the tool with your 16 GB system.

  • In model_advisor.py, add the recommendation function below find_max_context():

def recommend_models(system_ram_gb: float, gpu_vram_gb: float = 0.0) -> list:
    """Recommend feasible model + quantization combinations."""
    if gpu_vram_gb > 0:
        available = gpu_vram_gb * 0.9
    else:
        available = system_ram_gb * 0.75

    recommendations = []
    target_context = 4096

    for model_name, arch in MODEL_ARCHITECTURES.items():
        for quant in ["Q4_K_M", "Q5_K_M", "Q8_0"]:
            mem = calculate_total_memory_gb(
                arch["params_b"],
                quant,
                arch["layers"],
                arch["kv_heads"],
                arch["head_dim"],
                target_context,
            )
            if mem["total_gb"] <= available:
                max_ctx = find_max_context(
                    arch["params_b"],
                    quant,
                    arch["layers"],
                    arch["kv_heads"],
                    arch["head_dim"],
                    available,
                )
                recommendations.append({
                    "model": model_name,
                    "quant": quant,
                    "total_gb": mem["total_gb"],
                    "max_context": min(max_ctx, 131072),
                    "headroom_gb": round(available - mem["total_gb"], 2),
                })

    recommendations.sort(key=lambda x: (-x["max_context"], x["total_gb"]))
    return recommendations

What does this code do?

recommend_models() loops through every model in MODEL_ARCHITECTURES and tests three common quantization levels (Q4_K_M, Q5_K_M, Q8_0). For each combination that fits, it calculates the maximum context length and stores it as a recommendation.

The results are sorted so the combination with the largest context window appears first. This gives you an instant answer to "what is the best model I can run on this hardware?"

  • Save model_advisor.py.

Now add the report printer that formats the output for the terminal, followed by a main() function that demonstrates the tool on your system.

  • Add the following below recommend_models():

def print_report(
    model_name: str,
    quant: str,
    context_length: int,
    system_ram_gb: float,
    gpu_vram_gb: float = 0.0,
):
    """Print a full feasibility report."""
    if model_name in MODEL_ARCHITECTURES:
        arch = MODEL_ARCHITECTURES[model_name]
    else:
        print(f"Unknown model: {model_name}")
        print(f"Available models: {list(MODEL_ARCHITECTURES.keys())}")
        return

    mem = calculate_total_memory_gb(
        arch["params_b"],
        quant,
        arch["layers"],
        arch["kv_heads"],
        arch["head_dim"],
        context_length,
    )

    feasibility = assess_feasibility(mem["total_gb"], system_ram_gb, gpu_vram_gb=gpu_vram_gb)

    max_ctx = find_max_context(
        arch["params_b"],
        quant,
        arch["layers"],
        arch["kv_heads"],
        arch["head_dim"],
        feasibility["available_gb"],
    )

    print("=" * 60)
    print("LOCAL AI MODEL ADVISOR")
    print("=" * 60)
    print(f"\nModel: {model_name} ({arch['params_b']}B parameters)")
    print(f"Quantization: {quant} ({QUANT_BPW[quant]} bits/weight)")
    print(f"Context Length: {context_length} tokens")
    print(f"\n--- Memory Breakdown ---")
    print(f"  Model weights: {mem['weight_memory_gb']:.2f} GB")
    print(f"  KV cache:      {mem['kv_cache_gb']:.2f} GB")
    print(f"  Overhead:      {mem['overhead_gb']:.2f} GB")
    print(f"  TOTAL:         {mem['total_gb']:.2f} GB")
    print(f"\n--- Hardware Assessment ---")
    print(f"  Target: {feasibility['target']}")
    print(f"  Available: {feasibility['available_gb']:.2f} GB")
    print(f"  Required:  {feasibility['required_gb']:.2f} GB")
    print(f"  Headroom:  {feasibility['headroom_gb']:.2f} GB")

    if feasibility["fits"]:
        print(f"\n  [PASS] Model fits with {feasibility['headroom_gb']:.1f} GB to spare")
        print(f"  Max context at this quant: {min(max_ctx, 131072)} tokens")
    else:
        print(f"\n  [FAIL] Model does NOT fit. Need {abs(feasibility['headroom_gb']):.1f} GB more.")
        print(f"  Max context that WOULD fit: {max_ctx} tokens")
        if max_ctx >= 2048:
            print(f"  Suggestion: Reduce context to {max_ctx} tokens")
        else:
            print(f"  Suggestion: Try a smaller quantization or smaller model")

    print("=" * 60)

What does this code do?

print_report() is the human-readable frontend. It takes a model name, quantization, context length, and hardware specs, then prints a structured report showing the memory breakdown, whether the model fits, and what the maximum context length would be.

The [PASS] or [FAIL] verdict gives you an instant answer. When a model does not fit, it suggests either reducing context or switching to a smaller quantization.

  • Add the main() function and entry point at the very bottom of the file:

def main():
    """Run the advisor with example configurations."""
    print("\n--- Example: Your System (16GB RAM, no GPU) ---\n")
    print_report("llama-3.1-8b", "Q4_K_M", 4096, system_ram_gb=16)

    print("\n\n--- Example: Same model, larger context ---\n")
    print_report("llama-3.1-8b", "Q4_K_M", 32768, system_ram_gb=16)

    print("\n\n--- Example: Too-large model for your system ---\n")
    print_report("llama-3.1-70b", "Q4_K_M", 4096, system_ram_gb=16)

    print("\n\n--- Recommendations for 16GB RAM (no GPU) ---\n")
    recs = recommend_models(system_ram_gb=16)
    print(f"{'Model':<20} {'Quant':<8} {'Memory':<10} {'Max Context':<12} {'Headroom'}")
    print("-" * 65)
    for r in recs[:10]:
        print(f"{r['model']:<20} {r['quant']:<8} {r['total_gb']:<10.1f} {r['max_context']:<12} {r['headroom_gb']:.1f} GB")


if __name__ == "__main__":
    main()

What does this code do?

main() demonstrates three scenarios matching your 16 GB system: an 8B model at a comfortable context (should pass), the same model at 32K context (should be tight or fail), and a 70B model (should definitely fail). It finishes with a recommendations table showing all feasible combinations.

  • Save model_advisor.py.
  • Run the tool by executing:
python3 model_advisor.py
  • You should see four sections of output: a [PASS] report for llama-3.1-8b at 4096 context, a second report for the same model at 32768 context, a [FAIL] report for llama-3.1-70b, and a recommendations table listing feasible models.

What should I see?

The first report should show llama-3.1-8b at Q4_K_M fitting comfortably in 12 GB of available RAM with roughly 5 GB of headroom. The 70B model report should show a clear [FAIL] requiring far more memory than your system offers.

The recommendations table should list models sorted by maximum context length. Compare the memory values against what you observed during benchmarking in Step 3. They should be close to your real measurements.

Script not producing output?

  • If you see IndentationError, check that all functions are indented with exactly 4 spaces and there is a blank line between each function definition.
  • If you see NameError, make sure the functions are defined in order. calculate_total_memory_gb() calls calculate_weight_memory_gb() and calculate_kv_cache_gb(), so those must appear earlier in the file.
  • If the script runs but the numbers seem wrong, double-check that the QUANT_BPW values match exactly. A small difference in bits-per-weight produces noticeable drift in the final GB calculation.

Help me debug my model_advisor.py script

✔️ Awesome, I've got everything!

Great. Double check you have saved model_advisor.py and the output includes both [PASS] and [FAIL] verdicts plus the recommendations table.

ⓧ I'd like to double check the full code

Here is the complete model_advisor.py for comparison:

#!/usr/bin/env python3
"""Local AI Model Advisor - Calculate whether a model fits your hardware."""

import json
import sys

# Effective bits-per-weight for common GGUF quantization levels
# These are measured averages, not theoretical minimums
QUANT_BPW = {
    "Q2_K": 2.97,
    "Q3_K_S": 3.50,
    "Q3_K_M": 3.91,
    "Q4_K_S": 4.67,
    "Q4_K_M": 4.89,
    "Q5_K_S": 5.54,
    "Q5_K_M": 5.70,
    "Q6_K": 6.57,
    "Q8_0": 8.50,
    "FP16": 16.00,
}

# Common model architectures (layers, kv_heads, head_dim)
MODEL_ARCHITECTURES = {
    "llama-3.1-8b": {"layers": 32, "kv_heads": 8, "head_dim": 128, "params_b": 8.0},
    "llama-3.1-70b": {"layers": 80, "kv_heads": 8, "head_dim": 128, "params_b": 70.0},
    "qwen2.5-7b": {"layers": 28, "kv_heads": 4, "head_dim": 128, "params_b": 7.0},
    "qwen2.5-3b": {"layers": 36, "kv_heads": 2, "head_dim": 128, "params_b": 3.0},
    "mistral-7b": {"layers": 32, "kv_heads": 8, "head_dim": 128, "params_b": 7.0},
    "gemma-2-9b": {"layers": 42, "kv_heads": 8, "head_dim": 256, "params_b": 9.0},
}


def calculate_weight_memory_gb(params_billions: float, quant: str) -> float:
    """Calculate model weight memory in GB."""
    if quant not in QUANT_BPW:
        raise ValueError(f"Unknown quantization: {quant}. Options: {list(QUANT_BPW.keys())}")
    bpw = QUANT_BPW[quant]
    return params_billions * (bpw / 8)


def calculate_kv_cache_gb(
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    context_length: int,
    bytes_per_element: float = 2.0,
) -> float:
    """Calculate KV cache memory in GB.

    Formula: 2 * layers * kv_heads * head_dim * context_length * bytes_per_element
    The leading 2 accounts for separate Key and Value tensors.
    """
    kv_bytes = 2 * num_layers * num_kv_heads * head_dim * context_length * bytes_per_element
    return kv_bytes / (1024**3)


def calculate_total_memory_gb(
    params_billions: float,
    quant: str,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    context_length: int,
    overhead_gb: float = 1.5,
) -> dict:
    """Calculate total memory required and return breakdown."""
    weight_mem = calculate_weight_memory_gb(params_billions, quant)
    kv_cache = calculate_kv_cache_gb(num_layers, num_kv_heads, head_dim, context_length)
    total = weight_mem + kv_cache + overhead_gb

    return {
        "weight_memory_gb": round(weight_mem, 2),
        "kv_cache_gb": round(kv_cache, 2),
        "overhead_gb": overhead_gb,
        "total_gb": round(total, 2),
    }


def assess_feasibility(
    total_memory_gb: float,
    system_ram_gb: float,
    usable_fraction: float = 0.75,
    gpu_vram_gb: float = 0.0,
) -> dict:
    """Assess whether a model fits on the given hardware."""
    if gpu_vram_gb > 0:
        available = gpu_vram_gb * 0.9
        target = "GPU"
    else:
        available = system_ram_gb * usable_fraction
        target = "CPU (system RAM)"

    fits = total_memory_gb <= available
    headroom = available - total_memory_gb

    return {
        "fits": fits,
        "target": target,
        "available_gb": round(available, 2),
        "required_gb": round(total_memory_gb, 2),
        "headroom_gb": round(headroom, 2),
    }


def find_max_context(
    params_billions: float,
    quant: str,
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    available_gb: float,
    overhead_gb: float = 1.5,
) -> int:
    """Find the maximum context length that fits in available memory."""
    weight_mem = calculate_weight_memory_gb(params_billions, quant)
    remaining = available_gb - weight_mem - overhead_gb

    if remaining <= 0:
        return 0

    # Solve for context_length from remaining GB
    # remaining_bytes = 2 * layers * kv_heads * head_dim * context * bytes_per_elem
    bytes_per_element = 2.0
    remaining_bytes = remaining * (1024**3)
    context = remaining_bytes / (2 * num_layers * num_kv_heads * head_dim * bytes_per_element)

    return int(context)


def recommend_models(system_ram_gb: float, gpu_vram_gb: float = 0.0) -> list:
    """Recommend feasible model + quantization combinations."""
    if gpu_vram_gb > 0:
        available = gpu_vram_gb * 0.9
    else:
        available = system_ram_gb * 0.75

    recommendations = []
    target_context = 4096

    for model_name, arch in MODEL_ARCHITECTURES.items():
        for quant in ["Q4_K_M", "Q5_K_M", "Q8_0"]:
            mem = calculate_total_memory_gb(
                arch["params_b"],
                quant,
                arch["layers"],
                arch["kv_heads"],
                arch["head_dim"],
                target_context,
            )
            if mem["total_gb"] <= available:
                max_ctx = find_max_context(
                    arch["params_b"],
                    quant,
                    arch["layers"],
                    arch["kv_heads"],
                    arch["head_dim"],
                    available,
                )
                recommendations.append({
                    "model": model_name,
                    "quant": quant,
                    "total_gb": mem["total_gb"],
                    "max_context": min(max_ctx, 131072),
                    "headroom_gb": round(available - mem["total_gb"], 2),
                })

    recommendations.sort(key=lambda x: (-x["max_context"], x["total_gb"]))
    return recommendations


def print_report(
    model_name: str,
    quant: str,
    context_length: int,
    system_ram_gb: float,
    gpu_vram_gb: float = 0.0,
):
    """Print a full feasibility report."""
    if model_name in MODEL_ARCHITECTURES:
        arch = MODEL_ARCHITECTURES[model_name]
    else:
        print(f"Unknown model: {model_name}")
        print(f"Available models: {list(MODEL_ARCHITECTURES.keys())}")
        return

    mem = calculate_total_memory_gb(
        arch["params_b"],
        quant,
        arch["layers"],
        arch["kv_heads"],
        arch["head_dim"],
        context_length,
    )

    feasibility = assess_feasibility(mem["total_gb"], system_ram_gb, gpu_vram_gb=gpu_vram_gb)

    max_ctx = find_max_context(
        arch["params_b"],
        quant,
        arch["layers"],
        arch["kv_heads"],
        arch["head_dim"],
        feasibility["available_gb"],
    )

    print("=" * 60)
    print("LOCAL AI MODEL ADVISOR")
    print("=" * 60)
    print(f"\nModel: {model_name} ({arch['params_b']}B parameters)")
    print(f"Quantization: {quant} ({QUANT_BPW[quant]} bits/weight)")
    print(f"Context Length: {context_length} tokens")
    print(f"\n--- Memory Breakdown ---")
    print(f"  Model weights: {mem['weight_memory_gb']:.2f} GB")
    print(f"  KV cache:      {mem['kv_cache_gb']:.2f} GB")
    print(f"  Overhead:      {mem['overhead_gb']:.2f} GB")
    print(f"  TOTAL:         {mem['total_gb']:.2f} GB")
    print(f"\n--- Hardware Assessment ---")
    print(f"  Target: {feasibility['target']}")
    print(f"  Available: {feasibility['available_gb']:.2f} GB")
    print(f"  Required:  {feasibility['required_gb']:.2f} GB")
    print(f"  Headroom:  {feasibility['headroom_gb']:.2f} GB")

    if feasibility["fits"]:
        print(f"\n  [PASS] Model fits with {feasibility['headroom_gb']:.1f} GB to spare")
        print(f"  Max context at this quant: {min(max_ctx, 131072)} tokens")
    else:
        print(f"\n  [FAIL] Model does NOT fit. Need {abs(feasibility['headroom_gb']):.1f} GB more.")
        print(f"  Max context that WOULD fit: {max_ctx} tokens")
        if max_ctx >= 2048:
            print(f"  Suggestion: Reduce context to {max_ctx} tokens")
        else:
            print(f"  Suggestion: Try a smaller quantization or smaller model")

    print("=" * 60)


def main():
    """Run the advisor with example configurations."""
    print("\n--- Example: Your System (16GB RAM, no GPU) ---\n")
    print_report("llama-3.1-8b", "Q4_K_M", 4096, system_ram_gb=16)

    print("\n\n--- Example: Same model, larger context ---\n")
    print_report("llama-3.1-8b", "Q4_K_M", 32768, system_ram_gb=16)

    print("\n\n--- Example: Too-large model for your system ---\n")
    print_report("llama-3.1-70b", "Q4_K_M", 4096, system_ram_gb=16)

    print("\n\n--- Recommendations for 16GB RAM (no GPU) ---\n")
    recs = recommend_models(system_ram_gb=16)
    print(f"{'Model':<20} {'Quant':<8} {'Memory':<10} {'Max Context':<12} {'Headroom'}")
    print("-" * 65)
    for r in recs[:10]:
        print(f"{r['model']:<20} {r['quant']:<8} {r['total_gb']:<10.1f} {r['max_context']:<12} {r['headroom_gb']:.1f} GB")


if __name__ == "__main__":
    main()

Your Model Advisor tool is working. You now have a script that can instantly answer "will this model run on my machine?" for any combination of model, quantization, and hardware. Next up, the Secret Mission extends this tool to calculate fine-tuning requirements and recommend GPUs.

Secret mission

Add Fine-Tuning Cost Calculator

Your Model Advisor can tell you whether a model fits for inference. But what about fine-tuning? Extend the tool to calculate training VRAM requirements for QLoRA, LoRA, and full fine-tuning, then recommend the cheapest GPU that can handle the job.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The only resources using disk space are the Ollama models you pulled and the Python file you created.

Decide whether to keep your resources as they are, pause them to free up memory, or delete them entirely.

Resources you used:

  • Ollama models pulled locally (qwen2.5:3b at Q4_K_M and Q8_0, plus your custom optimized model).
  • Ollama environment variables (OLLAMA_CONTEXT_LENGTH, OLLAMA_NUM_PARALLEL, OLLAMA_FLASH_ATTENTION) configured in the systemd service file.
  • Local project files (model_advisor.py, Modelfile.optimized).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with local models or want to use your advisor tool again.

  • Your pulled Ollama models stay available for instant use without re-downloading.
  • Your optimized Modelfile and environment variables keep Ollama tuned for your hardware.
  • Your model_advisor.py script is ready to assess any new model you find.

Pause - I'll come back to this later

Free up RAM by stopping any loaded models. Your files and pulled models stay on disk for next time.

  • Stop any running models to free RAM by running:
  • Check which models are loaded with ollama ps, then stop each one.
ollama stop qwen2.5:3b

Ollama will automatically unload models after 5 minutes of inactivity, so this step is optional if you simply close your terminal and wait.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space.

  • Remove the pulled Ollama models by running these commands:
ollama rm qwen2.5:3b
ollama rm qwen2.5:3b:q8_0
  • If you created a custom model from your Modelfile, remove it as well:
ollama rm optimized-qwen
  • Revert the Ollama environment variables by editing the systemd service file:
sudo systemctl edit ollama.service

Remove any lines you added that begin with Environment= (OLLAMA_CONTEXT_LENGTH, OLLAMA_NUM_PARALLEL, OLLAMA_FLASH_ATTENTION). Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama
  • Delete your local project files by running:
rm model_advisor.py Modelfile.optimized

Nice Work!

Nice Work!

You just built a complete mental framework and a working tool for local AI model selection. You can now look at any model on Hugging Face or Ollama's library, read its specs, and immediately know whether it will run on a given machine.

You've learned how to:

  • Calculate model memory requirements from first principles using parameter count, quantization level, and architecture details like layers, KV heads, and head dimension.
  • Benchmark quantization levels on real hardware and measure the speed, quality, and memory tradeoffs between them.
  • Build a Python feasibility tool that takes any hardware configuration and model spec, then outputs whether the model fits, how much headroom exists, and what context length is achievable.
  • Secret Mission: Extended the advisor tool to calculate fine-tuning VRAM requirements for QLoRA, LoRA, and full fine-tuning, and recommend the cheapest GPU for the job.

Ready to quiz yourself?