Build a Self-Learning Local LLM Router

Build AutoPili, a router that learns which local LLM handles each task best.

Introduction

30 Second Summary

You instinctively know which friend to call for specific advice. One is great at directions, another at emotional support. What if your computer could learn the same thing about its AI models?

In this project, you will build AutoPili, a Python application that learns which local LLM is best at handling different types of prompts. It uses a self-teaching algorithm called Thompson Sampling to get smarter with every query it routes.

What You'll Build

AutoPili is a local LLM router that classifies prompts by type, selects the best model using Thompson Sampling, and improves its routing decisions automatically over time.

By the end of this project, you'll have:

  • A prompt classifier that categorizes requests into code, creative, factual, and math.
  • A Thompson Sampling bandit engine that learns which model performs best for each category.
  • A self-learning router that connects classification to model selection to execution, logging every decision into a growing performance dataset.
  • Secret Mission: Add a fourth model at runtime and watch AutoPili discover its strengths from scratch.

Are there any prerequisites?

You need basic Python knowledge (functions, classes, and dictionaries). No machine learning experience is required. The algorithm you will implement is simple enough to understand with just Python fundamentals.

Set Up Your Environment

AutoPili needs three ingredients to work: a Python environment with the right packages, a set of local LLM models to route between, and a project structure to hold your code.

In this step, you will install the Ollama Python SDK and NumPy, pull three models of different sizes and specialties, and scaffold an empty project directory.

In this step, get ready to:
  • Install the Python packages AutoPili depends on.
  • Pull three Ollama models so they are available locally.
  • Create the project directory and empty Python files.
Install the Python dependencies

AutoPili uses the official Ollama Python SDK to communicate with your local models and NumPy to sample from probability distributions during Thompson Sampling.

  • Press Cmd+Space to open Spotlight search.
  • Type Terminal and press Enter to open it.
  • Install both packages by running this command:
pip install ollama==0.6.2 numpy

What does this command do?

This installs two packages from PyPI into your Python environment.

  • ollama==0.6.2 pins the Ollama SDK to version 0.6.2 so your code behaves exactly as this project expects.
  • numpy provides the numpy.random.beta() function you will use to sample from Beta distributions in the bandit engine.
  • Verify both packages import correctly by running:
python3 -c "import ollama; import numpy; print('Both packages imported successfully')"

You should see Both packages imported successfully printed in your terminal.

Seeing a ModuleNotFoundError?

  • Make sure you ran the pip install command in the same terminal session. If you opened a new tab, the environment may differ.
  • If your system uses pip3 instead of pip, re-run the install with pip3 install ollama==0.6.2 numpy.

Why can't I import ollama or numpy after installing?

Pull the three Ollama models

AutoPili routes prompts between three models, each with different strengths and sizes. You need to download them before the router can call them.

  • Pull the first model (Google Gemma 4, ~3.4GB) by running:
ollama pull batiai/gemma4-e2b:q4
  • Pull the second model (Meta Llama 3.2, ~2.0GB) by running:
ollama pull llama3.2:3b
  • Pull the third model (Alibaba Qwen Coder, ~986MB) by running:
ollama pull qwen2.5-coder:1.5b

Why these three models?

Each model brings a different specialty. Gemma 4 is a strong general-purpose model. Llama 3.2 handles creative and factual tasks well. Qwen Coder is optimized for code generation. The bandit engine will discover these strengths on its own.

The total download is roughly 6.4GB. Each pull may take a few minutes depending on your internet speed.

  • Verify all three models are available by running:
ollama list

You should see batiai/gemma4-e2b:q4, llama3.2:3b, and qwen2.5-coder:1.5b listed in the output.

A model isn't showing up in the list?

  • Check that the pull command completed without errors. Look for a progress bar that reached 100%.
  • Make sure Ollama is running. You can verify by checking for the Ollama icon in your menu bar. If it is not there, open the Ollama app first.

Help me troubleshoot why ollama pull didn't work

Create the project directory and files

Now that your tools are ready, create the project folder that will hold your four Python modules.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir autopili
cd autopili
  • Create the four empty Python files by running:
touch classifier.py bandit.py router.py main.py

What are these files for?

  • classifier.py will categorize incoming prompts by type (code, creative, math, factual).
  • bandit.py will hold the Thompson Sampling engine that learns which model performs best per category.
  • router.py will connect the classifier and bandit to Ollama, handling the full routing loop.
  • main.py will run the test suite and print the live learning dashboard.
  • Confirm the files exist by running:
ls

You should see bandit.py, classifier.py, main.py, and router.py in the output.

Don't see all four files?

  • Make sure you ran touch from inside the autopili folder. Run pwd to check your current directory. It should end with /autopili.
  • If you accidentally created files in the wrong folder, delete them and re-run the commands from the cd ~/Desktop/autopili step.

Help me figure out why my project files are missing

Your environment is locked and loaded. Next up, you will build the prompt classifier that teaches AutoPili how to recognize different types of questions.

Build the Prompt Classifier

Your environment is ready with all three models pulled and your project directory created. Now it's time to teach AutoPili how to understand what you're asking for.

Before AutoPili can route a prompt to the best model, it needs to know what kind of prompt it's looking at. A coding question needs different strengths than a creative writing request. In this step, you'll build a lightweight classifier that reads a prompt, matches it against keyword patterns, and returns both a category and a confidence score.

In this step, get ready to:
  • Build a keyword dictionary that maps prompt types to recognizable patterns.
  • Write a classification function that scores prompts and returns a confidence level.
  • Test the classifier to verify it handles each category correctly.
Define the keyword categories

AutoPili classifies prompts into four categories: code, creative, math, and factual. Each category has a list of keywords that signal what kind of task the prompt is asking for.

  • Open classifier.py in your code editor and add the keyword dictionary by pasting this code:
CATEGORY_KEYWORDS = {
    "code": [
        "function", "code", "program", "debug", "error", "syntax",
        "python", "javascript", "html", "css", "api", "class",
        "variable", "loop", "array", "algorithm", "implement",
        "refactor", "compile", "import", "def", "return"
    ],
    "creative": [
        "story", "poem", "write", "creative", "imagine", "fiction",
        "character", "narrative", "describe", "metaphor", "song",
        "dialogue", "scene", "fantasy", "adventure", "compose"
    ],
    "math": [
        "calculate", "math", "equation", "solve", "integral",
        "derivative", "probability", "statistics", "formula",
        "algebra", "geometry", "number", "sum", "multiply", "proof"
    ],
    "factual": [
        "what is", "explain", "how does", "define", "history",
        "science", "why", "difference between", "compare",
        "list", "facts", "tell me about", "describe how"
    ],
}

What does this code do?

This dictionary maps each of the four categories to a list of keywords that typically appear in that type of prompt.

  • code keywords include programming terms like "function", "debug", and language names.
  • creative keywords capture writing-related words like "story", "poem", and "imagine".
  • math keywords cover calculation terms like "integral", "solve", and "probability".
  • factual keywords include question patterns like "what is", "explain", and "how does".
  • Save classifier.py.
  • You won't see a change yet. The classify_prompt function in the next chunk uses these keywords.

Seeing a syntax error?

Make sure every keyword string has matching quotes and every list item has a trailing comma. Check that the closing brackets line up: each list ends with ], and the entire dictionary ends with }.

Help me fix a syntax error in my CATEGORY_KEYWORDS dictionary in classifier.py

Write the classification function

The classifier needs a function that takes a prompt, counts how many keywords from each category appear in it, and returns the best matching category. It also calculates a confidence score so the router knows when classification is uncertain.

  • In classifier.py, add the classify_prompt function below the CATEGORY_KEYWORDS dictionary by pasting this code:
def classify_prompt(prompt):
    prompt_lower = prompt.lower()
    scores = {}

    for category, keywords in CATEGORY_KEYWORDS.items():
        score = 0
        for keyword in keywords:
            if keyword in prompt_lower:
                score += 1
        scores[category] = score

    total = sum(scores.values())
    if total == 0:
        return "factual", 0.0

    best_category = max(scores, key=scores.get)
    confidence = scores[best_category] / total

    return best_category, confidence

What does this code do?

This function is the brain of your classifier. It works in three stages:

  • Scoring: It loops through each category's keyword list and counts how many keywords appear in the prompt. Each match adds 1 to that category's score.
  • Fallback: If no keywords match at all (total == 0), it defaults to "factual" with zero confidence. This prevents errors on ambiguous prompts.
  • Confidence: The confidence score is the winning category's score divided by the total. A prompt matching 3 code keywords and 1 factual keyword would give code a confidence of 0.75 (3 out of 4 matches).
  • Save classifier.py.
  • You'll test the full classifier in the next substep to confirm everything works.

Indentation looking off?

Python is sensitive to indentation. Make sure the for loop and if statement inside classify_prompt are indented with 4 spaces. The return statements should be at the same level as the for loop (one level of indentation inside the function).

Help me fix indentation in my classify_prompt function

✔️ Awesome, I've got everything!

Great. Double check you've saved classifier.py before moving on.

ⓧ I'd like to double check the full code

CATEGORY_KEYWORDS = {
    "code": [
        "function", "code", "program", "debug", "error", "syntax",
        "python", "javascript", "html", "css", "api", "class",
        "variable", "loop", "array", "algorithm", "implement",
        "refactor", "compile", "import", "def", "return"
    ],
    "creative": [
        "story", "poem", "write", "creative", "imagine", "fiction",
        "character", "narrative", "describe", "metaphor", "song",
        "dialogue", "scene", "fantasy", "adventure", "compose"
    ],
    "math": [
        "calculate", "math", "equation", "solve", "integral",
        "derivative", "probability", "statistics", "formula",
        "algebra", "geometry", "number", "sum", "multiply", "proof"
    ],
    "factual": [
        "what is", "explain", "how does", "define", "history",
        "science", "why", "difference between", "compare",
        "list", "facts", "tell me about", "describe how"
    ],
}


def classify_prompt(prompt):
    prompt_lower = prompt.lower()
    scores = {}

    for category, keywords in CATEGORY_KEYWORDS.items():
        score = 0
        for keyword in keywords:
            if keyword in prompt_lower:
                score += 1
        scores[category] = score

    total = sum(scores.values())
    if total == 0:
        return "factual", 0.0

    best_category = max(scores, key=scores.get)
    confidence = scores[best_category] / total

    return best_category, confidence
Test the classifier with sample prompts

Time to see if your classifier actually works. You'll feed it one prompt from each category and check that it returns the correct label.

  • In your terminal (which should still be in the autopili directory), run this test script:
python3 << 'EOF'
from classifier import classify_prompt

print("Code:", classify_prompt('Write a Python function to sort a list'))
print("Creative:", classify_prompt('Tell me a story about a dragon'))
print("Math:", classify_prompt('Calculate the integral of x squared'))
print("Factual:", classify_prompt('Explain how gravity works'))
EOF

What should I see?

You should see four lines of output, each showing a category and a confidence score as a tuple. For example:

Code: ('code', 0.5)
Creative: ('creative', 0.5)
Math: ('math', 1.0)
Factual: ('factual', 1.0)

Your exact confidence numbers might vary slightly depending on keyword overlaps. The important thing is that each prompt returns the correct category as the first element of the tuple.

Why do confidence scores differ?

A confidence of 1.0 means every matched keyword pointed to the same category. A confidence of 0.5 means the prompt's keywords were split between two categories.

For example, "Write a Python function" matches both "write" (creative) and "python" plus "function" (code). The code category still wins because it has more matches, but the confidence is lower because some keywords pointed elsewhere.

Seeing ModuleNotFoundError?

Make sure your terminal is inside the autopili directory where classifier.py lives. Run ls to confirm you can see classifier.py in the output.

If you're in the wrong directory, navigate to it:

cd ~/Desktop/autopili

Help me fix a ModuleNotFoundError when importing classifier

Your classifier is working. It can tell a coding question from a creative writing request in milliseconds. Next up, you'll build the Thompson Sampling engine that learns which model handles each category best.

Build the Thompson Sampling Bandit

Your classifier can now label prompts by category. The next question is: given a category, which model should AutoPili pick?

This is the multi-armed bandit problem. You have multiple options (models) and limited information about which is best for each task type. Thompson Sampling solves this by maintaining a belief about each model's performance and sampling from that belief to make decisions.

In this step, get ready to:
  • Build a BanditEngine class that tracks model performance using Beta distributions.
  • Implement Thompson Sampling to select the best model for a given category.
  • Implement learning through reward updates and utility methods.
Create the BanditEngine class with Beta priors

The core idea of Thompson Sampling is to represent your uncertainty about each model as a probability distribution. A Beta distribution is perfect for this because it lives between 0 and 1, just like a success probability.

  • Open bandit.py inside your autopili folder and add the imports and class initialization:
import numpy as np
import json
from pathlib import Path


class BanditEngine:
    def __init__(self, models, categories):
        # Store the list of models and categories this engine tracks
        self.models = list(models)
        self.categories = list(categories)
        # Alpha and beta_param form the Beta distribution for each (model, category) pair
        self.alpha = {}
        self.beta_param = {}

        # Initialize with uniform priors: Beta(1, 1) = no prior preference
        for model in self.models:
            for category in self.categories:
                key = (model, category)
                self.alpha[key] = 1.0
                self.beta_param[key] = 1.0

What does this code do?

Each (model, category) pair gets its own Beta distribution defined by two parameters: alpha (successes) and beta_param (failures).

Starting both at 1.0 creates a uniform prior. This means AutoPili starts with no opinion about which model is best. Think of it like a coin you have never flipped. You do not know if it is biased yet.

You won't see a change yet. The next chunk adds the selection logic that uses these distributions.

Implement the Thompson Sampling selection

The magic of Thompson Sampling is in how it picks a model. It draws a random sample from each model's Beta distribution and picks whichever model produced the highest sample.

  • Add the select_model() method below the __init__ method in bandit.py:
    def select_model(self, category):
        # Sample from each model's Beta distribution for this category
        samples = {}
        for model in self.models:
            key = (model, category)
            sample = np.random.beta(self.alpha[key], self.beta_param[key])
            samples[model] = sample
        # Pick the model with the highest sampled value
        best_model = max(samples, key=samples.get)
        return best_model, samples

What does this code do?

For each model, np.random.beta() draws a random number between 0 and 1 from that model's Beta distribution. A model with many past successes (high alpha) tends to produce higher samples. A model with less data (alpha=1, beta=1) produces samples spread evenly between 0 and 1.

This randomness is what drives exploration. A model with little data might randomly sample a high value and get selected. That is how Thompson Sampling naturally tries out unfamiliar options without needing an explicit exploration rule.

You won't see a change yet. The next chunk adds the learning mechanism that updates these distributions after each query.

Implement the update logic and utility methods

After AutoPili routes a prompt and gets a response, it needs to learn from the outcome. The update rule is simple: a good reward pushes the distribution toward higher values, and a bad reward pushes it lower.

  • Add the update() method below select_model() in bandit.py:
    def update(self, model, category, reward):
        # Update the Beta distribution based on reward quality
        key = (model, category)
        if reward > 0.5:
            # Good response: increment alpha (successes)
            self.alpha[key] += 1.0
        else:
            # Poor response: increment beta_param (failures)
            self.beta_param[key] += 1.0

What does this code do?

The reward is a number between 0 and 1. If the reward is above 0.5, the response was good, so we increment alpha (making the distribution skew higher for this model-category pair). If 0.5 or below, we increment beta_param (skewing it lower).

Over time, a model that consistently scores well for "code" prompts accumulates a high alpha for that category. Thompson Sampling then selects it more often. This is how AutoPili learns without a pre-existing dataset.

  • Now add the utility methods below update() to complete the class. These handle adding new models, inspecting learned preferences, and saving state:
    def add_model(self, model_name):
        # Register a new model with fresh uninformative priors
        if model_name in self.models:
            return
        self.models.append(model_name)
        for category in self.categories:
            key = (model_name, category)
            self.alpha[key] = 1.0
            self.beta_param[key] = 1.0

    def get_preferences(self):
        # Calculate which model has the highest mean reward per category
        preferences = {}
        for category in self.categories:
            best_model = None
            best_mean = -1
            for model in self.models:
                key = (model, category)
                mean = self.alpha[key] / (self.alpha[key] + self.beta_param[key])
                if mean > best_mean:
                    best_mean = mean
                    best_model = model
            preferences[category] = {
                "best_model": best_model,
                "confidence": round(best_mean, 3),
            }
        return preferences

    def save_state(self, filepath="bandit_state.json"):
        # Serialize the full bandit state to a JSON file
        state = {
            "models": self.models,
            "categories": self.categories,
            "alpha": {f"{k[0]}|{k[1]}": v for k, v in self.alpha.items()},
            "beta": {f"{k[0]}|{k[1]}": v for k, v in self.beta_param.items()},
        }
        Path(filepath).write_text(json.dumps(state, indent=2))

What do these methods do?

  • add_model() registers a brand new model with fresh Beta(1,1) priors across all categories. You will use this in the Secret Mission to hot-add a model at runtime.
  • get_preferences() calculates the mean (alpha / (alpha + beta)) for each model-category pair and returns the current best model per category.
  • save_state() writes the full bandit state to a JSON file so you can inspect how beliefs evolved after a run.
  • Save bandit.py.
  • Verify the class works by running this quick test in your terminal:
cd ~/Desktop/autopili && python3 -c "
from bandit import BanditEngine
engine = BanditEngine(['modelA', 'modelB'], ['code', 'math'])
model, samples = engine.select_model('code')
print(f'Selected: {model}')
print(f'Samples: {samples}')
engine.update('modelA', 'code', 0.9)
engine.update('modelA', 'code', 0.8)
prefs = engine.get_preferences()
print(f'Preferences: {prefs}')
"

You should see a selected model name, the sampled values for each model, and the preferences dictionary showing modelA preferred for code (since we gave it two good rewards).

Seeing an ImportError?

  • Make sure you are running the command from inside the autopili folder. The cd ~/Desktop/autopili prefix in the command handles this.
  • Check that numpy is installed. Run pip install numpy if you see a module-not-found error for numpy.
  • Confirm you saved bandit.py with all four methods (__init__, select_model, update, and the three utility methods).

Help me debug my BanditEngine import error.

✔️ Awesome, I've got everything!

Great. Double check you have saved bandit.py and your test printed a selected model with preferences.

ⓧ I'd like to double check the full code

Here is the complete bandit.py file:

import numpy as np
import json
from pathlib import Path


class BanditEngine:
    def __init__(self, models, categories):
        self.models = list(models)
        self.categories = list(categories)
        self.alpha = {}
        self.beta_param = {}

        for model in self.models:
            for category in self.categories:
                key = (model, category)
                self.alpha[key] = 1.0
                self.beta_param[key] = 1.0

    def select_model(self, category):
        samples = {}
        for model in self.models:
            key = (model, category)
            sample = np.random.beta(self.alpha[key], self.beta_param[key])
            samples[model] = sample
        best_model = max(samples, key=samples.get)
        return best_model, samples

    def update(self, model, category, reward):
        key = (model, category)
        if reward > 0.5:
            self.alpha[key] += 1.0
        else:
            self.beta_param[key] += 1.0

    def add_model(self, model_name):
        if model_name in self.models:
            return
        self.models.append(model_name)
        for category in self.categories:
            key = (model_name, category)
            self.alpha[key] = 1.0
            self.beta_param[key] = 1.0

    def get_preferences(self):
        preferences = {}
        for category in self.categories:
            best_model = None
            best_mean = -1
            for model in self.models:
                key = (model, category)
                mean = self.alpha[key] / (self.alpha[key] + self.beta_param[key])
                if mean > best_mean:
                    best_mean = mean
                    best_model = model
            preferences[category] = {
                "best_model": best_model,
                "confidence": round(best_mean, 3),
            }
        return preferences

    def save_state(self, filepath="bandit_state.json"):
        state = {
            "models": self.models,
            "categories": self.categories,
            "alpha": {f"{k[0]}|{k[1]}": v for k, v in self.alpha.items()},
            "beta": {f"{k[0]}|{k[1]}": v for k, v in self.beta_param.items()},
        }
        Path(filepath).write_text(json.dumps(state, indent=2))

Your bandit engine is complete. Next up, you will build the router that connects the classifier and bandit to Ollama, turning all three components into a working self-learning system.

Build the Router and Scoring System

You now have a classifier that categorizes prompts and a bandit engine that learns which model performs best per category. The missing piece is the router that ties everything together.

The router is the orchestration layer. It takes a user prompt, classifies it, asks the bandit which model to use, calls that model via Ollama, scores the response, and feeds the score back to the bandit so it can learn.

In this step, get ready to:
  • Build the AutoPiliRouter class that connects the classifier, bandit, and Ollama.
  • Implement a scoring function that turns latency and response length into a 0-1 reward signal.
  • Add JSON logging that records every routing decision for later analysis.
Set up the router class

The AutoPiliRouter class is the brain of AutoPili. Its constructor wires together an Ollama client, a BanditEngine, and a log to record decisions.

  • Open router.py in your autopili folder and add the following code:
import time
import json
from datetime import datetime
from pathlib import Path
from ollama import Client
from classifier import classify_prompt
from bandit import BanditEngine


# The three local models AutoPili routes between
MODELS = [
    "batiai/gemma4-e2b:q4",
    "llama3.2:3b",
    "qwen2.5-coder:1.5b",
]

# The prompt categories the classifier can return
CATEGORIES = ["code", "creative", "math", "factual"]


class AutoPiliRouter:
    def __init__(self):
        # Connect to the local Ollama server
        self.client = Client()
        # Initialize the bandit with our models and categories
        self.bandit = BanditEngine(MODELS, CATEGORIES)
        # Store every routing decision for analysis
        self.log = []

What does this code do?

The MODELS list holds the exact model names you pulled from Ollama in Step 1. The CATEGORIES list matches the four categories your classifier returns.

Inside __init__(), Client() connects to your local Ollama server (no arguments needed since it defaults to localhost:11434). The BanditEngine starts with uniform priors for every model-category pair, meaning it has no preferences yet.

  • Save router.py.

You won't see any output yet. The next chunk adds the scoring logic that the routing method depends on.

Implement the scoring function

AutoPili needs a way to judge how well a model responded. Without a separate judge model, you can use two proxy signals: how fast the model responded (latency) and whether it generated a meaningful-length answer.

  • Add the score_response method inside the AutoPiliRouter class, right below __init__:
    def score_response(self, response_text, latency_seconds):
        # Faster responses score higher (30s = 0, instant = 1)
        reward_latency = max(0.0, 1.0 - (latency_seconds / 30.0))
        # Longer responses score higher up to 200 chars
        response_length = len(response_text)
        reward_length = min(1.0, response_length / 200.0)
        # Weighted combination: latency matters more
        reward = 0.6 * reward_latency + 0.4 * reward_length
        return round(reward, 4)

How does the scoring formula work?

The reward is a number between 0 and 1. Two components make it up:

  • reward_latency: a response that takes 0 seconds scores 1.0. A response that takes 30 seconds scores 0.0. Anything in between scales linearly.
  • reward_length: a response of 200+ characters scores 1.0. Shorter responses score proportionally less. This penalizes empty or one-word answers.
  • The final reward weights latency at 60% and length at 40%, because speed matters more for a local router.
  • Save router.py.

You won't see a change yet. The next chunk uses this scoring function inside the routing logic.

Add the routing logic and JSON logging

The route() method is the full pipeline: classify the prompt, pick a model, call it, score the result, update the bandit, and log everything.

  • Add the route method below score_response in the AutoPiliRouter class:
    def route(self, prompt):
        # Step 1: Classify the prompt into a category
        category, confidence = classify_prompt(prompt)
        # Step 2: Ask the bandit which model to use
        selected_model, samples = self.bandit.select_model(category)

        # Step 3: Call the selected model via Ollama
        start_time = time.time()
        response = self.client.chat(
            model=selected_model,
            messages=[{"role": "user", "content": prompt}],
        )
        latency = time.time() - start_time

        # Step 4: Score the response and update the bandit
        response_text = response.message.content
        reward = self.score_response(response_text, latency)
        self.bandit.update(selected_model, category, reward)

        # Step 5: Log this routing decision
        entry = {
            "timestamp": datetime.now().isoformat(),
            "prompt": prompt[:80],
            "category": category,
            "confidence": round(confidence, 3),
            "selected_model": selected_model,
            "latency_seconds": round(latency, 2),
            "reward": reward,
            "response_length": len(response_text),
        }
        self.log.append(entry)

        return {
            "response": response_text,
            "model": selected_model,
            "category": category,
            "latency": latency,
            "reward": reward,
        }

What does route() do step by step?

  • classify_prompt() returns the category and a confidence score from your keyword classifier.
  • select_model() samples from the Beta distributions and picks the model with the highest sample for that category. This is Thompson Sampling in action.
  • client.chat() sends the prompt to the selected model via Ollama and waits for the full response.
  • The reward feeds back into self.bandit.update(), shifting the Beta distribution toward or away from this model for this category.
  • Every decision gets logged as a dictionary with all the metadata you need for analysis.
  • Save router.py.

You won't see output yet. The final chunk adds methods to persist and display the log.

  • Add the save_log and print_summary methods below route to complete the class:
    def save_log(self, filepath="autopili_log.json"):
        # Write all routing decisions to a JSON file
        Path(filepath).write_text(json.dumps(self.log, indent=2))

    def print_summary(self):
        print("\n" + "=" * 60)
        print("AUTOPILI LEARNING SUMMARY")
        print("=" * 60)

        # Show which model the bandit prefers for each category
        preferences = self.bandit.get_preferences()
        print("\nLearned Model Preferences:")
        for category, info in preferences.items():
            print(f"  {category:10s} -> {info['best_model']} (confidence: {info['confidence']})")

        # Show aggregate routing statistics
        print(f"\nTotal queries routed: {len(self.log)}")
        if self.log:
            avg_reward = sum(e["reward"] for e in self.log) / len(self.log)
            avg_latency = sum(e["latency_seconds"] for e in self.log) / len(self.log)
            print(f"Average reward: {avg_reward:.3f}")
            print(f"Average latency: {avg_latency:.2f}s")

What do these methods do?

  • save_log() writes every routing decision to autopili_log.json. This is your self-generated performance dataset that grows with every query.
  • print_summary() displays which model the bandit currently prefers for each category. The confidence value is the mean of the Beta distribution (alpha / (alpha + beta)).
  • Save router.py.
  • Verify the file has no syntax errors by running this command from inside the autopili folder:
python -c "from router import AutoPiliRouter; print('Router ready!')"

You should see Router ready! printed with no errors.

Seeing an import error?

  • If you see a message about No module named 'ollama', make sure you installed the package with pip install ollama==0.6.2 in Step 1.
  • If you see a message about No module named 'classifier', make sure you're running the command from inside the autopili folder where classifier.py lives.
  • If you see a SyntaxError, double check the indentation. Every method should be indented with 4 spaces inside the class.

Help me fix my import error in router.py

✔️ Awesome, I've got everything!

Great. Make sure you've saved router.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete router.py file:

import time
import json
from datetime import datetime
from pathlib import Path
from ollama import Client
from classifier import classify_prompt
from bandit import BanditEngine


MODELS = [
    "batiai/gemma4-e2b:q4",
    "llama3.2:3b",
    "qwen2.5-coder:1.5b",
]

CATEGORIES = ["code", "creative", "math", "factual"]


class AutoPiliRouter:
    def __init__(self):
        self.client = Client()
        self.bandit = BanditEngine(MODELS, CATEGORIES)
        self.log = []

    def score_response(self, response_text, latency_seconds):
        reward_latency = max(0.0, 1.0 - (latency_seconds / 30.0))
        response_length = len(response_text)
        reward_length = min(1.0, response_length / 200.0)
        reward = 0.6 * reward_latency + 0.4 * reward_length
        return round(reward, 4)

    def route(self, prompt):
        category, confidence = classify_prompt(prompt)
        selected_model, samples = self.bandit.select_model(category)

        start_time = time.time()
        response = self.client.chat(
            model=selected_model,
            messages=[{"role": "user", "content": prompt}],
        )
        latency = time.time() - start_time

        response_text = response.message.content
        reward = self.score_response(response_text, latency)
        self.bandit.update(selected_model, category, reward)

        entry = {
            "timestamp": datetime.now().isoformat(),
            "prompt": prompt[:80],
            "category": category,
            "confidence": round(confidence, 3),
            "selected_model": selected_model,
            "latency_seconds": round(latency, 2),
            "reward": reward,
            "response_length": len(response_text),
        }
        self.log.append(entry)

        return {
            "response": response_text,
            "model": selected_model,
            "category": category,
            "latency": latency,
            "reward": reward,
        }

    def save_log(self, filepath="autopili_log.json"):
        Path(filepath).write_text(json.dumps(self.log, indent=2))

    def print_summary(self):
        print("\n" + "=" * 60)
        print("AUTOPILI LEARNING SUMMARY")
        print("=" * 60)

        preferences = self.bandit.get_preferences()
        print("\nLearned Model Preferences:")
        for category, info in preferences.items():
            print(f"  {category:10s} -> {info['best_model']} (confidence: {info['confidence']})")

        print(f"\nTotal queries routed: {len(self.log)}")
        if self.log:
            avg_reward = sum(e["reward"] for e in self.log) / len(self.log)
            avg_latency = sum(e["latency_seconds"] for e in self.log) / len(self.log)
            print(f"Average reward: {avg_reward:.3f}")
            print(f"Average latency: {avg_latency:.2f}s")

Your router is fully wired up. Next up, you'll create a test suite of 20 prompts and watch AutoPili learn in real-time which model handles each category best.

Run the Learning Loop

Your router can now classify prompts, pick a model using Thompson Sampling, call Ollama, and feed the reward back into the bandit engine.

Now it's time to see the full system learn. In this step, you'll create a test suite of 20 prompts across all four categories and run them through AutoPili. After each query, you'll see exactly which model was selected, how fast it responded, and how the reward signal shaped the bandit's future decisions.

In this step, get ready to:
  • Define a test suite of 20 prompts spanning code, creative, math, and factual categories.
  • Build a main function that routes each prompt and prints a live progress table.
  • Run the full learning loop and observe AutoPili's learned model preferences.
Define the test prompt suite

For Thompson Sampling to learn meaningful preferences, it needs a diverse set of prompts that exercise all four categories equally. Five prompts per category gives the bandit enough signal to start differentiating model performance.

  • In main.py, replace all existing content with the following import and test prompt list:
from router import AutoPiliRouter


# 20 test prompts: 5 per category (code, creative, math, factual)
TEST_PROMPTS = [
    "Write a Python function to reverse a linked list",
    "Debug this code: for i in range(10) print(i)",
    "Implement a binary search algorithm in Python",
    "What is the difference between a list and a tuple in Python?",
    "Write a class for a stack data structure",
    "Write a short poem about the ocean at sunset",
    "Create a story about a robot learning to paint",
    "Imagine a dialogue between the sun and the moon",
    "Compose a haiku about programming",
    "Describe a fantasy world where gravity works backwards",
    "Calculate the derivative of x^3 + 2x^2 - 5x + 3",
    "What is the probability of rolling two sixes in a row?",
    "Solve: 3x + 7 = 22",
    "Explain the formula for compound interest",
    "What is the integral of sin(x) dx?",
    "What is photosynthesis and how does it work?",
    "Explain the difference between TCP and UDP",
    "How does a refrigerator work?",
    "What caused World War I?",
    "Describe how the immune system fights viruses",
]

What does this code do?

The import pulls in your AutoPiliRouter class, which already has the classifier, bandit, and Ollama client wired together.

The TEST_PROMPTS list is balanced across categories. Prompts 1-5 target code, 6-10 target creative, 11-15 target math, and 16-20 target factual. This balance ensures the bandit gets equal opportunity to learn about each model across every category.

  • Save main.py.
  • You won't see a change yet. The next chunk adds the function that actually runs these prompts.
Build the learning loop and summary

The main function ties everything together. It creates a router, iterates through every test prompt, prints a live table row for each query, then outputs the bandit's learned preferences at the end.

  • Add the following main() function below your TEST_PROMPTS list in main.py:
def main():
    router = AutoPiliRouter()

    # Print header with system info
    print("=" * 60)
    print("AUTOPILI - Self-Learning LLM Router")
    print("=" * 60)
    print(f"Models: {len(router.bandit.models)}")
    print(f"Categories: {router.bandit.categories}")
    print(f"Test prompts: {len(TEST_PROMPTS)}")
    print("=" * 60)

    # Route each prompt and print a live progress row
    for i, prompt in enumerate(TEST_PROMPTS, 1):
        result = router.route(prompt)
        print(
            f"[{i:2d}/{len(TEST_PROMPTS)}] "
            f"{result['category']:8s} | "
            f"{result['model']:25s} | "
            f"{result['latency']:.1f}s | "
            f"reward: {result['reward']:.3f} | "
            f"{prompt[:40]}..."
        )

    # Print learned preferences and save data to disk
    router.print_summary()
    router.save_log()
    router.bandit.save_state()
    print("\nLog saved to autopili_log.json")
    print("Bandit state saved to bandit_state.json")


if __name__ == "__main__":
    main()

What does this code do?

  • router.route(prompt) classifies the prompt, samples from the bandit, calls Ollama, scores the response, and updates the Beta distributions. All in one call.
  • The formatted print statement creates a live table. Each row shows the query number, detected category, which model was selected, how long the response took, and the reward score.
  • router.print_summary() outputs which model the bandit now considers best for each category.
  • router.save_log() writes every routing decision to autopili_log.json. router.bandit.save_state() writes the learned Beta distribution parameters to bandit_state.json.
  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before running it.

ⓧ I'd like to double check the full code

from router import AutoPiliRouter


TEST_PROMPTS = [
    "Write a Python function to reverse a linked list",
    "Debug this code: for i in range(10) print(i)",
    "Implement a binary search algorithm in Python",
    "What is the difference between a list and a tuple in Python?",
    "Write a class for a stack data structure",
    "Write a short poem about the ocean at sunset",
    "Create a story about a robot learning to paint",
    "Imagine a dialogue between the sun and the moon",
    "Compose a haiku about programming",
    "Describe a fantasy world where gravity works backwards",
    "Calculate the derivative of x^3 + 2x^2 - 5x + 3",
    "What is the probability of rolling two sixes in a row?",
    "Solve: 3x + 7 = 22",
    "Explain the formula for compound interest",
    "What is the integral of sin(x) dx?",
    "What is photosynthesis and how does it work?",
    "Explain the difference between TCP and UDP",
    "How does a refrigerator work?",
    "What caused World War I?",
    "Describe how the immune system fights viruses",
]


def main():
    router = AutoPiliRouter()

    print("=" * 60)
    print("AUTOPILI - Self-Learning LLM Router")
    print("=" * 60)
    print(f"Models: {len(router.bandit.models)}")
    print(f"Categories: {router.bandit.categories}")
    print(f"Test prompts: {len(TEST_PROMPTS)}")
    print("=" * 60)

    for i, prompt in enumerate(TEST_PROMPTS, 1):
        result = router.route(prompt)
        print(
            f"[{i:2d}/{len(TEST_PROMPTS)}] "
            f"{result['category']:8s} | "
            f"{result['model']:25s} | "
            f"{result['latency']:.1f}s | "
            f"reward: {result['reward']:.3f} | "
            f"{prompt[:40]}..."
        )

    router.print_summary()
    router.save_log()
    router.bandit.save_state()
    print("\nLog saved to autopili_log.json")
    print("Bandit state saved to bandit_state.json")


if __name__ == "__main__":
    main()
Run AutoPili and watch it learn

This is the moment where Thompson Sampling goes from theory to observable behavior. Early queries will show the bandit exploring different models. Later queries will show it exploiting what it has learned.

  • Make sure Ollama is running on your machine. If it is not, open it from your Applications folder or menu bar.
  • In your terminal, make sure you are inside the autopili folder, then run the script:
python main.py

This will take a few minutes. Each prompt requires a full model response from Ollama, and the first time each model is loaded takes extra time.

What should I see?

A live table prints one row per query. Each row shows the category, which model was selected, the response latency, and the reward score. You will notice the bandit sometimes picks different models for the same category early on. This is exploration.

After all 20 queries, a summary section prints the learned model preferences. Each category shows which model the bandit considers best and its confidence level. Two JSON files are saved to disk containing the full decision log and the bandit's learned state.

Seeing a connection error?

If you see Failed to connect to Ollama, the Ollama server is not running. Open Ollama from your Applications folder or check your menu bar to start it.

If a model takes a very long time on the first call, that is normal. Ollama loads models into memory on first use and keeps them cached for subsequent calls.

help me debug a connection or timeout error when running AutoPili

  • Wait for the full run to complete. You should see all 20 rows printed followed by the AUTOPILI LEARNING SUMMARY section.
  • Confirm that autopili_log.json and bandit_state.json now exist in your autopili folder by running:
ls

You should see autopili_log.json and bandit_state.json alongside your four Python files.

Why do results vary between runs?

Thompson Sampling is stochastic. Each run samples different values from the Beta distributions, so the bandit might explore different models in different orders. The learned preferences will converge over more queries, but a single 20-query run shows the algorithm's exploration tendency clearly.

Secret mission

Hot-Add a New Model

AutoPili already knows which of your three models handles each category best. But what happens when you introduce a brand-new model at runtime, with zero history? In this mission, you'll pull a fourth model, register it with fresh uninformative priors, and watch Thompson Sampling's natural exploration discover its strengths from scratch.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Local project folder: autopili/ on your Desktop (contains classifier.py, bandit.py, router.py, main.py, autopili_log.json, and bandit_state.json).
  • Ollama models: batiai/gemma4-e2b:q4, llama3.2:3b, qwen2.5-coder:1.5b, and phi4-mini (~7.4 GB total on disk).
  • Python packages: ollama and numpy.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with AutoPili or want to try routing different prompts through your models.

  • Your Ollama models stay downloaded and ready to use for other projects.
  • Your autopili/ folder and generated log files remain on your Desktop.
  • The ollama and numpy Python packages remain installed.

Pause - I'll come back to this later

Free up memory by unloading any running Ollama models. Your files and downloaded model weights stay on disk so you can pick up where you left off.

  • Unload the models from memory by running:
ollama stop batiai/gemma4-e2b:q4
ollama stop llama3.2:3b
ollama stop qwen2.5-coder:1.5b
ollama stop phi4-mini
  • Your project files and model weights stay on disk, ready to resume anytime.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up approximately 7.4 GB of disk space from the Ollama models plus your project files.

  • Remove the four Ollama models by running:
ollama rm batiai/gemma4-e2b:q4
ollama rm llama3.2:3b
ollama rm qwen2.5-coder:1.5b
ollama rm phi4-mini
  • Delete the project folder and all its files by running:
cd ~/Desktop
rm -rf autopili
  • Uninstall the Python packages by running:
pip uninstall ollama numpy -y

That's a Wrap!

That's a Wrap!

Nice work! You just built AutoPili, a self-learning LLM router that classifies prompts, picks the best local model using Thompson Sampling, and gets smarter with every query it handles.

You've learned how to:

  • Implemented Thompson Sampling from scratch using Beta distributions to solve the exploration-exploitation tradeoff automatically.
  • Built a self-learning router that generates its own training data and improves with every query it routes.
  • Created a complete online learning pipeline connecting prompt classification, model selection, response scoring, and JSON performance logging.
  • Secret Mission: Hot-added a new model at runtime and observed Thompson Sampling's natural exploration discovering its strengths from uninformative priors.

Ready to quiz yourself?