Fine-Tune an AI Chatbot with LoRA

Fine-tune a language model into a Python tutor using LoRA and Hugging Face.

Introduction

30 Second Summary

AI chatbots like ChatGPT sound smart because they were trained on specific conversations, not just raw internet text. That training process is called fine-tuning, and it is the most in-demand skill in AI engineering today.

In this project, you will fine-tune your own AI chatbot using industry-standard tools on a free GPU. You will take a small open-source Large Language Model and train it to become a concise Python tutor that responds differently from its original generic self.

What You'll Build

You will ask your fine-tuned chatbot a Python question and watch it respond with focused, step-by-step teaching that the base model never produced on its own.

By the end of this project, you'll have:

  • A before vs. after comparison showing how your model's responses transform from generic to focused after training.
  • A complete LoRA fine-tuning pipeline in a reusable notebook you can adapt for any model or dataset.
  • A trained adapter published on your Hugging Face profile that anyone can download and use.
  • Secret Mission: Teach your model to prefer good responses over bad ones using DPO (Direct Preference Optimization).

Are there any prerequisites?

You need a Google account (for Google Colab) and a Hugging Face account. No local setup, installations, or paid subscriptions required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your AI Lab in Google Colab

Fine-tuning a language model requires serious computing power. A GPU accelerates the math that neural networks rely on, turning hours of training into minutes.

Google Colab gives you a free NVIDIA Tesla T4 GPU right in your browser. In this step, you will create a Colab notebook, install the fine-tuning libraries, and connect to Hugging Face Hub so you can push your trained model later.

In this step, get ready to:
  • Enable the T4 GPU runtime in Google Colab.
  • Install the fine-tuning libraries (transformers, trl, peft, bitsandbytes, datasets).
  • Log in to Hugging Face Hub for model uploads.
Create a new Colab notebook and enable the T4 GPU

Google Colab is a free cloud notebook environment that gives you access to GPUs without installing anything on your machine.

  • Open colab.research.google.com in your browser.
  • Click New notebook to create a fresh notebook.
  • Click the notebook title in the top left (it says Untitled0.ipynb) and rename it to fine-tune-python-tutor.

Now enable the GPU so your notebook has access to the T4 hardware.

  • Click Runtime in the top menu bar.
  • Click Change runtime type.
  • Under Hardware accelerator, select T4 GPU.
  • Click Save.

Before you run the next cell, what do you think the GPU check will show if the runtime connected successfully?

  • Verify the GPU is available by running this command in your first code cell:
!nvidia-smi

You should see Tesla T4 listed with approximately 15 GB of memory available.

Not seeing a GPU?

If you see an error or no GPU listed, confirm you selected T4 GPU (not CPU) in Runtime > Change runtime type. After changing the runtime type, Colab restarts the session. Re-run the cell after the restart completes.

If Colab says GPUs are unavailable, wait a few minutes and try again. The free tier has limited GPU availability during peak hours.

help me troubleshoot why my Colab notebook is not showing a T4 GPU after changing the runtime type

Install the fine-tuning libraries

You need five libraries to build the fine-tuning pipeline. Each plays a specific role in loading, training, and optimizing the model.

  • Install all five libraries by running this in a new cell:
# Install the core fine-tuning stack
!pip install transformers trl peft bitsandbytes datasets

What do these libraries do?

  • transformers loads and runs pre-trained language models from Hugging Face Hub.
  • trl provides the SFTTrainer for supervised fine-tuning.
  • peft enables LoRA adapters so you can train a fraction of the model's parameters.
  • bitsandbytes handles 4-bit quantization to shrink the model and fit it in GPU memory.
  • datasets structures your training data into the format the trainer expects.

The install takes about one to two minutes. Once it finishes, you will see a long output ending with successfully installed packages.

  • Confirm the install worked by running this in a new cell:
import transformers
print(transformers.__version__)

You should see a version number printed with no errors.

Seeing an ImportError?

If the import fails, your install cell may not have completed successfully. Scroll up to the pip install cell output and look for red error text. A common fix is to restart the runtime (Runtime > Restart session) and re-run both cells.

help me fix an ImportError when trying to import transformers in Google Colab after running pip install

Log in to Hugging Face Hub

At the end of this project, you will push your trained model to Hugging Face Hub. To do that, your notebook needs write access to your account.

  • Open huggingface.co/settings/tokens in a new browser tab.
  • Click Create new token.
  • Select the Write preset to grant push permissions.
  • Give the token a name (like colab-fine-tune) and click Create token.
  • Copy the token value. You will paste it into Colab in a moment.

Why a write token?

A read token only lets you download models. A write token lets you upload your fine-tuned adapter to your Hugging Face profile so anyone can use it.

The token is stored in your Colab session memory only and disappears when the session ends.

  • Back in your Colab notebook, authenticate by running this in a new cell:
from huggingface_hub import login
login()

What happens when you run this?

Colab will display a text input prompting you to paste your token. Paste the write token you just created and press Enter. You will see a confirmation message once authentication succeeds.

  • When prompted, paste your write token and press Enter.

You should see a message confirming you are logged in to Hugging Face.

Login not working?

Make sure you copied the full token string (it starts with hf_). If you accidentally created a read-only token, go back to huggingface.co/settings/tokens and create a new one with the Write preset.

help me troubleshoot my Hugging Face login in Google Colab

✔️ Awesome, I've got everything!

Great. Make sure all four cells are saved in your notebook and that the login succeeded.

ⓧ I'd like to double check the full code

Your notebook should have these four cells from top to bottom:

Cell 1: Install libraries.

# Install the core fine-tuning stack
!pip install transformers trl peft bitsandbytes datasets

Cell 2: Verify GPU.

!nvidia-smi

Cell 3: Verify transformers import.

import transformers
print(transformers.__version__)

Cell 4: Hugging Face login.

from huggingface_hub import login
login()

Your AI lab is ready. You have a GPU-powered notebook with all the fine-tuning tools installed and a live connection to Hugging Face Hub. Next up, you will load the base model and see exactly how it responds before any fine-tuning.

Meet the Base Model

Your AI lab is set up with a T4 GPU, fine-tuning libraries, and Hugging Face Hub access. Before you fine-tune anything, you need a concrete "before" picture of how the model already behaves.

In this step, you'll load SmolLM2-1.7B-Instruct and ask it a Python tutoring question. This gives you a baseline to measure exactly what fine-tuning changes later.

In this step, get ready to:
  • Load the model in 4-bit quantization so it fits on the free GPU.
  • Ask the model a Python tutoring question and observe its response.
  • Inspect tokenization to see what the model actually processes.
Load SmolLM2-1.7B-Instruct in 4-bit quantization

SmolLM2-1.7B-Instruct has 1.7 billion parameters. At full precision, it would take about 3.4 GB of GPU memory. Quantization compresses the model's weights from 16-bit to 4-bit, shrinking it to roughly 1 GB so it fits comfortably on your T4's 15 GB of VRAM.

  • In your Colab notebook, create a new code cell and paste the following code to configure 4-bit quantization and load the model:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

# Configure 4-bit quantization to shrink the model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

# Load the model and tokenizer from Hugging Face Hub
model_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

print(f"Model loaded: {model_id}")

What does this code do?

BitsAndBytesConfig tells the transformers library how to compress the model. load_in_4bit=True enables 4-bit quantization, and nf4 is the quantization format optimized for neural network weights.

The bnb_4bit_compute_dtype=torch.float16 setting tells the GPU to use 16-bit precision for calculations. The T4 GPU does not support bfloat16 in hardware, so float16 is the correct choice here.

Finally, device_map="auto" automatically places the model on the available GPU.

  • Run the cell.
  • You should see download progress bars followed by the message Model loaded: HuggingFaceTB/SmolLM2-1.7B-Instruct.

Seeing an out-of-memory error?

Make sure your runtime is set to T4 GPU. Go to Runtime > Change runtime type and confirm T4 GPU is selected. If it was already set, try Runtime > Restart session to clear any leftover memory.

Help me debug an out-of-memory error when loading SmolLM2 in 4-bit on a T4 GPU in Google Colab.

Ask the model a Python tutoring question

The model is loaded and ready. Now you'll ask it a Python tutoring question and see how it responds. Before you run this, think about what you expect: will a general-purpose 1.7B parameter model give a focused, concise tutoring answer?

  • Create a new code cell and paste the following code to ask the model a question using its chat template:
# Define your question for the model
question = "Explain what a Python list comprehension is, step by step for a beginner"

# Format the question using the model's chat template
messages = [{"role": "user", "content": question}]
input_text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

# Tokenize and generate a response
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300)

# Decode and print the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Base model response:")
print(response)

What does this code do?

The apply_chat_template method formats your question into the special token structure the model expects. SmolLM2 uses the ChatML format with <|im_start|> and <|im_end|> tokens to mark where each message begins and ends.

Setting add_generation_prompt=True appends a <|im_start|>assistant token at the end. This signals to the model that it should start writing an assistant response.

  • Run the cell and read the response carefully.

You'll see that the model produces a response, but it's generic, verbose, or wandering. It might include correct information, but it is not the focused, step-by-step tutoring style you want for a beginner.

This is exactly why fine-tuning exists. The model has general language ability, but it has not been trained on your specific tutoring style.

Inspect tokenization to see what the model actually processes

You read your question as English words. The model reads it as a sequence of numbers called token IDs. Tokenization is the process that converts text into these numbers. Understanding tokens helps you see why training data format matters so much.

  • Create a new code cell and paste the following code to see tokenization in action:
# Tokenize the question string directly
tokens = tokenizer(question, return_tensors="pt")
token_ids = tokens["input_ids"][0].tolist()

# Print token IDs alongside their decoded text pieces
print(f"Total tokens: {len(token_ids)}\n")
for i, token_id in enumerate(token_ids):
    decoded_piece = tokenizer.decode([token_id])
    print(f"Token {i}: ID={token_id:>6} | Text: '{decoded_piece}'")

What does this code do?

The tokenizer splits your question into subword pieces and maps each piece to an integer ID. Some tokens are whole words, and some are fragments of words. This is how every Large Language Model processes text: numbers in, numbers out.

When you fine-tune the model later, your training examples go through this exact same process. The model learns patterns over these token sequences, not over raw text.

  • Run the cell.
  • You should see a list of integers paired with their text pieces, showing how your question breaks down into tokens.

Seeing a NameError for question?

This means your cells ran out of order. The question variable was defined in the previous cell. Go back and run that cell first, then re-run this one.

Help me fix a NameError in Google Colab when running cells that depend on variables from earlier cells.

✔️ Awesome, I've got everything!

Great. Double check you have three new cells in your notebook: the model loading cell, the chat generation cell, and the tokenization cell.

ⓧ I'd like to double check the full code

Scroll up and confirm you have these three cells, in order:

  • Cell 5: Imports, BitsAndBytesConfig, AutoTokenizer, and AutoModelForCausalLM loading HuggingFaceTB/SmolLM2-1.7B-Instruct.
  • Cell 6: Chat template applied to the question, model.generate() called, response decoded and printed.
  • Cell 7: Tokenizer applied to the question string, token IDs printed alongside decoded text pieces.

Now you've seen the gap. The base model can answer, but not in the focused, step-by-step tutoring style you want. In the next step, you'll build the training dataset that shows the model what great tutoring looks like.

Build Your Training Dataset

In the previous step, you asked the base model a Python tutoring question and saw a generic, unfocused response. The model is not broken. It simply has not seen examples of what concise, step-by-step tutoring looks like.

Fine-tuning fixes this by showing the model exactly the style you want. In this step, you will build the training dataset that teaches the model how a concise Python tutor responds.

In this step, get ready to:
  • Create 15 conversational training examples in the chat template format.
  • Convert your examples into a Hugging Face Dataset object.
  • Verify the dataset structure matches what the trainer expects.
Create conversational training examples in the correct format

The SFTTrainer you will use in the next step expects a very specific data format. Each training example must be a dictionary with a messages key containing a list of role/content pairs.

This is the same conversational format used by ChatGPT, Claude, and every modern chat model. The trainer reads this format and automatically applies the model's chat template to convert it into the token sequence the model trains on.

  • Create a new code cell in your notebook and paste the first set of training examples:
# Python tutoring conversations for fine-tuning
training_data = [
    {"messages": [
        {"role": "user", "content": "What is a Python list comprehension?"},
        {"role": "assistant", "content": "A list comprehension creates a new list by applying an expression to each item in an iterable. The syntax is [expression for item in iterable]. For example, [x * 2 for x in range(5)] gives [0, 2, 4, 6, 8]. Add a condition to filter: [x for x in range(10) if x % 2 == 0] keeps only even numbers."}]},
    {"messages": [
        {"role": "user", "content": "How do I define a function in Python?"},
        {"role": "assistant", "content": "Use the def keyword, then the function name and parentheses. Parameters go inside the parentheses. The body is indented on the next line. Example: def greet(name): return f'Hello, {name}'. Call it with greet('Alice') to get 'Hello, Alice'."}]},
    {"messages": [
        {"role": "user", "content": "What is the difference between a for loop and a while loop?"},
        {"role": "assistant", "content": "A for loop iterates over a known sequence like a list or range. A while loop repeats as long as a condition stays True. Use for when you know the number of iterations: for i in range(5). Use while when you loop until something changes: while user_input != 'quit'."}]},
    {"messages": [
        {"role": "user", "content": "How do Python dictionaries work?"},
        {"role": "assistant", "content": "A dictionary stores key-value pairs in curly braces. Create one: scores = {'alice': 95, 'bob': 87}. Access values by key: scores['alice'] returns 95. Add entries with scores['carol'] = 91. Keys must be unique and immutable like strings or numbers."}]},
    {"messages": [
        {"role": "user", "content": "What does the strip() method do?"},
        {"role": "assistant", "content": "strip() removes whitespace from both ends of a string. Example: '  hello  '.strip() gives 'hello'. Use lstrip() for leading whitespace only, rstrip() for trailing only. Pass characters to strip specific ones: '##hello##'.strip('#') gives 'hello'."}]},
    {"messages": [
        {"role": "user", "content": "How does if/elif/else work?"},
        {"role": "assistant", "content": "Start with if and a condition followed by a colon. Add elif for additional conditions. End with else for everything remaining. Python checks top to bottom and runs the first match only. Example: if score >= 90: grade = 'A' elif score >= 80: grade = 'B' else: grade = 'C'."}]},
    {"messages": [
        {"role": "user", "content": "What is a Python class?"},
        {"role": "assistant", "content": "A class is a blueprint for creating objects. Define with class Dog:. Add __init__ to set attributes: def __init__(self, name): self.name = name. Create an instance: my_dog = Dog('Rex'). Access attributes with dot notation: my_dog.name returns 'Rex'."}]},
    {"messages": [
        {"role": "user", "content": "What are lambda functions?"},
        {"role": "assistant", "content": "A lambda is a small anonymous function on one line. Syntax: lambda arguments: expression. Example: double = lambda x: x * 2, then double(5) returns 10. Useful as quick functions for sorted() or map(). Use a regular def for anything beyond a single expression."}]},
]

What does this format do?

Each dictionary has a single messages key containing a conversation. The role field marks who is speaking (user or assistant).

The content field is the actual text. Notice how every assistant response is short and step-by-step. This is the tutoring style you are teaching the model to produce.

  • Still in the same cell, continue the list with more training examples below the last entry:
    {"messages": [
        {"role": "user", "content": "How does try/except work for error handling?"},
        {"role": "assistant", "content": "Wrap risky code in a try block. If an error occurs, Python jumps to except instead of crashing. Example: try: result = 10 / 0 except ZeroDivisionError: result = 0. Catch specific errors or use bare except for any error. Add finally: for code that runs regardless."}]},
    {"messages": [
        {"role": "user", "content": "How does list slicing work?"},
        {"role": "assistant", "content": "Slicing extracts a portion with list[start:stop:step]. Start is included, stop is excluded. Example: nums = [0,1,2,3,4], nums[1:4] gives [1,2,3]. Omit start to begin at 0, omit stop to go to the end. Negative step reverses: nums[::-1] gives [4,3,2,1,0]."}]},
    {"messages": [
        {"role": "user", "content": "What are f-strings?"},
        {"role": "assistant", "content": "F-strings embed expressions inside string literals. Put f before the quote and use curly braces: name = 'Alice'; f'Hello, {name}' gives 'Hello, Alice'. Any expression works inside braces: f'2+2={2+2}' gives '2+2=4'. Faster and more readable than .format() or concatenation."}]},
    {"messages": [
        {"role": "user", "content": "What is the difference between a tuple and a list?"},
        {"role": "assistant", "content": "Lists use square brackets and are mutable: you can change items. Tuples use parentheses and are immutable: once created, they cannot change. Use lists for data that changes: shopping = ['milk', 'eggs']. Use tuples for fixed data: point = (3, 7). Tuples can be dictionary keys."}]},
    {"messages": [
        {"role": "user", "content": "What are *args and **kwargs?"},
        {"role": "assistant", "content": "*args lets a function accept any number of positional arguments as a tuple. **kwargs accepts any number of keyword arguments as a dictionary. Example: def show(*args, **kwargs): print(args, kwargs). Calling show(1, 2, name='Alice') prints (1, 2) {'name': 'Alice'}."}]},
    {"messages": [
        {"role": "user", "content": "How do I read a file in Python?"},
        {"role": "assistant", "content": "Use open() with a with statement for safe handling. with open('data.txt', 'r') as f: content = f.read() reads the whole file as a string. Use f.readlines() for a list of lines. The with block automatically closes the file when done, even if an error occurs."}]},
    {"messages": [
        {"role": "user", "content": "What is a Python decorator?"},
        {"role": "assistant", "content": "A decorator wraps a function to extend its behavior without changing its code. Apply with @ above a function definition: @timer above def slow_func(): adds timing automatically. Decorators take a function as input and return a modified version. Common built-ins include @property and @staticmethod."}]},
]

Why only 15 examples?

For a 1.7B parameter model with LoRA adapters, even a small dataset produces measurable style changes. In production, you would use hundreds or thousands of examples. For learning the pipeline, 15 is enough to see the difference while keeping training under 5 minutes on the free T4 GPU.

  • Run the cell to create your training data.
  • Verify the list was created by running this in a new cell:
# Quick check: how many examples did we create?
print(f"Training examples: {len(training_data)}")
print(f"First example roles: {[m['role'] for m in training_data[0]['messages']]}")
print(f"First question: {training_data[0]['messages'][0]['content'][:50]}...")

You should see 15 training examples, with roles showing ['user', 'assistant'] and the first question about list comprehensions.

Seeing a syntax error?

The most common mistake is a missing comma between examples or a mismatched bracket. Scroll through your cell and check that every }], closing has a trailing comma except the very last example before the final ].

help me fix a syntax error in my Python training data list

Convert to a Hugging Face Dataset and verify the structure

Your training data is currently a plain Python list. The SFTTrainer expects a Hugging Face Dataset object, which provides efficient batching and shuffling during training.

  • Convert your list into a Dataset by running this in a new cell:
from datasets import Dataset

# Convert Python list to a Hugging Face Dataset
dataset = Dataset.from_list(training_data)
print(f"Dataset size: {len(dataset)} examples")
print(f"Features: {dataset.features}")
print("\nFirst example:")
print(dataset[0])

What does this code do?

Dataset.from_list() takes your Python list of dictionaries and converts it into an Arrow-backed Dataset object. Arrow is a columnar format that makes data access fast and memory-efficient.

The features output shows you the dataset schema. You should see a messages column with role and content string fields inside it.

Before you run this, what do you expect dataset[0] to look like?

You should see output showing 15 examples, and dataset[0] prints a dictionary with a messages field containing your first conversation: a user asking about list comprehensions and an assistant giving a concise answer.

Seeing an error about mixed types?

If you get an ArrowInvalid error, one of your examples likely has a different structure. Every example must have the exact same keys and nesting. Check that all examples have messages as the only top-level key, and each message has both role and content.

help me fix an ArrowInvalid error when creating a Hugging Face Dataset from my training data list

✔️ Awesome, I've got everything!

Great. Double check you have saved and run both cells before moving on.

ⓧ I'd like to double check the full code

Your notebook should have two new cells from this step. Scroll up and confirm each is present in order:

  • Cell 8: training_data list with 15 dictionaries. Each has a messages key containing a user/assistant pair. Topics range from list comprehensions to decorators.
  • Cell 9: from datasets import Dataset, then Dataset.from_list(training_data), then print(dataset[0]) showing the first example's structure.

Your dataset is built and verified. Next up, you will configure LoRA adapters and set up the trainer so this data can actually teach the model a new style.

Configure LoRA Adapters

Your dataset is ready, but you have a problem. SmolLM2 has 1.7 billion parameters. Retraining all of them on a free T4 GPU is not practical.

LoRA (Low-Rank Adaptation) solves this by attaching small "adapter" layers to the model and training only those. You end up training about 0.1% of the total parameters, roughly 2 million instead of 1.7 billion. This is the technique used in production AI engineering to fine-tune large models affordably.

In this step, get ready to:
  • Create a LoRA configuration that tells the model which layers get adapters.
  • Set up training arguments that control how long and how fast the model learns.
  • Initialize the SFTTrainer that combines your model, dataset, and LoRA config into one training pipeline.
Create the LoRA configuration

LoRA works by injecting small trainable matrices into specific layers of the model. You need to tell it which layers to target and how large those matrices should be.

  • Create a new code cell in your notebook and configure the LoRA adapter by running this code:
from peft import LoraConfig

# Configure LoRA adapter settings
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=['q_proj', 'v_proj'],
    task_type='CAUSAL_LM'
)

print(lora_config)

What does each parameter control?

  • r=16 is the rank of the adapter matrices. Higher rank means more learning capacity but more parameters to train. 16 is a common sweet spot.
  • lora_alpha=32 is the scaling factor that controls how much the adapters influence the model output. A common pattern is to set alpha to 2x the rank.
  • target_modules=['q_proj', 'v_proj'] tells LoRA to attach adapters to the query and value projection layers inside each attention block. These are the layers where the model decides what to pay attention to.
  • task_type='CAUSAL_LM' tells PEFT this is a text generation (causal language model) task.
  • Run the cell.

You should see the LoRA configuration printed out, showing all the parameters you just set.

Seeing an import error?

Make sure you ran the pip install cell from Step 1 that includes peft. If the kernel restarted, re-run all previous cells from the top.

Help me fix this import error

Set up the training arguments

The training arguments control the mechanics of how training runs: how many times the model sees your data, how fast it learns, and how often it reports progress.

  • Create a new code cell and configure the training settings by running this code:
from trl import SFTConfig, SFTTrainer

# Set training parameters
sft_config = SFTConfig(
    output_dir='smollm2-python-tutor',
    num_train_epochs=3,
    per_device_train_batch_size=2,
    learning_rate=2e-4,
    logging_steps=5,
    max_seq_length=512
)

print("Training config ready!")

What does each training setting do?

  • output_dir='smollm2-python-tutor' is where training checkpoints get saved.
  • num_train_epochs=3 means the model sees your entire dataset 3 times. More passes give it more chances to learn the patterns.
  • per_device_train_batch_size=2 processes 2 examples at a time. Small batches fit in GPU memory.
  • learning_rate=2e-4 controls how big the weight updates are each step. This value (0.0002) is the recommended rate for LoRA fine-tuning.
  • logging_steps=5 prints the training loss every 5 steps so you can watch progress in real time.
  • max_seq_length=512 caps how long each training example can be in tokens. This keeps memory usage under control.
  • Run the cell. You should see "Training config ready!" printed in the output.

Seeing a deprecation warning?

TRL may print a deprecation warning about certain arguments moving to SFTConfig. This is just a notice about future versions. Your code still works correctly.

Help me understand this warning

Initialize the trainer and verify trainable parameters

Now you bring everything together. The SFTTrainer combines your quantized model, dataset, training settings, and LoRA config into a single training pipeline. It also handles applying the LoRA adapters to the model automatically.

  • Create a new code cell and initialize the trainer by running this code:
# Initialize the trainer with model, dataset, and LoRA config
trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    peft_config=lora_config
)

# Check how many parameters are actually trainable
trainer.model.print_trainable_parameters()

What does this code do?

SFTTrainer takes your already-loaded quantized model, wraps it with the LoRA adapters you configured, and connects it to your training dataset. You pass peft_config directly to the trainer so it handles applying LoRA internally.

print_trainable_parameters() shows you exactly how many parameters LoRA added versus the total model size. This confirms the efficiency of LoRA: millions trained instead of billions.

Before you run this cell, what percentage of the total parameters do you think LoRA will make trainable? Remember, the model has 1.7 billion parameters and LoRA targets only the query and value projection layers.

  • Run the cell.

You should see output in the format trainable params: X || all params: Y || trainable%: Z where the trainable percentage is around 0.1-0.5%. That tiny fraction is all LoRA needs to teach the model your tutoring style.

Seeing an error about model or dataset not defined?

Make sure you have run all previous cells in order. The variable model was created when you loaded SmolLM2 in Step 2. The variable dataset was created when you built your training data in Step 3. If your Colab session disconnected, re-run all cells from the top.

Help me fix this NameError

✔️ Awesome, I've got everything!

Great. Make sure you see the trainable parameters output showing a very small percentage (around 0.1-0.5%) before moving on.

ⓧ I'd like to double check the full code

Here are the three cells you should have added in this step. Scroll up and confirm each one is present and matches.

Cell 10 (LoRA configuration):

from peft import LoraConfig

# Configure LoRA adapter settings
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=['q_proj', 'v_proj'],
    task_type='CAUSAL_LM'
)

print(lora_config)

Cell 11 (training arguments):

from trl import SFTConfig, SFTTrainer

# Set training parameters
sft_config = SFTConfig(
    output_dir='smollm2-python-tutor',
    num_train_epochs=3,
    per_device_train_batch_size=2,
    learning_rate=2e-4,
    logging_steps=5,
    max_seq_length=512
)

print("Training config ready!")

Cell 12 (trainer initialization):

# Initialize the trainer with model, dataset, and LoRA config
trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    peft_config=lora_config
)

# Check how many parameters are actually trainable
trainer.model.print_trainable_parameters()

Your training pipeline is locked and loaded. Next up, you will actually run the training and watch the model learn from your dataset in real time.

Train and Evaluate Your Chatbot

Your trainer is configured, your dataset is formatted, and your LoRA adapters are attached. Now it is time to actually run training and see whether the model learns from your data.

This is the moment the fine-tuning pipeline becomes real. You will watch the training loss decrease in real-time, compare the model's responses before and after training, and push your trained adapter to Hugging Face Hub so anyone can use it.

In this step, get ready to:
  • Run training and observe the loss curve decreasing.
  • Compare the base model vs. fine-tuned model on the same question.
  • Push your trained LoRA adapter to Hugging Face Hub.
Run training and watch the loss decrease

Training loss is a number that tells you how far the model's predictions are from your training examples. As training progresses, this number should decrease, meaning the model is getting better at producing your tutoring-style responses.

  • Start training by running this code in a new cell:
# Start the fine-tuning process
# This trains the LoRA adapters on your dataset
trainer.train()

What does this code do?

This single line kicks off the entire training loop. The trainer iterates through your dataset for 3 epochs (passes), updating only the LoRA adapter weights each time.

Every 5 steps (your logging_steps setting), it prints the current training loss. You should see the loss start around 1.5-2.5 and gradually decrease toward 0.5-1.0 over the course of training.

Before you run the cell, what do you predict will happen to the loss number over time?

  • Run the cell and watch the output.
  • Training should complete in under 5 minutes on your T4 GPU.
  • You should see a table of loss values printed every 5 steps, with the numbers trending downward.

What does decreasing loss mean?

Each loss value represents how surprised the model is by your training examples. A high loss means the model's predictions are far from your tutoring responses.

As loss decreases, the model is learning to produce text that matches the concise, step-by-step style in your dataset. The model is literally becoming a better Python tutor with each training step.

Training seems stuck or loss isn't decreasing?

If training takes longer than 10 minutes, check that your GPU is active by running !nvidia-smi in a new cell. If it shows 0% GPU utilization, go to **Runtime** > **Change runtime type** and confirm T4 GPU is selected.

If loss stays flat or increases, your learning rate may be too high. Check that learning_rate=2e-4 is set correctly in your SFTConfig from the previous step.

Need more help? Ask: My training loss isn't decreasing after multiple steps. What could be wrong with my LoRA fine-tuning setup?

Compare before vs. after on the same question

The real test of fine-tuning is not a loss number. It is whether the model actually responds differently when you ask it a question.

  • Run this code in a new cell to compare the base model and fine-tuned model side by side:
# The same question from Step 2
messages = [
    {"role": "user", "content": "Explain what a Python list comprehension is, step by step for a beginner"}
]

# Format the prompt using the chat template
input_text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

What does this code do?

This prepares the exact same question you asked the base model back in Step 2. By using the same input, you can directly compare how the model's response has changed after training.

The apply_chat_template method wraps your message in the special ChatML tokens (<|im_start|>, <|im_end|>) that the model expects.

  • Add this code to the same cell to generate both responses and print them:
# Generate with the fine-tuned model (adapters active)
fine_tuned_output = trainer.model.generate(
    **inputs, max_new_tokens=256
)
fine_tuned_response = tokenizer.decode(
    fine_tuned_output[0], skip_special_tokens=True
)

# Generate with the base model (adapters disabled)
with trainer.model.disable_adapter():
    base_output = trainer.model.generate(
        **inputs, max_new_tokens=256
    )
    base_response = tokenizer.decode(
        base_output[0], skip_special_tokens=True
    )

# Print both responses side by side
print("=== BASE MODEL (before fine-tuning) ===")
print(base_response)
print("\n" + "=" * 50 + "\n")
print("=== FINE-TUNED MODEL (after training) ===")
print(fine_tuned_response)

What's happening here?

The disable_adapter() context manager temporarily turns off the LoRA adapters, letting you see what the original base model would produce. Outside the context manager, the adapters are active and you get the fine-tuned response.

This is a clean way to compare before and after without needing to reload the model. The fine-tuned response should be more concise, step-by-step, and focused on teaching compared to the base model's generic output.

Before you run this cell, do you think the fine-tuned model will produce a noticeably different response from the base model?

  • Run the cell and compare the two outputs.
  • The fine-tuned model should give a more concise, tutoring-style response compared to the base model's verbose or unfocused answer.

Responses look similar?

If both responses look nearly identical, check that your training actually completed by scrolling up to confirm you see decreasing loss values in Cell 13's output.

With only 15-20 training examples, the shift in style may be subtle rather than dramatic. Look for differences in length, structure (numbered steps vs. paragraphs), and directness of explanation.

Need more help? Ask: My fine-tuned model's output looks the same as the base model. How can I improve my LoRA fine-tuning results?

Push your adapter to Hugging Face Hub

Your fine-tuned model exists only in this Colab session's memory right now. If the session ends, your training work disappears.

You will save the trained LoRA adapter and upload it to your Hugging Face profile. The key insight is that you are only uploading the small adapter weights (about 6 MB), not the full 1.7 billion parameter model.

  • Save and push your adapter by running this code in a new cell:
# Save the adapter locally first
trainer.model.save_pretrained("smollm2-python-tutor")

# Push the adapter to your Hugging Face profile
trainer.model.push_to_hub(
    "[[HF_USERNAME="your-huggingface-username"]]/smollm2-python-tutor"
)

What does this code do?

The save_pretrained() call writes two files to a local folder: adapter_model.safetensors (the trained weights) and adapter_config.json (the LoRA settings).

The push_to_hub() call creates a new repository on your Hugging Face profile and uploads these files. Anyone can later load your adapter with a single line of code and apply it to the base SmolLM2 model.

  • Run the cell. You should see a progress bar as the adapter uploads.

Getting a permission error?

Make sure you logged into Hugging Face Hub with a write token back in Step 1. If your token was read-only, re-run the login cell with a new token that has write permissions.

You can create a new write token at huggingface.co/settings/tokens.

Need more help? Ask: I'm getting a permission error when trying to push my PEFT model to Hugging Face Hub. How do I fix my token?

Now verify your adapter is live on the Hub.

  • Open a new browser tab and go to your Hugging Face profile at huggingface.co.
  • Click on your profile icon, then **Models**.
  • You should see a new repository called smollm2-python-tutor.
  • Click into it and confirm you see adapter_model.safetensors and adapter_config.json in the **Files** tab.

Why only ~6 MB?

LoRA trains a tiny fraction of the model's parameters (around 0.1%). The adapter files contain only those small trained matrices, not the full 1.7 billion parameter model.

To use your fine-tuned model, someone loads the base SmolLM2 model and then applies your adapter on top. This is what makes LoRA so practical for sharing and deploying fine-tuned models.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three cells in your notebook.

ⓧ I'd like to double check the full code

Here are the three cells you added in this step. Scroll up and confirm each one is present, in order:

  • Cell 13: trainer.train() with loss output visible.
  • Cell 14: Message formatting, fine-tuned generation, base model generation with disable_adapter(), and side-by-side print.
  • Cell 15: save_pretrained("smollm2-python-tutor") and push_to_hub("<username>/smollm2-python-tutor").

You just trained your own AI chatbot and published it for the world to use. Next up, you will take on a bonus challenge that teaches the model to prefer better responses over worse ones.

Secret mission

Add DPO Training to Teach Quality Preferences

Your SFT-trained model knows what good tutoring looks like, but it cannot tell the difference between a good and bad response. DPO (Direct Preference Optimization) teaches the model to actively prefer helpful responses over unhelpful ones. This is the same technique companies use to align production AI systems.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely in Google Colab with free GPU resources. There are no ongoing costs, but you may want to free up your GPU allocation or remove repositories you no longer need.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Google Colab notebook (saved to your Google Drive).
  • Hugging Face model repository (SFT adapter): <username>/smollm2-python-tutor.
  • Hugging Face model repository (DPO adapter): <username>/smollm2-python-tutor-dpo.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your fine-tuned models or want to share them with others.

  • Your Colab notebook stays in Google Drive and you can reopen it anytime.
  • Your Hugging Face repositories remain public so anyone can download and use your adapters.
  • Colab automatically disconnects your GPU runtime after it idles, so there is no ongoing cost.

Pause - I'll come back to this later

Free up your GPU allocation now and come back to the notebook later. Your work is saved.

  • In your Colab notebook, click Runtime in the top menu.
  • Select Disconnect and delete runtime.
  • This releases the T4 GPU back to Colab's pool. Your notebook code and outputs remain saved in Google Drive.
  • Your Hugging Face repositories stay online with no action needed.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • In your Colab notebook, click Runtime in the top menu.
  • Select Disconnect and delete runtime to release the GPU.
  • To delete the notebook itself, open Google Drive and find your .ipynb file.
  • Right-click the file and select Move to trash.
  • To delete your SFT adapter, go to huggingface.co and open your smollm2-python-tutor model page.
  • Click the Settings tab on the model page.
  • Scroll to the bottom and click Delete this repository. Confirm the deletion when prompted.
  • Repeat for your DPO adapter repository (smollm2-python-tutor-dpo).

Nice Work!

Nice Work!

You just fine-tuned your own AI chatbot from scratch. Starting with a generic language model, you built a complete training pipeline, watched it learn from your data, and published the result to the world.

You've learned how to:

  • Build a complete fine-tuning pipeline using transformers, trl, peft, and bitsandbytes in Google Colab.
  • Configure LoRA adapters and 4-bit quantization to train a 1.7B parameter model on a free GPU.
  • Train a custom Python tutor chatbot, observe the concrete before/after difference, and push your trained adapter to Hugging Face Hub for anyone to use.
  • Secret Mission: Applied DPO (Direct Preference Optimization) to teach the model quality preferences, completing the modern RLHF training pipeline.

Ready to quiz yourself?