Fine-Tune Your First LLM with Unsloth

Fine-tune a Llama 3B model using LoRA and 4-bit quantization on free hardware.

Introduction

30 Second Summary

Every major AI company takes a general-purpose language model and customizes it for their specific needs. That customization process is called fine-tuning, and until recently, it required expensive hardware and deep expertise.

In this project, you will fine-tune your own large language model using free cloud hardware. You will take a general-purpose Llama 3.2 model and train it to follow instructions in a structured, concise style.

What You'll Build

A fine-tuned Llama 3.2 3B language model that responds to instructions in a specialised format, trained entirely on Google Colab's free T4 GPU using Unsloth.

By the end of this project, you'll have:

  • A working Google Colab notebook with the complete fine-tuning pipeline from model loading to inference.
  • A fine-tuned Llama 3.2 3B model with LoRA adapters that responds in a structured, instruction-following style.
  • A before-and-after comparison showing how fine-tuning changed your model's behaviour.
  • Secret Mission: Export your fine-tuned model to GGUF format and run it locally with Ollama.

Are there any prerequisites?

You only need basic Python knowledge (variables, functions, loops). No machine learning experience is required. This project teaches you the fine-tuning concepts from the ground up.

Set Up Google Colab with GPU Access

Fine-tuning a large language model requires a GPU to handle the heavy matrix computations involved in training. Most personal laptops don't have the right hardware for this.

Google Colab gives you free access to an NVIDIA T4 GPU with 16 GB of VRAM directly in your browser. In this step, you will create a Colab notebook and connect it to a T4 GPU so you are ready to run training code.

In this step, get ready to:
  • Open Google Colab and create a new notebook.
  • Switch the runtime to a T4 GPU.
  • Verify that the GPU is detected and ready for training.
Open Google Colab and create a new notebook

Google Colab is a free, browser-based notebook environment hosted by Google. It lets you write and run Python code without installing anything locally.

  • Open your browser and go to colab.research.google.com.
  • Sign in with your Google account if prompted.
  • Click New notebook in the bottom-left corner of the welcome dialog.

What is Google Colab?

Colab is Google's hosted version of Jupyter Notebooks. It runs entirely in the cloud, which means your code executes on Google's servers rather than your own machine.

This is crucial for fine-tuning because it gives you free access to GPUs that would otherwise cost thousands of dollars to own.

You should now see a blank notebook with one empty code cell.

Change the runtime to T4 GPU

By default, Colab runs your code on a standard CPU. For fine-tuning, you need to switch to a GPU runtime so the model training runs fast enough to be practical.

  • Click Runtime in the top menu bar.
  • Click Change runtime type.
  • In the dialog that appears, find the Hardware accelerator dropdown.
  • Select T4 GPU from the dropdown.
  • Click Save.

Why the T4 GPU?

The NVIDIA T4 has 16 GB of VRAM, which is just enough to load a 3-billion parameter model in 4-bit precision and run training. Colab offers it for free, making it the most accessible option for learning fine-tuning without any cost.

Don't see T4 GPU in the list?

Google's free GPU quota fluctuates based on demand. If T4 is not available, try again in a few minutes or at a different time of day.

Make sure you are signed into a personal Google account. Workspace or school accounts sometimes have restricted access to GPU runtimes.

help me get T4 GPU access on Google Colab

Verify GPU access by running a quick check cell

Before you install anything, confirm that Colab actually connected you to a GPU. A quick Python cell will show you exactly which hardware is available.

  • Click into the empty code cell in your notebook and paste the following code:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
  • Run the cell by clicking the play button on the left side of the cell, or by pressing Shift+Enter.

What does this code do?

torch.cuda.is_available() checks whether CUDA (NVIDIA's GPU computing platform) is accessible from this notebook.

torch.cuda.get_device_name(0) prints the name of the connected GPU. You should see Tesla T4 if everything is set up correctly.

✔️ I see CUDA available: True and Tesla T4

Your GPU is connected and ready. You can move on to the next step where you will install Unsloth.

ⓧ I see CUDA available: False

This means the runtime is not connected to a GPU.

  • Go to Runtime in the top menu.
  • Click Change runtime type.
  • Confirm that T4 GPU is selected under Hardware accelerator.
  • Click Save, then re-run the cell.

If it still shows False, try disconnecting and reconnecting: click Runtime then Disconnect and delete runtime, then reconnect.

Your Colab environment is set up and connected to a GPU. Next up, you will install Unsloth and its companion libraries so you can start loading and fine-tuning a model.

Install Unsloth and Dependencies

Your Colab notebook is connected to a T4 GPU and ready for action. Now you need the fine-tuning toolkit that makes training an LLM possible on free hardware.

Unsloth is a library that speeds up LLM fine-tuning while dramatically reducing memory usage. It works alongside companion libraries like TRL (training loop), PEFT (parameter-efficient adapters), and bitsandbytes (quantization). In this step, you'll install the full stack and confirm everything is wired up correctly.

In this step, get ready to:
  • Install Unsloth and its companion libraries.
  • Import core modules and confirm the GPU is detected with sufficient VRAM.
Install Unsloth and its companion libraries
  • Click + Code in the toolbar to create a new code cell.
  • Paste the following code into the cell:
%%capture
!pip install unsloth
!pip install --no-deps trl peft accelerate bitsandbytes
  • Run the cell by pressing Shift+Enter.

What does this code do?

%%capture is a Colab magic command that suppresses all output from the cell. Installation logs can be hundreds of lines long, so this keeps your notebook clean.

The first !pip install unsloth installs Unsloth and its core dependencies. The second line installs the companion libraries (trl, peft, accelerate, bitsandbytes) with --no-deps to prevent pulling in conflicting dependency versions that Colab already provides.

You won't see any output from this cell. That's expected because %%capture suppresses it. The next cell will confirm everything installed correctly.

Cell seems stuck?

Installation typically takes 1-2 minutes. The spinning icon next to the cell means it's still running. If it runs for more than 5 minutes, click Runtime then Restart session and try again.

help me debug a stuck pip install cell in Google Colab

Import core modules and verify GPU access

With the libraries installed, you need to import them and confirm the GPU is accessible to PyTorch. This cell acts as your go/no-go check before loading a model.

  • Click + Code to create another new code cell below.
  • Paste this code into the new cell:
import torch
from unsloth import FastLanguageModel

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")
  • Run the cell by pressing Shift+Enter.

What does this code do?

The first two lines import torch (PyTorch, the deep learning framework) and FastLanguageModel (Unsloth's main class for loading and fine-tuning models).

The print statements report your CUDA status, GPU name, and available VRAM. If any of these fail, you'll know immediately that something is wrong before investing time in training.

  • You should see output showing CUDA available: True, a GPU name of Tesla T4, and VRAM of approximately 15.0 GB.

Seeing CUDA available: False?

This means your runtime isn't connected to a GPU. Click Runtime in the menu bar, then Change runtime type, and confirm T4 GPU is selected. After changing it, you'll need to re-run Cell 1 (the install cell) before running this cell again.

If you see ModuleNotFoundError: No module named 'unsloth', go back to Cell 1, remove the %%capture line temporarily, and re-run it to see the full installation output. Look for error messages in red.

help me fix CUDA or import errors in my Colab fine-tuning notebook

✔️ Awesome, I've got everything!

Great. Double check you've run both cells successfully before moving on.

ⓧ I'd like to double check the full code

Here are both cells for this step. Make sure your notebook matches:

%%capture
!pip install unsloth
!pip install --no-deps trl peft accelerate bitsandbytes
import torch
from unsloth import FastLanguageModel

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")

Your fine-tuning toolkit is installed and your GPU is confirmed. Next up, you'll load a real language model into memory using 4-bit quantization to fit it within your T4's VRAM.

Load the Base Model in 4-Bit Precision

Your environment is verified and ready to go. Now it's time to load the actual language model you'll be fine-tuning.

The challenge is that Llama 3.2 3B has 3 billion parameters. At full precision, it would need far more memory than your T4 GPU's 16 GB of VRAM. The solution is 4-bit quantization, which compresses the model so it fits comfortably within your free GPU.

In this step, get ready to:
  • Load the Llama 3.2 3B model in 4-bit precision using Unsloth.
  • Verify the model fits within your T4 GPU's VRAM.
Load Llama 3.2 3B with 4-bit quantization

Quantization reduces the number of bits used to store each model parameter. A standard model uses 16 or 32 bits per parameter. With 4-bit quantization, each parameter uses only 4 bits, shrinking the model's memory footprint by roughly 4x.

  • Create a new code cell in your notebook and paste the following code:
# Configuration for model loading
max_seq_length = 2048  # Maximum number of tokens per input sequence
dtype = None  # Auto-detect: Float16 for T4, BFloat16 for newer GPUs
load_in_4bit = True  # Compress model to 4-bit to fit in 16GB VRAM

# Load the base model and its tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

print("Model loaded successfully!")

What does this code do?

This cell configures three settings and then loads the model:

  • max_seq_length = 2048 sets the maximum number of tokens the model can process in a single input. This is a good default for training efficiency on a free Colab account.
  • dtype = None tells Unsloth to auto-detect the best floating-point format for your GPU. The T4 uses Float16.
  • load_in_4bit = True enables QLoRA-style 4-bit quantization. This is what lets the 3-billion-parameter model fit inside your 16 GB of VRAM.
  • FastLanguageModel.from_pretrained() downloads the pre-quantized model from Hugging Face and returns two objects: the model (the neural network itself) and the tokenizer (which converts text into numbers the model can read).
  • Run the cell by pressing the play button or pressing Shift+Enter.
  • Wait for the download and loading to complete. This takes 1-2 minutes depending on your connection speed.
  • You should see Unsloth's loading output followed by Model loaded successfully! at the bottom.

What should I see?

Unsloth prints several lines while loading, including the model architecture and memory usage details. The key confirmation is the final line: Model loaded successfully!.

Notice how the model occupies roughly 2-4 GB of VRAM rather than the 6+ GB it would need at 16-bit precision. That is 4-bit quantization at work.

Cell stuck or showing an error?

  • If the cell appears to hang for more than 5 minutes, check that your runtime is still connected. Click Runtime > Manage sessions and confirm your session is active.
  • If you see a CUDA out of memory error, go to Runtime > Restart session and re-run all cells from the top. Another program may have allocated VRAM before your cell ran.
  • If you see a model-not-found error, double-check that the model_name string is exactly unsloth/Llama-3.2-3B-Instruct with no typos.

help me troubleshoot my model loading error in Unsloth on Google Colab

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook and the Model loaded successfully! message is visible in the cell output.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 3:

max_seq_length = 2048
dtype = None
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

print("Model loaded successfully!")

Your model is loaded and sitting on the GPU, ready for fine-tuning. Next up, you'll attach LoRA adapters to make training incredibly efficient.

Attach LoRA Adapters for Efficient Training

Your Llama 3.2 3B model is loaded and compressed into 4-bit precision. It fits comfortably in your T4 GPU's memory.

The model still has over 3 billion parameters. Training all of them would take days and far more VRAM than your free GPU provides. LoRA solves this by freezing the original model and training only a small set of adapter layers on top.

In this step, get ready to:
  • Configure and attach LoRA adapters to the model's key layers.
  • Check how many trainable parameters you have versus the full model.
Configure and attach LoRA adapters to the model

LoRA (Low-Rank Adaptation) works by injecting small trainable matrices into the model's existing layers. The base model's billions of parameters stay frozen, and only the adapters get updated during training.

  • Create a new code cell in your notebook and attach LoRA adapters to your model by running this code:
# Attach LoRA adapters to make only a small fraction of parameters trainable
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

# Print how many parameters are now trainable vs total
model.print_trainable_parameters()

What does this code do?

FastLanguageModel.get_peft_model() wraps your base model with LoRA adapter layers. Only these adapters will be updated during training.

  • r = 16 sets the rank of the adapter matrices. Higher rank means more capacity to learn, but also more memory. 16 is a solid starting point.
  • target_modules lists which layers inside the model get adapters. These are the attention and feed-forward layers where the model does most of its "thinking".
  • use_gradient_checkpointing = "unsloth" saves about 60% VRAM by recomputing activations during backpropagation instead of storing them all in memory.
  • random_state = 3407 fixes the random seed so your results are reproducible.
  • Run the cell by pressing the play button or Shift+Enter.
  • You should see output showing the trainable parameter count. It will look something like: trainable params: X || all params: Y || trainable%: Z where the trainable percentage is roughly 1-2% of total parameters.

Why is only ~1% of parameters trainable?

The base model's 3 billion parameters stay completely frozen. LoRA only adds and trains a tiny set of adapter matrices (around 40-80 million parameters depending on configuration).

This is what makes fine-tuning possible on free hardware. Training 100% of parameters would require multiple expensive GPUs. Training ~1% gives you most of the benefit at a fraction of the cost.

Seeing an out-of-memory error?

If you see a CUDA out of memory error, your GPU session may have leftover memory from a failed previous run.

  • Go to Runtime > Restart session in the Colab menu and re-run all cells from the top.
  • Make sure you selected T4 GPU as your runtime type, not CPU.

Help me fix a CUDA out of memory error when attaching LoRA adapters in Unsloth on Google Colab

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 4:

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

model.print_trainable_parameters()

Your model is now ready to learn. Next up, you'll prepare the training dataset that teaches it how to respond.

Prepare the Training Dataset

Your model now has LoRA adapters attached and is ready to learn. The next ingredient is data.

Fine-tuning works by showing the model thousands of example conversations until it picks up the pattern you want. In this step, you will load a curated instruction-following dataset and format each example into a prompt template that the model can learn from.

In this step, get ready to:
  • Load the Alpaca GPT-4 dataset from Hugging Face.
  • Format examples using the Alpaca prompt template.
  • Preview the formatted data to confirm it looks right.
Load and format the Alpaca GPT-4 dataset

The Alpaca GPT-4 dataset contains about 52,000 instruction-following examples. Each example has three fields: an instruction telling the model what to do, an optional input providing context, and a response showing the ideal answer.

Why this dataset?

The Alpaca GPT-4 dataset uses high-quality outputs generated by GPT-4. This produces better fine-tuning results than datasets with lower-quality responses.

Its instruction/input/output structure maps cleanly to the prompt template you will define, making formatting straightforward.

  • Create a new code cell in your notebook and paste the following code:
from datasets import load_dataset

# Load the Alpaca GPT-4 instruction-following dataset
dataset = load_dataset("vicgalle/alpaca-gpt4", split="train")

# Define the prompt template that structures each training example
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

# End-of-sequence token tells the model where a response ends
EOS_TOKEN = tokenizer.eos_token

def formatting_prompts_func(examples):
    """Format each example into the Alpaca template for training."""
    instructions = examples["instruction"]
    inputs = examples["input"]
    outputs = examples["output"]
    texts = []
    for instruction, input_text, output in zip(instructions, inputs, outputs):
        # Fill the template and append EOS so the model learns when to stop
        text = alpaca_prompt.format(instruction, input_text, output) + EOS_TOKEN
        texts.append(text)
    return {"text": texts}

# Apply the formatting function to every example in the dataset
dataset = dataset.map(formatting_prompts_func, batched=True)

# Confirm the dataset loaded and preview the first example
print(f"Dataset size: {len(dataset)} examples")
print(f"\nExample:\n{dataset[0]['text'][:500]}...")

What does this code do?

  • load_dataset("vicgalle/alpaca-gpt4", split="train") downloads the full 52K-example training set directly from Hugging Face.
  • alpaca_prompt defines the exact text structure the model sees during training. Each example gets slotted into the Instruction / Input / Response sections.
  • EOS_TOKEN is appended to every example so the model learns exactly where to stop generating text.
  • formatting_prompts_func takes batches of raw examples, fills in the template for each one, and returns a text field that the trainer will consume.
  • dataset.map(..., batched=True) applies the formatting function across the entire dataset in one efficient pass.
  • Run the cell by pressing Shift+Enter.
  • You should see output showing the dataset size (around 52,000 examples) followed by a preview of the first formatted example.

Seeing a download error?

  • Make sure your Colab notebook has internet access. If your session disconnected, reconnect by clicking Connect in the top right, then re-run the cell.
  • If you see a NameError for tokenizer, go back and run Cell 3 (which loads the model and tokenizer) before running this cell.

Why is my dataset not loading in Colab?

Understand the Alpaca prompt template

Look at the preview output in your cell. Each training example follows the same three-part structure.

What is the Alpaca prompt template?

The Alpaca format is a standardised way to structure instruction-following data. Every example has three clearly labelled sections:

  • Instruction tells the model what task to perform (e.g. "Explain photosynthesis").
  • Input provides optional extra context. Many examples leave this blank.
  • Response is the ideal answer the model should learn to produce.

By training on thousands of these structured examples, the model learns to follow instructions and produce well-formatted responses.

Why append the EOS token?

Without an end-of-sequence token, the model would not know where to stop generating. By appending EOS_TOKEN to every training example, the model learns to produce a stop signal after completing its response.

✔️ Awesome, I've got everything!

Great. Double check you have Cell 5 saved and the output shows about 52,000 examples.

ⓧ I'd like to double check the full code

Here is the complete Cell 5 code for comparison:

from datasets import load_dataset

dataset = load_dataset("vicgalle/alpaca-gpt4", split="train")

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

EOS_TOKEN = tokenizer.eos_token

def formatting_prompts_func(examples):
    instructions = examples["instruction"]
    inputs = examples["input"]
    outputs = examples["output"]
    texts = []
    for instruction, input_text, output in zip(instructions, inputs, outputs):
        text = alpaca_prompt.format(instruction, input_text, output) + EOS_TOKEN
        texts.append(text)
    return {"text": texts}

dataset = dataset.map(formatting_prompts_func, batched=True)

print(f"Dataset size: {len(dataset)} examples")
print(f"\nExample:\n{dataset[0]['text'][:500]}...")

Your dataset is formatted and ready to go. Next up, you will configure the training loop and watch your model learn from these 52,000 examples.

Configure Training and Run Fine-Tuning

Your dataset is formatted and ready. Each example follows the Alpaca template with an instruction, optional input, and expected response.

Now comes the core of fine-tuning: configuring how the model learns from your data and running the actual training loop. You will set training hyperparameters that control speed, memory usage, and how well the model converges.

In this step, get ready to:
  • Configure the SFTTrainer with settings optimised for the T4 GPU.
  • Run training and watch the loss decrease over 60 steps.
  • Understand what key hyperparameters control.
Set up the SFTTrainer with training arguments

The SFTTrainer from the TRL library handles Supervised Fine-Tuning. It wraps the standard Hugging Face training loop with defaults that work well for instruction-following tasks.

  • Create a new code cell in your Colab notebook.
  • Enter the training configuration by pasting this code:
from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    packing = False,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none",
    ),
)

print("Starting training...")
trainer_stats = trainer.train()
print(f"Training complete! Final loss: {trainer_stats.training_loss:.4f}")

What does this code do?

The first section creates an SFTTrainer and passes it everything it needs: your model with LoRA adapters, the tokenizer, and your formatted dataset.

The TrainingArguments block controls how training runs. It sets the batch size, number of steps, learning rate, and memory optimizations. The fp16/bf16 lines auto-detect the best floating-point precision for your GPU.

The last three lines kick off training, wait for it to finish, and print the final loss value so you can confirm convergence.

Launch training and monitor loss

When you run this cell, the model starts learning from your 52,000 examples. Training with max_steps = 60 means the model will only see a subset of the data. This is enough to demonstrate the fine-tuning effect without waiting for a full pass through the dataset.

  • Run the cell by clicking the play button or pressing Shift+Enter.
  • Watch the training logs that appear below the cell. You should see a table updating with each step, showing the loss value decreasing over time.

What should I see?

The output logs a row for every step because logging_steps = 1. Each row shows the current step number and the training loss. Early steps may show loss around 1.5-2.0. By step 60, you should see it drop noticeably.

The final line prints your overall training loss. A lower number means the model is getting better at predicting the expected responses in your dataset.

Training crashed or ran out of memory?

If you see a CUDA out-of-memory error, your GPU session may have leftover memory from previous runs. Go to Runtime > Restart session, then run all cells from the top again.

If your session disconnected, Google Colab may have reclaimed your GPU. Reconnect and re-run all cells from Cell 1.

help me fix a CUDA out-of-memory error during fine-tuning in Google Colab

Understand key hyperparameters

The training arguments you just set control three important aspects of how the model learns. Here is what each group does.

What are these hyperparameters doing?

  • learning_rate = 2e-4 controls how big each update step is. Too high and the model overshoots good answers. Too low and it barely changes. 2e-4 (0.0002) is a standard starting point for LoRA fine-tuning.
  • per_device_train_batch_size = 2 means the GPU processes 2 examples at a time. This is small to fit within 16GB VRAM.
  • gradient_accumulation_steps = 4 simulates a larger batch by accumulating gradients over 4 mini-batches before updating weights. The effective batch size becomes 2 × 4 = 8. This gives more stable training without using more memory.

Why only 60 steps instead of a full epoch?

A full pass through all 52,000 examples would take much longer. Setting max_steps = 60 lets you see the fine-tuning effect quickly. For production fine-tuning, you would increase this to 1-3 full epochs (thousands of steps).

✔️ Awesome, I've got everything!

Great. Double check you see a final loss printed below the cell before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 6. Compare it against what you have in your notebook.

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    packing = False,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none",
    ),
)

print("Starting training...")
trainer_stats = trainer.train()
print(f"Training complete! Final loss: {trainer_stats.training_loss:.4f}")

Your model just learned from real instruction-following data. Next up, you will switch it to inference mode and test how fine-tuning changed its responses.

Test Your Fine-Tuned Model

Your model has finished training. The decreasing loss values suggest the model learned patterns from the dataset, but numbers alone don't tell you whether the model actually responds differently.

In this step, you'll switch your model into inference mode and send it a real prompt. You'll see firsthand how fine-tuning shaped the way your model generates responses.

In this step, get ready to:
  • Switch the model to inference mode.
  • Send a prompt and generate a response.
  • Observe how fine-tuning changed the model's response style.
Switch to inference mode and generate a response

During training, the model tracks gradients and extra metadata to learn. For generating text, you don't need any of that overhead.

Unsloth provides FastLanguageModel.for_inference() which enables native 2x faster inference by optimizing the model for text generation.

  • Create a new code cell (Cell 7) in your notebook and paste this code:
# Enable 2x faster inference with Unsloth optimizations
FastLanguageModel.for_inference(model)

# Format a test prompt using the same Alpaca template from training
prompt = alpaca_prompt.format(
    "Explain what machine learning is in simple terms.",
    "",
    ""
)

# Tokenize the prompt and move to GPU
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

# Generate a response with controlled randomness
outputs = model.generate(
    **inputs,
    max_new_tokens = 256,
    temperature = 0.7,
    top_p = 0.9,
)

# Decode token IDs back into readable text and print
response = tokenizer.batch_decode(outputs)[0]
print(response)

What does this code do?

FastLanguageModel.for_inference(model) optimizes the model for fast text generation. This enables Unsloth's native inference speedups so responses come back quickly.

alpaca_prompt.format(...) builds a prompt using the same template the model trained on. By keeping the format consistent, the model knows exactly where to start writing its response.

temperature controls randomness (lower = more predictable, higher = more creative). top_p limits which tokens the model considers at each step, keeping outputs coherent.

  • Run Cell 7 by clicking the play button or pressing Shift+Enter.
  • You should see the full Alpaca-formatted prompt printed, followed by the model's generated response under the ### Response: section.

Seeing a CUDA out of memory error?

The T4 GPU has limited VRAM. If training used most of it, the generation step may run out. Try reducing max_new_tokens from 256 to 128. If the error persists, restart the runtime and re-run all cells from the beginning.

help me fix a CUDA out of memory error when running model.generate in Google Colab

Observe how fine-tuning changed the response

Look at the response your model generated. A base Llama model without fine-tuning would often produce verbose, meandering answers or continue generating beyond the actual answer.

Your fine-tuned model learned from the Alpaca GPT-4 dataset, which contains concise, well-structured responses. You should notice the model produces a focused answer that stays on topic and stops cleanly.

What if the response seems short or repetitive?

You only trained for 60 steps, which is a small demonstration run. The model picked up some patterns from the dataset but hasn't fully internalized the response style. With a full training run (1-3 epochs over all 52,000 examples), the difference would be much more dramatic.

Try changing the instruction string in alpaca_prompt.format() to ask different questions and see how the model handles various topics.

Try different prompts

Replace the instruction text with your own questions. Try asking about different topics like history, science, or creative writing. Each prompt reveals more about what the model learned during fine-tuning.

✔️ Awesome, I've got everything!

Great. Make sure your Cell 7 output shows a generated response from your fine-tuned model.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 7:

FastLanguageModel.for_inference(model)

prompt = alpaca_prompt.format(
    "Explain what machine learning is in simple terms.",
    "",
    ""
)

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens = 256,
    temperature = 0.7,
    top_p = 0.9,
)

response = tokenizer.batch_decode(outputs)[0]
print(response)

Secret mission

Export to GGUF and Run Locally with Ollama

Your fine-tuned model is running in Colab right now. But what if you could run it completely offline, on your own machine, with no internet connection required? In this secret mission, you will export your model to GGUF format and serve it locally using Ollama.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on Google Colab's free tier, so there are no ongoing costs. Decide whether to keep your notebook for future experiments, disconnect the GPU to free up your usage quota, or delete everything entirely.

Resources you used:

  • Google Colab notebook (stored in your Google Drive).
  • Colab GPU runtime (T4 GPU session).
  • Training output files (stored in the Colab runtime's temporary storage).

Keep everything running

No action needed. Choose this if you plan to continue experimenting with your fine-tuned model or want to try different hyperparameters.

  • Your notebook stays in Google Drive and you can reopen it anytime.
  • The GPU runtime will automatically disconnect after a period of inactivity. You can reconnect later, but the training outputs in temporary storage will be lost.
  • If you want to preserve your fine-tuned model, download the outputs folder before the runtime disconnects.

Pause - I'll come back to this later

Disconnect the GPU runtime to free up your Colab usage quota. Your notebook stays saved in Google Drive.

  • In your Colab notebook, click Runtime in the top menu bar.
  • Click Disconnect and delete runtime.
  • This frees the GPU session and deletes temporary files (including training outputs). Your notebook code is preserved in Google Drive.

Delete - I don't want to use this again

Remove all project resources completely. This deletes your notebook and disconnects the GPU.

  • In your Colab notebook, click Runtime in the top menu bar, then click Disconnect and delete runtime. This frees the GPU and removes all temporary files.
  • Open Google Drive at drive.google.com.
  • Navigate to the Colab Notebooks folder.
  • Right-click your fine-tuning notebook and select Move to trash.
  • Empty your Google Drive trash to permanently delete the file.

That's a Wrap!

That's a Wrap!

You just trained your own large language model. Not many people can say that.

You've learned how to:

  • Fine-tuned Llama 3.2 3B using QLoRA on Google Colab's free T4 GPU, proving you can customise open-source models without expensive hardware.
  • Applied LoRA adapters and 4-bit quantization to make training efficient enough to run on 16GB of VRAM.
  • Built a complete Supervised Fine-Tuning (SFT) pipeline from scratch, including dataset preparation, hyperparameter configuration, and model inference.
  • Exported your fine-tuned model to GGUF format and ran it locally with Ollama.

Ready to quiz yourself?