Fine-Tune an LLM with QLoRA

Fine-tune Llama 3.2 to reason step-by-step using QLoRA on a free GPU.

Introduction

30 Second Summary

Large language models can write essays and answer questions, but getting them to reason through problems step-by-step requires targeted training. Companies fine-tune models to add this capability, but the hardware costs have traditionally been enormous.

In this project, you will fine-tune Meta's Llama 3.2 3B model to produce detailed step-by-step reasoning using QLoRA, a technique that makes fine-tuning possible on free hardware.

What You'll Build

A fine-tuned version of Llama 3.2 3B Instruct that produces chain-of-thought reasoning, trained entirely for free on Google Colab's T4 GPU using LoRA adapters and 4-bit quantization.

By the end of this project, you'll have:

  • A working Google Colab notebook with the full fine-tuning pipeline from model loading to inference.
  • Trained LoRA adapter weights saved to Google Drive, ready to share or deploy.
  • A before/after comparison proving your fine-tuned model reasons more thoroughly than the base model on the same prompt.
  • Secret Mission: Push your fine-tuned model to Hugging Face Hub and create a standalone merged checkpoint.

Are there any prerequisites?

You need intermediate Python knowledge and a basic understanding of what LLMs are. No GPU, no paid accounts, and no local machine setup required. Everything runs in your browser.

Set Up Your Environment

Fine-tuning a large language model requires three things: a GPU to run the training, a model hub to download from, and a notebook to write code in. In this project, you get all three for free.

You will create a Hugging Face account to access the base model, open a Google Colab notebook with a free T4 GPU, and install Unsloth to make fine-tuning fast and memory-efficient.

In this step, get ready to:
  • Create a Hugging Face account and access token.
  • Open a Google Colab notebook with a T4 GPU.
  • Install Unsloth and verify GPU access.
Create a Hugging Face account and access token

Hugging Face hosts the pre-trained model you will fine-tune. You need an account and an access token so your notebook can download it.

  • Go to huggingface.co and click Sign Up to create a free account.
  • Confirm your email address if prompted.
  • Once signed in, click your profile icon in the top right corner.
  • Select Settings from the dropdown menu.
  • Click the Access Tokens tab in the left sidebar.
  • Click the New token button.
  • Give your token a name (e.g. colab-finetune) and select Read as the role.
  • Click Create token.

Copy your token now

Your token is only shown once at creation time. Copy it and paste it somewhere safe (a notes app or password manager). You will need it later in the Secret Mission.

Why a Read token?

A Read token lets your notebook download models from the Hub. You do not need Write access until you push a model back to the Hub in the Secret Mission.

Open Google Colab and enable the T4 GPU

Google Colab gives you a free cloud notebook with GPU access. The T4 GPU has 16 GB of VRAM, which is more than enough for fine-tuning a 3B model with QLoRA.

  • Open colab.research.google.com in your browser and sign in with your Google account.
  • Click New notebook to create a blank notebook.
  • In the top menu, click Runtime.
  • Select Change runtime type.
  • Under Hardware accelerator, select T4 GPU.
  • Click Save.

Colab reconnects your notebook to a server with an NVIDIA T4 GPU attached. You should see RAM and disk usage graphs appear near the top right corner.

What is a T4 GPU?

The NVIDIA Tesla T4 is a Turing-architecture GPU with about 15 GB of usable VRAM. Google provides it for free (subject to availability) to accelerate machine learning notebooks.

With 4-bit quantization, a 3B parameter model needs only about 6 GB of VRAM. That leaves plenty of headroom on the T4 for training.

Install Unsloth and verify GPU access

Unsloth is an open-source library that makes LLM fine-tuning 2x faster and uses 70% less memory compared to standard Hugging Face tooling. It wraps the complexity of quantization and LoRA configuration into simpler API calls.

  • Install Unsloth and its dependencies by pasting this code into the first cell of your notebook and pressing Shift+Enter to run it:
%%capture
!pip install unsloth
!pip install --no-deps xformers trl peft accelerate bitsandbytes

What does this code do?

%%capture is a Colab magic command that suppresses all output from the cell. Install logs are noisy, so this keeps your notebook clean.

!pip install unsloth installs the Unsloth library and its core dependencies.

!pip install --no-deps xformers trl peft accelerate bitsandbytes installs additional packages without resolving their dependencies. The --no-deps flag prevents version conflicts with Colab's pre-installed PyTorch.

  • The cell produces no visible output (thanks to %%capture). Verify that your GPU is connected by creating a new cell below and running this command:
!nvidia-smi

You should see a table that includes Tesla T4 as the GPU name and approximately 15,000 MiB of memory listed.

Not seeing a GPU?

If you see an error mentioning that no GPU was found, you likely skipped the runtime change. Go back to Runtime > Change runtime type and confirm T4 GPU is selected under Hardware accelerator.

If the install cell failed silently (remember %%capture hides errors too), remove the %%capture line temporarily and rerun the cell to see the full output.

help me troubleshoot my Colab GPU setup

✔️ Awesome, I've got everything!

Great. Double check you have the install cell saved. You will not need to rerun it unless the Colab session resets.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 1. Make sure yours matches exactly:

%%capture
!pip install unsloth
!pip install --no-deps xformers trl peft accelerate bitsandbytes

Your environment is locked and loaded. Next up, you will load the Llama 3.2 3B model in 4-bit precision and attach LoRA adapters to it.

Load the Base Model in 4-Bit

Your Colab environment is ready with all dependencies installed. Now it's time to load the actual model you'll fine-tune: Meta's Llama 3.2 3B Instruct.

A 3-billion parameter model normally requires around 12GB of GPU memory in full precision. QLoRA solves this by quantizing the model to 4-bit precision, shrinking its memory footprint to approximately 6GB. This fits comfortably on the free T4 GPU's 16GB of VRAM.

In this step, get ready to:
  • Load Llama 3.2 3B in 4-bit precision using Unsloth.
  • Attach LoRA adapters to the model's attention layers.
  • Test the base model's reasoning ability as a "before" snapshot.
Load Llama 3.2 3B in 4-bit precision

4-bit quantization compresses a model's weights from 16 bits down to 4 bits per parameter. You lose minimal quality but gain massive memory savings.

  • Create a new code cell in your Colab notebook (click the + Code button at the top).
  • Load the model and set configuration variables by pasting this code:
from unsloth import FastLanguageModel

# Configuration for model loading
max_seq_length = 2048
dtype = None
load_in_4bit = True

# Download and load the model in 4-bit precision
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

What does this code do?

max_seq_length = 2048 sets the maximum number of tokens the model can process in one pass. This is the model's context window for training.

dtype = None lets Unsloth auto-detect the best precision format for your GPU. On the T4, it selects float16.

load_in_4bit = True enables 4-bit quantization, which compresses the 3B parameter model from ~12GB down to ~6GB of VRAM.

FastLanguageModel.from_pretrained downloads the model from Hugging Face, quantizes it to 4-bit on the fly, and loads it onto your GPU.

You won't run this cell yet. The next chunk completes it by attaching LoRA adapters.

Attach LoRA adapters to the model

LoRA (Low-Rank Adaptation) freezes the base model's weights and injects small trainable matrices into specific layers. Instead of updating all 3 billion parameters during training, you only train about 1% of them.

  • In the same cell, paste this code directly below the from_pretrained call:
# Attach LoRA adapters to key attention and feed-forward layers
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

What does this code do?

r=16 is the LoRA rank. It controls the size of the trainable matrices injected into each layer. Rank 16 is a solid balance between quality and memory.

target_modules lists the specific layers that receive LoRA adapters. These are the attention projection layers (q_proj, k_proj, v_proj, o_proj) and the feed-forward layers (gate_proj, up_proj, down_proj).

use_gradient_checkpointing="unsloth" enables Unsloth's optimized gradient checkpointing, which trades a small amount of compute time for significantly less VRAM usage during training.

  • Run this cell by pressing Shift+Enter.
  • Wait for the model to download and load. This takes 1-2 minutes depending on your connection speed.

You should see Unsloth print model details including the device (cuda:0) and a summary of trainable parameters.

Seeing an out-of-memory error?

Make sure you selected the T4 GPU runtime in Step 1. Go to Runtime > Change runtime type and confirm T4 GPU is selected.

If you already have a T4 and still see the error, your Colab session may have leftover memory from a previous run. Go to Runtime > Restart session and run your cells again from the top.

Still stuck? Help me debug this out-of-memory error when loading my model in Colab

✔️ Awesome, I've got everything!

Great. Double check you see the trainable parameter summary before moving on.

ⓧ I'd like to double check the full code

Here is the complete Cell 2 code for comparison:

from unsloth import FastLanguageModel

max_seq_length = 2048
dtype = None
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)
Test the base model on a reasoning prompt

Before you fine-tune anything, you need to see how the base model responds to a reasoning problem. This gives you a "before" snapshot to compare against later.

  • Create a new code cell and paste the following code to test the base model:
# Define a math reasoning prompt
messages = [
    {"role": "user", "content": "What is 23 * 47? Think step by step."}
]

# Format the prompt using Llama's chat template
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

# Set up streaming so you see tokens as they generate
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer, skip_prompt=True)

# Generate the base model's response
print("=== BASE MODEL OUTPUT (Before Fine-Tuning) ===")
_ = model.generate(
    input_ids=inputs,
    streamer=text_streamer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
)

What does this code do?

tokenizer.apply_chat_template formats your prompt into Llama 3.2's expected input structure. The add_generation_prompt=True flag tells the model it should start generating a response.

TextStreamer prints tokens to the screen as the model generates them, so you see the output word by word instead of waiting for the full response.

temperature=0.7 and top_p=0.9 control the randomness of the output. These are standard values for getting varied but coherent text.

  • Run this cell by pressing Shift+Enter.
  • Watch the model's response stream in. Notice whether it reasons step-by-step or jumps straight to an answer.

The base model typically gives a brief answer without detailed reasoning steps. This is the behaviour your fine-tuning will improve.

Seeing a CUDA error or no output?

If you see a CUDA-related error, make sure you ran Cell 2 (the model loading cell) successfully first. The model and tokenizer variables must exist before this cell can run.

If the cell hangs with no output for more than 60 seconds, the model may not have loaded correctly. Restart your session and re-run all cells from the top.

Help me troubleshoot this model generation error in Colab

✔️ Awesome, I've got everything!

Great. Double check you see the base model's response printed under the header before moving on.

ⓧ I'd like to double check the full code

Here is the complete Cell 3 code for comparison:

messages = [
    {"role": "user", "content": "What is 23 * 47? Think step by step."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer, skip_prompt=True)

print("=== BASE MODEL OUTPUT (Before Fine-Tuning) ===")
_ = model.generate(
    input_ids=inputs,
    streamer=text_streamer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
)

You now have your "before" snapshot saved. Next up, you'll prepare a dataset of step-by-step reasoning traces to teach the model how to think through problems.

Prepare the Training Dataset

You have your model loaded and you've captured a "before" snapshot of how it responds to a reasoning prompt. The output was likely brief or jumped straight to an answer without showing its work.

To teach the model to reason step-by-step, you need training data that demonstrates what good reasoning looks like. In this step, you'll load a dataset of chain-of-thought reasoning traces and format them into the exact chat template structure your model expects.

In this step, get ready to:
  • Load a reasoning dataset from Hugging Face.
  • Format the data into Llama 3.2's chat template structure.
  • Verify that formatted samples have the correct user/assistant structure.
Load the reasoning dataset from Hugging Face

The ServiceNow-AI/R1-Distill-SFT dataset contains 171,647 reasoning traces distilled from DeepSeek-R1-32B. Each sample has a math or logic problem paired with a detailed step-by-step solution.

  • Create a new code cell in your Colab notebook and add this code to load a subset of the dataset:
from datasets import load_dataset

# Load 1000 reasoning examples from the R1-Distill dataset
dataset = load_dataset(
    "ServiceNow-AI/R1-Distill-SFT",
    "v0",
    split="train[:1000]",
)

What does this code do?

load_dataset downloads the dataset directly from Hugging Face. The "v0" argument selects the dataset configuration, and "train[:1000]" grabs only the first 1000 samples to keep training time short on the free T4 GPU.

Each sample in the v0 config has two key fields: problem (the question) and reannotated_assistant_content (the step-by-step reasoning answer generated by DeepSeek-R1-32B).

Format the data into Llama 3.2 chat templates

LLMs expect training data in a specific token format. Llama 3.2 uses a chat template with special tokens that mark where the user message starts, where the assistant response begins, and so on.

  • Below the dataset loading code in the same cell, add this formatting code:
# Convert each sample into the chat format Llama 3.2 expects
def format_sample(sample):
    messages = [
        {"role": "user", "content": sample["problem"]},
        {"role": "assistant", "content": sample["reannotated_assistant_content"]},
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}

# Apply formatting to every sample and print a preview
dataset = dataset.map(format_sample)
print(dataset[0]["text"][:500])

What does this code do?

The format_sample function takes one dataset sample and structures it as a conversation: the problem becomes the user message and the reannotated_assistant_content becomes the assistant's reply.

tokenizer.apply_chat_template wraps those messages with Llama 3.2's special tokens (like <|begin_of_text|> and role markers). Setting tokenize=False returns a string rather than token IDs, which is what the trainer needs.

dataset.map(format_sample) applies this function to all 1000 samples, adding a "text" column to the dataset with the properly formatted training strings.

  • Run the cell by clicking the play button or pressing Shift+Enter.
  • Wait for the dataset to download and format. You should see the first 500 characters of a formatted sample printed below the cell, showing special tokens like <|begin_of_text|> followed by the user/assistant conversation structure.

Seeing a download error or timeout?

  • Make sure you are connected to the internet in your Colab session. Check the top-right corner for a green checkmark on the RAM/Disk indicator.
  • If you see a ConnectionError, try running the cell again. Hugging Face downloads can occasionally time out on the first attempt.
  • If you see a NameError: name 'tokenizer' is not defined, make sure you ran Cell 2 (Load Model and Attach LoRA) first. The tokenizer must exist before formatting.

Help me debug my dataset loading error in Colab.

✔️ Awesome, I've got everything!

Great. Double check you see the formatted output with special tokens before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 4. Make sure your cell matches this exactly:

from datasets import load_dataset

dataset = load_dataset(
    "ServiceNow-AI/R1-Distill-SFT",
    "v0",
    split="train[:1000]",
)

def format_sample(sample):
    messages = [
        {"role": "user", "content": sample["problem"]},
        {"role": "assistant", "content": sample["reannotated_assistant_content"]},
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}

dataset = dataset.map(format_sample)
print(dataset[0]["text"][:500])

Why not use the full 171,647 samples?

Training on 1000 samples keeps the training time around 15-20 minutes on the free T4 GPU. The full dataset would take hours and might exceed Colab's session limits.

Even with just 1000 examples, you'll see a measurable improvement in the model's reasoning. In production, you would scale up the data and training steps for better results.

Your dataset is loaded and formatted. Each sample now contains the exact token structure Llama 3.2 expects during training. Next up, you'll feed this data into the trainer and watch the model learn to reason.

Fine-Tune the Model

Your dataset is formatted and loaded with 1000 reasoning traces ready to go. Now comes the main event: actually training the model to produce step-by-step reasoning.

In this step, you'll configure a supervised fine-tuning trainer with specific hyperparameters and kick off the training loop. The model will iterate over your formatted dataset, gradually learning to produce chain-of-thought reasoning.

In this step, get ready to:
  • Configure the SFTTrainer with training hyperparameters.
  • Run training for 60 steps on the T4 GPU.
  • Confirm the model learned by observing the final training loss.
Configure and launch the training run

The SFTTrainer from the trl library handles the entire training loop for you. You provide the model, tokenizer, dataset, and a set of hyperparameters that control how training proceeds.

  • Create a new code cell in your Colab notebook and paste the following training configuration:
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)

trainer_stats = trainer.train()
print(f"Training complete! Final loss: {trainer_stats.training_loss:.4f}")

What do these hyperparameters do?

  • per_device_train_batch_size=2 processes 2 samples at a time. gradient_accumulation_steps=4 simulates a larger batch of 8 by accumulating gradients before updating weights.
  • max_steps=60 limits training to 60 optimization steps. This is enough to show a clear improvement without requiring hours of compute.
  • learning_rate=2e-4 is the standard learning rate for QLoRA fine-tuning. Too high and the model forgets what it already knows. Too low and it learns nothing.
  • optim="adamw_8bit" uses an 8-bit optimizer that reduces memory usage while maintaining training quality.
  • is_bfloat16_supported() checks whether your GPU supports bf16. The T4 does not, so training automatically uses fp16 instead.
  • Run this cell by pressing the play button or hitting Shift+Enter.

Training takes approximately 15-20 minutes

The T4 GPU needs about 15-20 minutes to complete all 60 training steps. You'll see a progress bar and log entries printing in real time as the cell runs. This is a great time to grab a coffee or review the hyperparameters above.

Because logging_steps=1 is set, you'll see a new log line printed for every single training step. Each line shows the current loss value.

Seeing an out of memory error?

  • Make sure your runtime is set to T4 GPU. Go to Runtime > Change runtime type and confirm T4 GPU is selected.
  • If you ran other cells that loaded additional models or data, restart the runtime (Runtime > Restart runtime) and run all cells from the beginning.
  • Confirm you loaded the model with 4-bit quantization in Cell 2. Without 4-bit, the model uses too much VRAM.

Help me debug my CUDA out of memory error during training.

Monitor training and confirm the model learned

The key signal that training is working is the loss value decreasing over time. Loss measures how far the model's predictions are from the correct answers in your dataset.

  • Watch the loss values in the training logs as each step completes.
  • The loss should start relatively high (around 1.0-2.0) and decrease as training progresses.
  • When training finishes, the cell prints Training complete! Final loss: followed by the average training loss.

What is a good final loss?

For this project, a final training loss below 1.0 indicates the model has learned from the reasoning data. The exact number varies between runs, but a clear downward trend in the logs is what matters most.

You are not aiming for zero loss. A loss of zero would mean the model memorized the training data exactly, which is actually a problem called overfitting.

Loss not decreasing?

  • If the loss stays flat or increases, double check that your dataset formatting cell (Cell 4) ran without errors. A broken dataset produces garbage training signal.
  • Make sure the LoRA adapters were attached in Cell 2. Without them, the frozen base model has nothing to train.

Why isn't my training loss going down?

✔️ Awesome, I've got everything!

Great. Make sure training completed and you can see the final loss printed before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for Cell 5. Compare it with what you have in your notebook:

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)

trainer_stats = trainer.train()
print(f"Training complete! Final loss: {trainer_stats.training_loss:.4f}")

Your model has now learned from 1000 reasoning traces. Next up, you'll test it on the same math prompt from earlier and see the dramatic difference in its output.

Test Your Fine-Tuned Model and Save

Your model just completed 60 training steps on reasoning traces from DeepSeek-R1. The training loss dropped over those steps, which means the model's weights shifted toward producing step-by-step reasoning.

Now comes the payoff: you will send the exact same math prompt from earlier and compare the fine-tuned output with the base model's original response. Then you will save your trained LoRA adapter to Google Drive so it persists after the Colab session ends.

In this step, get ready to:
  • Run inference on the fine-tuned model using the same prompt from Step 2.
  • Compare the before and after outputs to see the reasoning difference.
  • Save your LoRA adapter weights to Google Drive.
Run inference on the fine-tuned model

During training, the model was in training mode with gradient checkpointing enabled. Before generating text, you need to switch it back to inference mode so it produces outputs efficiently.

  • Create a new code cell in your notebook and add the following code to test the fine-tuned model:
# Switch model from training mode to inference mode
FastLanguageModel.for_inference(model)

# Use the same prompt from Step 2 to compare outputs
messages = [
    {"role": "user", "content": "What is 23 * 47? Think step by step."}
]

# Tokenize and send to GPU
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

# Generate the fine-tuned model's response
print("=== FINE-TUNED MODEL OUTPUT (After Fine-Tuning) ===")
_ = model.generate(
    input_ids=inputs,
    streamer=text_streamer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
)

What does this code do?

FastLanguageModel.for_inference(model) disables gradient checkpointing and puts the model into evaluation mode. This makes text generation faster and uses less memory.

The rest of the code is identical to what you ran in Step 2. Same prompt, same generation parameters. The only difference is the model weights now contain what it learned from the reasoning dataset.

  • Do not run this cell yet. You will add the save code below first, then run the complete cell.
Save the LoRA adapter to Google Drive

Colab VMs are ephemeral. All local files disappear when your session ends or disconnects. Saving to Google Drive ensures your trained adapter persists.

  • In the same code cell, add the following code below the inference code:
# Mount Google Drive to save files persistently
from google.colab import drive
drive.mount("/content/drive")

# Save the LoRA adapter weights and tokenizer
model.save_pretrained("/content/drive/MyDrive/llama-3.2-3b-reasoning-lora")
tokenizer.save_pretrained("/content/drive/MyDrive/llama-3.2-3b-reasoning-lora")
print("LoRA adapter saved to Google Drive!")

What does this code do?

drive.mount("/content/drive") connects your Google Drive to the Colab file system at /content/drive. This lets you read and write files that survive beyond your session.

model.save_pretrained() saves only the LoRA adapter weights (a few MB), not the entire 3B parameter model. The tokenizer files are saved alongside so you can reload the adapter later without reconfiguring.

Why only the adapter and not the full model?

LoRA works by training small adapter matrices that modify the base model's behavior. The base model weights stay frozen and unchanged. Saving just the adapter is fast, takes minimal storage, and you can reload it onto the base model anytime.

  • Run the cell by pressing Shift+Enter.
  • Colab will show a popup asking you to grant permission to access your Google Drive. Click Allow to proceed.

What should I see?

You should see two things happen in sequence. First, the fine-tuned model streams its response to the math prompt. Unlike the base model's output from Step 2, this response should include explicit reasoning steps (breaking the multiplication into parts, showing intermediate calculations).

Second, after the Drive mount completes, you should see LoRA adapter saved to Google Drive! printed at the bottom of the output.

Scroll up in the cell output and compare this response with the base model output from Cell 3 in Step 2. The base model likely gave a short answer or skipped steps. Your fine-tuned model should walk through the problem methodically.

Not seeing step-by-step reasoning?

If the output looks similar to the base model, check that your training in the previous step completed successfully. Look at the final training loss printed by Cell 5. A loss below 1.0 indicates the model learned from the data.

If the Drive mount fails or hangs, make sure you are running this in the Colab browser interface (not a VS Code extension). Also confirm you are signed into the same Google account for both Colab and Drive.

help me troubleshoot why my fine-tuned model isn't showing step-by-step reasoning after QLoRA training

✔️ Awesome, I've got everything!

Great. Double check you have saved the cell and can see the success message in the output.

ⓧ I'd like to double check the full code

Here is the complete Cell 6 for comparison:

FastLanguageModel.for_inference(model)

messages = [
    {"role": "user", "content": "What is 23 * 47? Think step by step."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

print("=== FINE-TUNED MODEL OUTPUT (After Fine-Tuning) ===")
_ = model.generate(
    input_ids=inputs,
    streamer=text_streamer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
)

from google.colab import drive
drive.mount("/content/drive")

model.save_pretrained("/content/drive/MyDrive/llama-3.2-3b-reasoning-lora")
tokenizer.save_pretrained("/content/drive/MyDrive/llama-3.2-3b-reasoning-lora")
print("LoRA adapter saved to Google Drive!")
Compare your before and after results

This is the moment that proves your fine-tuning worked. You trained on just 1000 reasoning examples for 60 steps, and the model's behavior changed measurably.

  • Scroll to Cell 3's output (your base model snapshot from Step 2).
  • Compare it side by side with Cell 6's output from this step.

What does a good before/after look like?

The base model (before) typically gives a short, direct answer like "23 * 47 = 1081" with minimal or no working shown. It might attempt a step or two but often jumps to the final answer.

The fine-tuned model (after) should break the problem into explicit steps. For example, decomposing 23 * 47 into (20 + 3) * 47, computing each part, and summing them. The reasoning style mirrors the DeepSeek-R1 traces it trained on.

Secret mission

Push to Hugging Face Hub and Merge

Your fine-tuned model lives on Google Drive, but what if you want to share it with the world? In this secret mission, you will push your LoRA adapter to the Hugging Face Hub and merge it into a standalone model that anyone can load without needing Unsloth or PEFT.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on Google Colab's free tier and Hugging Face's free tier, so there are no ongoing costs. Decide whether to keep your resources, pause them to come back later, or delete them entirely.

Resources you used:

  • Google Colab notebook with T4 GPU runtime.
  • Google Drive folder: llama-3.2-3b-reasoning-lora.
  • Google Drive folder: llama-3.2-3b-reasoning-merged (Secret Mission only).
  • Hugging Face Hub repositories (Secret Mission only).
  • Hugging Face access token.

Keep everything running

No action needed. Choose this if you plan to continue experimenting with your fine-tuned model.

  • Your Colab runtime will disconnect automatically after a period of inactivity. The notebook file stays in your Google Drive.
  • Your LoRA adapter weights remain on Google Drive and are ready to reload in a new session.
  • Your Hugging Face access token stays active for future Hub interactions.

Pause - I'll come back to this later

Free up your GPU allocation so other Colab users can access it. Your files and notebook remain intact.

  • In your Colab notebook, click Runtime in the top menu bar.
  • Select Disconnect and delete runtime. This releases the T4 GPU back to the shared pool.
  • Your notebook file and Google Drive folders remain untouched. You can reconnect to a GPU runtime anytime by reopening the notebook.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below.

  • Disconnect your Colab runtime: click Runtime > Disconnect and delete runtime.
  • Delete the notebook file: open Google Drive (drive.google.com), locate the .ipynb notebook file (usually in the Colab Notebooks folder), right-click it, and select Move to bin.
  • Delete the LoRA adapter folder: in Google Drive, navigate to My Drive, right-click the llama-3.2-3b-reasoning-lora folder, and select Move to bin.
  • Delete the merged model folder (Secret Mission only): right-click the llama-3.2-3b-reasoning-merged folder in Google Drive and select Move to bin.
  • Delete your Hugging Face Hub repositories (Secret Mission only): go to each model page on huggingface.co, click the Settings tab, scroll to the bottom, and delete the repository. Do this for both your-username/llama-3.2-3b-reasoning-lora and your-username/llama-3.2-3b-reasoning-merged.
  • Delete your Hugging Face access token: go to Settings > Access Tokens on huggingface.co, click the Manage button next to the token you created, and delete it.
  • Empty your Google Drive bin to permanently free up storage: in Google Drive, click Bin in the left sidebar, then click Empty bin.

Nice Work!

Nice Work!

Nice work! You just fine-tuned Meta's Llama 3.2 3B model to reason step-by-step through math and logic problems, all for free on a Google Colab T4 GPU.

You've learned how to:

  • Fine-tune an LLM using QLoRA, the industry-standard parameter-efficient technique that freezes base weights and trains only ~1% of parameters through LoRA adapters.
  • Use 4-bit quantization to fit a 3B parameter model comfortably on a free T4 GPU with 16GB VRAM.
  • Prepare and format a reasoning dataset for supervised fine-tuning, then compare before/after model outputs to verify the training's impact.
  • Secret Mission: Published your fine-tuned model to the Hugging Face Hub and created a standalone merged checkpoint that loads without the peft library.

Ready to quiz yourself?