The Fine-Tuning Spectrum: LoRA
Compare Feature Extraction, Full Fine-Tuning, and LoRA on the same model.
Introduction
30 Second Summary
You have probably used an AI assistant that seems to know exactly what you need. Behind the scenes, someone took a general-purpose model and fine-tuned it to do that specific job.
In this project, you will apply three different fine-tuning strategies to the same model in a single Google Colab notebook. You will compare trainable parameters and accuracy side by side, then instruction-tune a small language model using the industry-standard LoRA technique.
What You'll Build
You will run your notebook, scroll to the comparison table at the bottom, and see proof that training less than 1% of a model's parameters can match the accuracy of training all of them.
By the end of this project, you'll have:
- A printed comparison table showing three fine-tuning strategies applied to DistilBERT, with trainable parameter counts and accuracy scores you generated yourself.
- A working instruction-tuned language model that responds to your prompts, fine-tuned with LoRA adapters on a conversational dataset.
- A clear mental model of the full fine-tuning taxonomy, from Feature Extraction to Instruction Fine-Tuning, ready to explain in a technical interview.
- Secret Mission: Add QLoRA with 4-bit quantization to fine-tune models that would not fit in GPU memory at full precision.
Are there any prerequisites?
This project assumes you are comfortable with Python and understand basic machine learning concepts like training, loss, and epochs. You also need a Google account to use Google Colab.
Before We Start
Before we dive in, let's lock in what you're about to build and why it matters for your career.
Set Up Your Colab Environment
Training machine learning models requires serious computing power. Even a small model like DistilBERT trains orders of magnitude faster on a GPU than on a CPU.
Google Colab gives you free access to a GPU directly in your browser. In this step, you will create a Colab notebook, attach a T4 GPU to it, and install the Hugging Face libraries that power every training run in this project.
In this step, get ready to:
- Create a Google Colab notebook with a T4 GPU runtime.
- Install the Hugging Face ecosystem libraries.
- Confirm PyTorch sees the GPU.
Open Google Colab and enable GPU
Google Colab is a free cloud notebook environment. By default, Colab runs on a CPU. You need to switch to a GPU runtime so that model training finishes in seconds instead of minutes.
- Open colab.research.google.com in your browser and sign in with your Google account.
- Click New notebook to create a blank notebook.
- In the top menu, click Runtime.
- Select Change runtime type.
- In the dropdown, select T4 GPU, then click Save.
- Confirm the GPU is attached by pasting this command into the first cell and pressing Shift+Enter to run it:
!nvidia-smiYou should see a table listing a Tesla T4 GPU with approximately 15GB of memory.
Don't see a GPU listed?
Make sure you changed the runtime type to T4 GPU and clicked Save. The notebook may need to reconnect after the change. If Colab says no GPU is available, wait a few minutes and try again. Free-tier GPU availability varies by demand.
Help me troubleshoot why nvidia-smi is not showing a GPU in Google Colab.
Install libraries and verify your setup
This project uses several libraries from the Hugging Face ecosystem. Transformers gives you pretrained models. PEFT adds parameter-efficient adapters like LoRA. TRL provides trainers for instruction fine-tuning.
- Create a new code cell below the nvidia-smi cell by clicking the + Code button at the top left. Paste the following code into it:
# Cell 1: Setup
!pip install transformers peft trl datasets evaluate accelerate -q
import torch
import transformers
import peft
import trl
import datasets as ds_lib
print(f"transformers: {transformers.__version__}")
print(f"peft: {peft.__version__}")
print(f"trl: {trl.__version__}")
print(f"datasets: {ds_lib.__version__}")
print(f"GPU available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f"GPU name: {torch.cuda.get_device_name(0)}")What does this code do?
- !pip install installs six libraries directly into the Colab environment. The -q flag keeps the output quiet so you only see errors.
- The import lines load each library into your Python session. You will import specific classes (like AutoTokenizer and Trainer) from these libraries in later steps.
- The print statements confirm each library version loaded successfully. The final two lines verify that PyTorch can communicate with your T4 GPU.
Before you run this cell, what do you expect to see? Will PyTorch detect the GPU you enabled earlier?
- Press Shift+Enter to run the cell.
You should see four version numbers printed (one each for transformers, peft, trl, and datasets), followed by GPU available: True and GPU name: Tesla T4.
Seeing a ModuleNotFoundError?
Make sure the !pip install line is included at the top of the cell. If you ran the imports in a separate cell without the install line, the packages will not be available yet.
If GPU available prints False, click Runtime then Restart session and re-run both cells. Confirm the runtime type is still set to T4 GPU.
Help me fix a ModuleNotFoundError or GPU detection issue in my Colab setup cell.
✔️ Awesome, I've got everything!
Great. Double check you see all four version numbers and GPU available: True in the output.
ⓧ I'd like to double check the full code
Here is the complete Cell 1. Make sure your cell matches this exactly:
# Cell 1: Setup
!pip install transformers peft trl datasets evaluate accelerate -q
import torch
import transformers
import peft
import trl
import datasets as ds_lib
print(f"transformers: {transformers.__version__}")
print(f"peft: {peft.__version__}")
print(f"trl: {trl.__version__}")
print(f"datasets: {ds_lib.__version__}")
print(f"GPU available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f"GPU name: {torch.cuda.get_device_name(0)}")Your environment is locked and loaded. Next up, you will load the IMDb dataset, freeze a pretrained model, and train your first classifier to establish a Feature Extraction baseline.
Feature Extraction - Train Only the Head
Your notebook is running on a GPU with all the libraries installed. Now it's time to train your first model and establish a baseline for the rest of the project.
The simplest fine-tuning approach is called Feature Extraction. You freeze the entire pretrained model and train only a small classification head on top. This is the fastest method and uses the least memory, but it gives the model the least room to adapt. Starting here establishes a baseline that you will compare against in every step that follows.
In this step, get ready to:
- Load the IMDb dataset and tokenize it for DistilBERT.
- Freeze the pretrained model and train only the classification head.
- Evaluate accuracy and store the result for comparison.
Load and tokenize the IMDb dataset
The IMDb dataset contains 50,000 movie reviews labeled as positive or negative. You will work with a small subset to keep training fast on the free Colab GPU.
- Load the IMDb dataset and the DistilBERT tokenizer by adding this code in a new cell:
from datasets import load_dataset
from transformers import AutoTokenizer
# Load the DistilBERT tokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Load IMDb and take a small subset for speed
imdb = load_dataset("stanfordnlp/imdb")
small_train = imdb["train"].shuffle(seed=42).select(range(2000))
small_test = imdb["test"].shuffle(seed=42).select(range(500))
# Tokenize the text data
def preprocess(examples):
return tokenizer(examples["text"], truncation=True, max_length=256)
tokenized_train = small_train.map(preprocess, batched=True)
tokenized_test = small_test.map(preprocess, batched=True)What does this code do?
The full IMDb dataset has 25,000 training and 25,000 test reviews. You take only 2,000 and 500 samples respectively so that each training run finishes in minutes instead of hours.
The preprocess function converts raw review text into token IDs that DistilBERT understands. Setting max_length=256 caps each review at 256 tokens so every input has a consistent size.
- Run the cell. You should see the IMDb dataset downloading and progress bars as the tokenizer processes each split.
Not seeing any download progress?
Make sure you created a new cell below your setup cell from Step 1. If you pasted the code into the same cell, re-run it as a standalone cell.
If you see a connection error, check that your Colab runtime is still connected by looking at the green checkmark in the top right.
Help me debug a dataset loading error in Google Colab.
- Set up the accuracy metric and a results dictionary by adding this code in a new cell:
import evaluate
accuracy_metric = evaluate.load("accuracy")
import numpy as np
# Define how to compute accuracy from model predictions
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return accuracy_metric.compute(predictions=predictions, references=labels)
# Store results for comparison across methods
results = {}What does this code do?
The compute_metrics function takes the model's raw output (logits) and picks the class with the highest score as the prediction. It then compares those predictions against the true labels to calculate accuracy.
The empty results dictionary is where you will store the accuracy from each fine-tuning method. By the end of the project, this dictionary powers your comparison table.
- Run the cell. It completes without errors.
Freeze the model and count trainable parameters
Feature Extraction works by loading a pretrained model and locking all of its learned weights in place. Only a small classification head on top remains trainable. The pretrained layers act as a fixed feature extractor, passing their representations up to the head without changing.
- Load DistilBERT for classification, freeze its base, and print the trainable parameter count by adding this code in a new cell:
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer
# Load DistilBERT with a 2-class classification head
model_fe = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased", num_labels=2
)
# Freeze all base model parameters
for param in model_fe.distilbert.parameters():
param.requires_grad = False
# Count trainable vs total parameters
trainable = sum(p.numel() for p in model_fe.parameters() if p.requires_grad)
total = sum(p.numel() for p in model_fe.parameters())
print(f"Feature Extraction - Trainable: {trainable:,} / Total: {total:,} ({100*trainable/total:.4f}%)")What does this code do?
The for param in model_fe.distilbert.parameters() loop walks through every weight in the DistilBERT transformer layers and sets requires_grad = False. This tells PyTorch to skip those parameters during backpropagation. They are frozen.
The classification layers that sit on top of the frozen base remain trainable. The final print statement shows exactly how many parameters are trainable versus how many are locked, so you can see the dramatic difference.
- Run the cell. You should see a printed line showing that only a tiny fraction of the model's ~66 million total parameters are trainable.
Why so few trainable parameters?
DistilBERT's classification model adds a pre_classifier layer and a classifier layer on top of the frozen transformer. These layers are the only ones learning from your movie review data. Everything else is locked to the knowledge DistilBERT already learned during pretraining.
Seeing all parameters as trainable?
Make sure the freeze loop targets model_fe.distilbert.parameters(), not model_fe.parameters(). The second form freezes everything, including the classification head.
Help me debug my parameter freezing code for DistilBERT.
Train and evaluate the frozen model
- Configure the training hyperparameters and the Trainer by adding this code in a new cell:
# Configure training for 2 epochs with fp16 precision
training_args_fe = TrainingArguments(
output_dir="./feat-extract",
num_train_epochs=2,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=2e-5,
fp16=True,
eval_strategy="epoch",
logging_steps=50,
report_to="none",
)
# Set up Trainer with frozen model and tokenized data
trainer_fe = Trainer(
model=model_fe,
args=training_args_fe,
train_dataset=tokenized_train,
eval_dataset=tokenized_test,
processing_class=tokenizer,
compute_metrics=compute_metrics,
)What does this code do?
TrainingArguments defines how training runs. fp16=True uses half-precision floats to speed things up on the T4 GPU. eval_strategy="epoch" tells the Trainer to evaluate accuracy after every full pass through the training data.
The Trainer wraps everything together: the frozen model, the training config, the tokenized datasets, and the accuracy metric. It handles the training loop so you do not have to write one manually.
- Run the cell. It completes without errors. The next cell kicks off training.
Before you run the training cell, think about this: you are training a model where only a tiny fraction of the parameters can update. Do you expect it to classify movie reviews accurately, or will such a small number of learnable weights hold it back?
- Train the model, evaluate it, and store the result by adding this code in a new cell:
# Train, evaluate, and store the accuracy
trainer_fe.train()
fe_metrics = trainer_fe.evaluate()
results["Feature Extraction"] = {
"trainable": trainable,
"total": total,
"accuracy": fe_metrics["eval_accuracy"],
}
print(f"Feature Extraction Accuracy: {fe_metrics['eval_accuracy']:.4f}")What does this code do?
trainer_fe.train() runs the full training loop for 2 epochs. trainer_fe.evaluate() runs the model on the 500 test reviews and returns the accuracy.
The result is saved into the results dictionary with the trainable count, total count, and accuracy. You will add entries for every fine-tuning method, and by the end of the project this dictionary powers a comparison table.
- Run the cell. You should see training loss decreasing over 2 epochs, followed by the Feature Extraction accuracy printed at the bottom.
Even with such a small number of trainable parameters, the model achieves a measurable accuracy score. The pretrained DistilBERT layers are doing the heavy lifting as a fixed feature extractor. The classification head just learns which features map to positive and negative sentiment.
Training is extremely slow or stuck?
Check that your Colab runtime is set to T4 GPU. Go to Runtime > Change runtime type and confirm T4 GPU is selected. If it shows CPU, switch to GPU and re-run all cells from the top.
If you see a CUDA out-of-memory error, go to Runtime > Restart session and re-run all cells. This clears GPU memory from any previous runs.
Help me troubleshoot slow training in Google Colab.
✔️ Awesome, I've got everything!
Great. Double check you have four cells in your notebook (including the setup cell from Step 1) and that each one shows a green checkmark.
ⓧ I'd like to double check the full code
Scroll through your notebook and confirm you have these cells in order:
- Cell 1 (Setup): Installs libraries, imports torch, transformers, peft, trl, and datasets, and prints version numbers and GPU info.
- Cell 2 (Data loading): Loads the tokenizer and IMDb dataset, creates small_train (2,000 samples) and small_test (500 samples), and tokenizes both into tokenized_train and tokenized_test.
- Cell 3 (Metrics): Loads the accuracy metric, defines compute_metrics, and creates an empty results dictionary.
- Cell 4 (Model + freeze): Loads model_fe, freezes model_fe.distilbert parameters, and prints the trainable count.
- Cell 5 (Trainer config): Creates training_args_fe and trainer_fe.
- Cell 6 (Train + evaluate): Runs trainer_fe.train(), evaluates, stores the result in results["Feature Extraction"], and prints the accuracy.
You have your Feature Extraction baseline locked in. In the next step, you will unlock every single parameter in the model and see how accuracy and trainable parameter count change when nothing is frozen.
Full Fine-Tuning - Unlock Everything
Your Feature Extraction model gave you a working baseline with roughly 1,500 trainable parameters. It trained quickly because the entire pretrained backbone stayed frozen. But those frozen weights never saw a single movie review during training.
Full Fine-Tuning takes the opposite approach. By unfreezing every parameter, the entire model adapts its language understanding to your specific task. In this step, you will unlock all of DistilBERT's weights and compare the results to your frozen baseline.
In this step, get ready to:
- Load a fresh DistilBERT model with all parameters trainable.
- Train for 2 epochs and compare accuracy to Feature Extraction.
Load a fresh model with all parameters unlocked
In the previous step, you froze the base model and only trained the classification head. This time, you will load a completely fresh copy of DistilBERT and leave every parameter trainable.
- Create a new code cell in your notebook and paste the following code:
# Cell 4: Full Fine-Tuning - All parameters trainable
model_full = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased", num_labels=2
)
trainable_full = sum(p.numel() for p in model_full.parameters() if p.requires_grad)
total_full = sum(p.numel() for p in model_full.parameters())
print(f"Full Fine-Tuning - Trainable: {trainable_full:,} / Total: {total_full:,} ({100*trainable_full/total_full:.4f}%)")What does this code do?
Notice what is missing compared to Cell 3. There is no loop setting requires_grad = False. When you load a model from Hugging Face, every parameter defaults to trainable.
- model_full is a brand-new copy of DistilBERT. No weights carry over from model_fe.
- trainable_full counts every parameter with requires_grad=True. Since nothing was frozen, this equals the total.
- Run the cell.
- You should see a line showing roughly 66 million trainable parameters out of 66 million total, at 100%.
Why is this number so different?
Feature Extraction trained roughly 1,500 parameters. Full Fine-Tuning trains roughly 66 million. That is a 44,000x increase in trainable parameters from a single change: not freezing the backbone.
Seeing an error when loading the model?
- Make sure you ran Cells 1 and 2 earlier in this session. The imports for AutoModelForSequenceClassification and TrainingArguments come from Cell 3.
- If your Colab runtime disconnected, reconnect and re-run all previous cells in order before running this one.
Still stuck? Help me debug a model loading error in my Full Fine-Tuning cell.
Train, evaluate, and compare to Feature Extraction
You now have 66 million trainable parameters instead of 1,500. Before you run training, think about what that means for accuracy.
Do you think Full Fine-Tuning will beat the Feature Extraction accuracy you saw earlier? By how much?
- Create another new code cell directly below and paste the following code:
training_args_full = TrainingArguments(
output_dir="./full-ft",
num_train_epochs=2,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=2e-5,
fp16=True,
eval_strategy="epoch",
logging_steps=50,
report_to="none",
)
trainer_full = Trainer(
model=model_full,
args=training_args_full,
train_dataset=tokenized_train,
eval_dataset=tokenized_test,
processing_class=tokenizer,
compute_metrics=compute_metrics,
)
trainer_full.train()
full_metrics = trainer_full.evaluate()
results["Full Fine-Tuning"] = {
"trainable": trainable_full,
"total": total_full,
"accuracy": full_metrics["eval_accuracy"],
}
print(f"Full Fine-Tuning Accuracy: {full_metrics['eval_accuracy']:.4f}")What does this code do?
- training_args_full uses the same hyperparameters as Feature Extraction (same learning rate, batch size, epochs). The only difference is output_dir="./full-ft" so the checkpoints go to a separate folder.
- trainer_full reuses the same tokenized datasets and compute_metrics function from Cell 2. The only change is the model itself.
- The accuracy result is stored in the results dictionary under the key "Full Fine-Tuning" so you can compare all methods at the end of the project.
Why keep the same hyperparameters?
Using identical training settings isolates the one variable you changed: which parameters are trainable. Any accuracy difference you see comes purely from unlocking the backbone, not from tweaking the learning rate or training longer.
- Run the cell. Training takes roughly 2-3 minutes on a T4 GPU.
- Watch the training log. You should see loss values printed every 50 steps, with evaluation accuracy reported at the end of each epoch.
- After training completes, the final line prints the Full Fine-Tuning accuracy. Compare it to your Feature Extraction accuracy from earlier.
Full Fine-Tuning accuracy should be noticeably higher than your Feature Extraction result. Unlocking all 66 million parameters let the model adapt its understanding of language to movie review sentiment, not just learn a simple decision boundary on top of frozen features.
Training seems stuck or very slow?
- Confirm your runtime is using a GPU. Go to **Runtime** > **Change runtime type** and verify **T4 GPU** is selected.
- Full Fine-Tuning trains 44,000x more parameters than Feature Extraction, so it is expected to take longer. If each step takes more than 10 seconds, your runtime may have fallen back to CPU.
- If you see an out-of-memory error, your Colab session may have limited VRAM. Try restarting the runtime with **Runtime** > **Restart session**, then re-run all cells from Cell 1.
Need more help? Help me debug slow training in Google Colab.
✔️ Awesome, I've got everything!
Great. Double check that both cells ran without errors and that you can see the Full Fine-Tuning accuracy printed.
ⓧ I'd like to double check the full code
Scroll up and confirm your two new cells match what you built in this step, in order:
- First cell: loads model_full with from_pretrained, counts trainable parameters, and prints the trainable/total line.
- Second cell: creates training_args_full with output_dir="./full-ft", sets up trainer_full, calls .train() and .evaluate(), stores results, and prints the accuracy.
Full Fine-Tuning gave better accuracy, but it came at a cost: 66 million trainable parameters, more memory, and longer training time. Next up, you will apply LoRA adapters to get similar accuracy with a tiny fraction of the parameters.
LoRA Fine-Tuning - The Best of Both Worlds
Your Full Fine-Tuning experiment proved that unlocking all parameters improves accuracy. But it came at a cost: training 66 million parameters takes more memory and more time.
LoRA (Low-Rank Adaptation) offers a middle ground. Instead of updating every weight in the model, LoRA injects small trainable matrices into specific layers while keeping the original weights frozen. In this step, you will apply LoRA adapters to the same DistilBERT model and see how it stacks up against both previous methods.
In this step, get ready to:
- Wrap a fresh DistilBERT model with LoRA adapters using PEFT.
- Train and evaluate the LoRA model on the same IMDb dataset.
- Print a three-way comparison table showing how LoRA compares to Feature Extraction and Full Fine-Tuning.
Load DistilBERT and apply LoRA adapters
LoRA works by inserting small, low-rank matrices alongside specific layers in the model. The original weights stay frozen, and only these small adapter matrices get updated during training.
- Create a new cell in your notebook. Load a fresh DistilBERT model, configure LoRA, and wrap the model by pasting this code:
# Import LoRA utilities from the PEFT library
from peft import LoraConfig, get_peft_model, TaskType
# Load a fresh DistilBERT model (no frozen layers this time)
model_lora = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased", num_labels=2
)
# Configure LoRA: small rank-8 matrices injected into attention layers
lora_config = LoraConfig(
r=8,
lora_alpha=16,
lora_dropout=0.1,
target_modules=["q_lin", "v_lin"],
task_type=TaskType.SEQ_CLS,
)
# Wrap the model with LoRA adapters and print what is trainable
model_lora = get_peft_model(model_lora, lora_config)
model_lora.print_trainable_parameters()What does this code do?
LoraConfig defines how the LoRA adapters are structured. r=8 sets the rank of the adapter matrices. A lower rank means fewer trainable parameters.
target_modules=["q_lin", "v_lin"] tells PEFT which layers to inject adapters into. These are DistilBERT's query and value attention projections. get_peft_model wraps the base model with these adapters, freezing the original weights and making only the adapter matrices trainable.
- Run the cell.
You should see a line like trainable params: ... || all params: ... || trainable%: ... showing that only a fraction of the total parameters are trainable. Compare that to the 66 million you trained in Full Fine-Tuning.
Seeing an import error?
Make sure your Cell 1 (the one with !pip install) ran successfully. If the peft import fails, try restarting the runtime and re-running all cells from the top.
Help me debug a PEFT import error in my Colab notebook.
Why LoRA over other PEFT methods?
PEFT is an umbrella category that includes several parameter-efficient techniques: prompt tuning, prefix tuning, IA3, and more. LoRA is the most widely used PEFT method in industry because it is simple to apply, adds minimal inference latency, and the adapters can be merged back into the base model after training.
Train and evaluate the LoRA model
The training setup looks almost identical to Full Fine-Tuning, with one key difference: the learning rate. LoRA adapters benefit from a higher learning rate because the adapter matrices are initialized near zero and need larger updates to learn effectively.
- Create a new cell. Set up the Trainer and run training by pasting this code:
# Use a 10x higher learning rate for LoRA (2e-4 vs 2e-5)
training_args_lora = TrainingArguments(
output_dir="./lora-ft",
num_train_epochs=2,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=2e-4,
fp16=True,
eval_strategy="epoch",
logging_steps=50,
report_to="none",
)
trainer_lora = Trainer(
model=model_lora,
args=training_args_lora,
train_dataset=tokenized_train,
eval_dataset=tokenized_test,
processing_class=tokenizer,
compute_metrics=compute_metrics,
)
trainer_lora.train()
lora_metrics = trainer_lora.evaluate()What changed from Full Fine-Tuning?
The only differences are output_dir="./lora-ft" and learning_rate=2e-4 (10x higher than the 2e-5 used in Full Fine-Tuning). Everything else stays the same. The LoRA adapters handle the parameter-efficiency behind the scenes.
- Run the cell. You will see training loss printed every 50 steps, and an evaluation run at the end of each epoch.
Training should complete in about the same time as Feature Extraction. Notice how much faster this is than Full Fine-Tuning, since the optimizer only tracks gradients for the small adapter matrices.
- In the same cell or a new cell, store the LoRA results and print the accuracy by pasting this code:
# Store LoRA results for the comparison table
trainable_lora = sum(p.numel() for p in model_lora.parameters() if p.requires_grad)
total_lora = sum(p.numel() for p in model_lora.parameters())
results["LoRA"] = {
"trainable": trainable_lora,
"total": total_lora,
"accuracy": lora_metrics["eval_accuracy"],
}
print(f"LoRA Accuracy: {lora_metrics['eval_accuracy']:.4f}")What does this code do?
This counts the trainable and total parameters after LoRA wrapping, then stores the results in the same results dictionary you have been building across steps. The final print line shows the LoRA evaluation accuracy so you can compare it immediately against your earlier numbers.
- Run the cell. You should see the LoRA accuracy printed.
Accuracy seems low or training loss is not decreasing?
Double check that your learning rate is 2e-4 (not 2e-5). LoRA adapters start near zero and need the higher rate to learn meaningful updates. Also confirm that the model_lora variable was created by get_peft_model in the previous cell.
Help me debug my LoRA training accuracy.
Print the three-way comparison table
You now have accuracy numbers and parameter counts for all three methods sitting in your results dictionary. Time to see them side by side.
Before you run the next cell, think about what you expect. Do you think LoRA's accuracy will be closer to Feature Extraction's or to Full Fine-Tuning's? How many trainable parameters do you think LoRA used compared to 66 million?
- Create a new cell and print the comparison table by pasting this code:
# Print the three-way comparison table
print("\n" + "=" * 65)
print(f"{'Method':<22} {'Trainable':>12} {'% of Total':>12} {'Accuracy':>10}")
print("=" * 65)
for method, data in results.items():
pct = 100 * data["trainable"] / data["total"]
print(f"{method:<22} {data['trainable']:>12,} {pct:>11.4f}% {data['accuracy']:>9.4f}")
print("=" * 65)What does this code do?
This loops through every entry in your results dictionary and prints each method's trainable parameter count, its percentage of total parameters, and its evaluation accuracy. The formatted columns make it easy to compare all three at a glance.
- Run the cell.
You should see a table with three rows: Feature Extraction, Full Fine-Tuning, and LoRA. The LoRA row shows roughly 150K to 300K trainable parameters (less than 1% of the total), yet its accuracy is comparable to Full Fine-Tuning's 66 million. That gap is why LoRA has become the industry standard for fine-tuning.
Table only showing one or two rows?
The results dictionary is built across cells. If you restarted your runtime at any point, earlier cells need to be re-run so that the Feature Extraction and Full Fine-Tuning entries exist. Use **Runtime**, then **Run all** to re-execute everything from the top.
Help me figure out why my comparison table is missing rows.
✔️ Awesome, I've got everything!
Great. Double check you have five cells in your notebook so far: Setup, Dataset, Feature Extraction, Full Fine-Tuning, and LoRA (which may span two cells for the config and training). Each one should have run without errors.
ⓧ I'd like to double check the full code
Scroll through your notebook and confirm these cells exist in order, each with output below it:
- Cell 1 (Setup): Installs libraries and prints version numbers plus GPU info.
- Cell 2 (Dataset): Loads IMDb, tokenizes 2,000 train and 500 test samples, defines compute_metrics and results.
- Cell 3 (Feature Extraction): Freezes the backbone, trains only the head, prints trainable params and accuracy.
- Cell 4 (Full Fine-Tuning): Trains all ~66M parameters and prints accuracy.
- Cell 5 (LoRA): Imports PEFT, wraps the model with LoRA adapters, trains, evaluates, stores results, and prints the three-way comparison table.
You have now seen the full spectrum of classification fine-tuning: from training just the head, to training everything, to LoRA's efficient middle ground. Next up, you will connect these concepts to the way modern LLMs are actually fine-tuned in production, using SFTTrainer with LoRA to instruction-tune a language model.
SFT with LoRA - Instruction Fine-Tuning a Language Model
Your three-way comparison table tells a clear story. LoRA matches Full Fine-Tuning accuracy with a tiny fraction of the trainable parameters. But so far, you have only applied LoRA to a classification task.
In the real world, the most common use of LoRA is instruction-tuning a language model to follow prompts. SFT (Supervised Fine-Tuning) and Instruction Fine-Tuning are the same technique applied to chat or instruction data. This step connects the parameter-efficiency concepts you proved to modern LLM post-training workflows using TRL's SFTTrainer.
In this step, get ready to:
- Load a generative language model and its tokenizer.
- Train the model with LoRA adapters on a conversational dataset using SFTTrainer.
- Print a final summary table comparing all fine-tuning methods across the spectrum.
Load the generative model and tokenizer
Every model you have trained so far was a classification model. It takes text in and outputs a label (positive or negative). A causal language model is different. It takes text in and generates new text, word by word. That is the type of model behind every chatbot and AI assistant.
- Create a new cell in your notebook and load the model by pasting the following code:
# SFT with LoRA - Instruction Fine-Tuning a Language Model
from transformers import AutoModelForCausalLM, AutoTokenizer as AT
from peft import LoraConfig as LC
from trl import SFTTrainer, SFTConfig
# Load a small generative language model
sft_model_name = "Qwen/Qwen3-0.6B"
sft_tokenizer = AT.from_pretrained(sft_model_name)
sft_model = AutoModelForCausalLM.from_pretrained(
sft_model_name, torch_dtype=torch.float16, device_map="auto"
)What does this code do?
You are loading Qwen/Qwen3-0.6B, a 0.6 billion parameter generative model from Alibaba. At 0.6B parameters in float16, it uses roughly 1.2 GB of VRAM, leaving plenty of headroom on your T4's 15 GB.
The imports use aliases (AT for AutoTokenizer, LC for LoraConfig) because you already imported the un-aliased versions earlier for the classification tasks.
Notice AutoModelForCausalLM instead of AutoModelForSequenceClassification. This tells Hugging Face to load a text-generation model, not a classifier.
- Run the cell.
- You should see download progress bars as the model weights load, followed by a message confirming the model is on your GPU.
Seeing a download error?
Make sure you typed the model name exactly as "Qwen/Qwen3-0.6B" with a capital Q and the correct dash placement. Hugging Face model names are case-sensitive.
If your runtime has disconnected, reconnect it from Runtime > Reconnect and re-run your earlier cells first.
Help me debug my model loading error in Google Colab.
Configure LoRA and train with SFTTrainer
You need three ingredients to instruction-tune: a LoRA config tuned for generative models, a conversational dataset, and TRL's SFTTrainer to tie them together.
- Create a new cell and set up the LoRA config and dataset by pasting this code:
# Configure LoRA for causal language modeling
sft_lora_config = LC(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj"],
task_type="CAUSAL_LM",
)
# Load a small subset of conversational training data
capybara = load_dataset("trl-lib/Capybara", split="train")
capybara_small = capybara.shuffle(seed=42).select(range(1000))What changed from the classification LoRA config?
The task_type is now "CAUSAL_LM" instead of SEQ_CLS. The target_modules changed to "q_proj" and "v_proj" because Qwen uses different attention layer names than DistilBERT.
The r=16 (rank) is higher than the r=8 you used for classification. Generative tasks are more complex, so the adapter matrices need a bit more capacity.
The Capybara dataset contains multi-turn conversations formatted as instruction-response pairs. This is what makes this step instruction fine-tuning.
- Run the cell. You should see the Capybara dataset download and a confirmation that 1,000 samples were selected.
- Create another new cell and set up the SFTTrainer by pasting this code:
# Configure training hyperparameters for SFT
sft_config = SFTConfig(
output_dir="./sft-lora",
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=2e-4,
fp16=True,
logging_steps=10,
max_seq_length=512,
report_to="none",
)
# Combine the model, data, and LoRA config into one trainer
sft_trainer = SFTTrainer(
model=sft_model,
args=sft_config,
train_dataset=capybara_small,
peft_config=sft_lora_config,
)
# Start instruction fine-tuning
sft_trainer.train()What does this code do?
SFTConfig works like TrainingArguments but adds SFT-specific options like max_seq_length. The batch size is smaller (2 instead of 16) because generative models use much more memory per sample than classifiers.
gradient_accumulation_steps=4 simulates a larger effective batch size of 8 by accumulating gradients across 4 mini-batches before updating weights. This keeps memory low while maintaining training stability.
Notice that SFTTrainer takes peft_config directly. You do not need to call get_peft_model() yourself. TRL handles wrapping the model with LoRA adapters automatically.
- Run the cell. Training takes a few minutes on the T4 GPU.
- You should see training progress logged every 10 steps, with the loss value decreasing over time.
Seeing an out-of-memory error?
Your GPU memory may be full from the earlier training runs. Go to Runtime > Restart session, then re-run all your cells from the top. The restart frees all GPU memory.
Help me fix an out-of-memory error during SFT training in Colab.
Test generation and print the final summary
Training is done. Now for the payoff. You will ask your instruction-tuned model a question and see it generate a real answer.
Before you run the next cell, think about this: the model just trained on 1,000 conversational examples for a single epoch. Do you think that is enough for it to give a coherent answer about LoRA?
- Create a new cell and test the model by pasting this code:
# Switch to evaluation mode for generation
sft_model.eval()
# Build a test prompt using the model's chat template
test_prompt = "Explain what LoRA fine-tuning is in simple terms."
messages = [{"role": "user", "content": test_prompt}]
input_text = sft_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = sft_tokenizer(input_text, return_tensors="pt").to(sft_model.device)
# Generate a response from the fine-tuned model
with torch.no_grad():
outputs = sft_model.generate(
**inputs, max_new_tokens=150, do_sample=True, temperature=0.7
)
# Decode and print only the model's response
response = sft_tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(f"\nPrompt: {test_prompt}")
print(f"Response: {response}")What does this code do?
apply_chat_template() formats your prompt using the model's expected conversation structure (with role tags and special tokens). This is how chat models know where the user's message ends and the assistant's response should begin.
The decoding line slices off the input tokens with outputs[0][inputs["input_ids"].shape[-1]:] so you see only the model's generated response, not the prompt echoed back.
- Run the cell.
- You should see the model produce a coherent response explaining LoRA. The quality may vary since this is a small model with minimal training, but it should produce readable text.
Seeing garbled or empty output?
Make sure you ran the training cell (the one with sft_trainer.train()) before this generation cell. The model needs the trained LoRA weights loaded to generate meaningful text.
If the response is very short or repetitive, that is normal for a 0.6B model trained on only 1,000 samples for 1 epoch. The goal is to see it generate something coherent, not perfect.
Help me understand why my SFT model output looks wrong.
Now for the final piece. You will print a summary table that maps the entire fine-tuning spectrum you have explored across all five steps.
Before you run it, think about all five methods. Can you name what each one trains and which Hugging Face API it uses?
- Create a new cell and print the summary table by pasting this code:
# Final comparison of all fine-tuning methods
print("\n" + "=" * 90)
print("FINE-TUNING SPECTRUM SUMMARY")
print("=" * 90)
print(f"{'Method':<25} {'What is Trained':<25} {'Params %':<12} {'Use Case':<28}")
print("-" * 90)
print(f"{'Feature Extraction':<25} {'Classification head only':<25} {'<0.01%':<12} {'Quick baseline, limited data':<28}")
print(f"{'Full Fine-Tuning':<25} {'All model parameters':<25} {'100%':<12} {'Max accuracy, small models':<28}")
print(f"{'LoRA (PEFT)':<25} {'Low-rank adapter matrices':<25} {'<1%':<12} {'Production fine-tuning':<28}")
print(f"{'SFT + LoRA':<25} {'LoRA on causal LM':<25} {'<1%':<12} {'Instruction/chat tuning':<28}")
print(f"{'Instruction FT':<25} {'= SFT with instruction data':<25} {'varies':<12} {'Chat models, assistants':<28}")
print("=" * 90)
print("\nKey insight: SFT and Instruction Fine-Tuning are training METHODOLOGIES.")
print("Feature Extraction, Full FT, and LoRA are STRATEGIES for which parameters to update.")
print("You can combine them: e.g., SFT + LoRA = instruction-tune with parameter efficiency.")What does this table reveal?
The table splits fine-tuning into two dimensions. Feature Extraction, Full Fine-Tuning, and LoRA are strategies for which parameters to update. SFT and Instruction Fine-Tuning are methodologies for how you train (supervised on input-output pairs, using instruction-formatted data).
You can combine them freely. SFT + LoRA means you use the SFT methodology with the LoRA parameter strategy. This is exactly what you just did in this step.
- Run the cell.
- You should see a formatted table with five rows covering Feature Extraction, Full Fine-Tuning, LoRA (PEFT), SFT + LoRA, and Instruction FT. The key insight at the bottom ties the whole spectrum together.
✔️ Awesome, I've got everything!
Great. Double check your notebook is saved. You should have all seven cells (setup, data loading, Feature Extraction, Full Fine-Tuning, LoRA, SFT with LoRA, and the summary table) running without errors.
ⓧ I'd like to double check the full code
Your notebook is too long to show in one block. Scroll through your cells and confirm each one is present in this order:
- Cell 1: Setup (pip install, version prints, GPU check).
- Cell 2: Load and Tokenize IMDb Dataset.
- Cell 3: Feature Extraction (freeze base, train head only).
- Cell 4: Full Fine-Tuning (all parameters trainable).
- Cell 5: LoRA Fine-Tuning (PEFT adapters + three-way comparison table).
- Cell 6: SFT with LoRA. This includes the model/tokenizer load, the sft_lora_config, Capybara dataset, SFTTrainer setup, training, and test generation.
- Cell 7: Final Summary Table (the five-row spectrum comparison).
If you split Cell 6 across multiple cells during this step, that is perfectly fine. The important thing is that all the code is present and runs without errors.
You now have a complete notebook covering the full fine-tuning spectrum, from freezing everything to instruction-tuning a generative model. The summary table proves that these methods are not competing alternatives. They are building blocks you can mix and match.
Next up, your Secret Mission pushes LoRA even further with 4-bit quantization. You will see how QLoRA slashes memory usage so you can fine-tune models that would not fit on your GPU otherwise.
Secret mission
QLoRA - Quantize and Adapt
LoRA kept the original weights frozen but still loaded them in full precision. QLoRA goes further by shrinking those frozen weights to 4-bit, slashing memory so you can fine-tune models that would otherwise crash your GPU. Can you add QLoRA to your notebook and prove the memory savings?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely on Google Colab's free tier, so there are no ongoing costs. Decide whether to keep your notebook saved, let the runtime expire on its own, or close the tab entirely.
Resources you used:
- Google Colab notebook (your .ipynb file with all cells and outputs).
- Colab GPU runtime session (T4 GPU, ephemeral storage for downloaded models, datasets, and training output directories).
Keep everything running
No action needed right now. Choose this if you are still actively working in the notebook or plan to come back within the next hour.
- Your Colab runtime stays active until it idles out (roughly 90 minutes of inactivity) or hits the maximum session length of about 12 hours.
- To preserve your notebook permanently, click File then Save a copy in Drive. This saves the .ipynb file (including cell outputs) to your Google Drive.
- Downloaded models, datasets, and training output directories live only in the ephemeral runtime storage. They are deleted automatically when the runtime disconnects.
Pause - I'll come back to this later
Save your notebook, then let the runtime disconnect on its own. No charges accrue while the runtime is idle or after it disconnects.
- Click File then Save a copy in Drive (or Download then Download .ipynb to save locally).
- The Colab runtime will disconnect automatically after roughly 90 minutes of idle time or about 12 hours total.
- When you return, open the saved notebook and reconnect to a new GPU runtime. You will need to re-run the cells from the top since all runtime state (models, variables, training outputs) is cleared on disconnect.
Delete - I don't want to use this again
Remove the notebook entirely. No persistent cloud resources were created during this project, so cleanup is straightforward.
- Close the Colab browser tab. The GPU runtime shuts down and all ephemeral storage (downloaded models, datasets, and training output directories like ./feat-extract, ./full-ft, ./lora-ft, ./sft-lora, and ./qlora-sft) is deleted automatically.
- If you saved a copy to Google Drive, open Google Drive, find the .ipynb file in your Colab Notebooks folder, right-click it, and select Move to trash.
Nice Work!
Nice Work!
You just built a complete fine-tuning lab from scratch. Your notebook now proves, with real numbers, how Feature Extraction, Full Fine-Tuning, and LoRA compare across trainable parameters, memory, and accuracy.
You've learned how to:
- Train a frozen feature extractor, fully fine-tuned model, and LoRA-adapted model on the same dataset and compare their trainable parameter counts and accuracy side by side.
- Use three Hugging Face APIs in a single notebook: Trainer for classification, PEFT's get_peft_model for LoRA adapters, and TRL's SFTTrainer for instruction-tuning a language model.
- Explain the full fine-tuning taxonomy: Feature Extraction freezes the backbone, Full Fine-Tuning trains everything, PEFT is the umbrella for parameter-efficient methods, LoRA is the most popular PEFT method, and SFT applied to instruction-formatted data is Instruction Fine-Tuning.
- Secret Mission: Applied QLoRA to push beyond float16 memory limits using 4-bit quantization with BitsAndBytesConfig.
Ready to quiz yourself?