Fine-Tune a Local LLM with SFT

Fine-tune Llama 3.1 8B on Dolly 15k and run it locally via Ollama.

Introduction

30 Second Summary

Every AI chatbot you talk to started as a raw language model that could barely follow instructions. The secret sauce that turns a base model into a helpful assistant is a process called fine-tuning.

In this project, you will fine-tune your own large language model using Supervised Fine-Tuning (SFT) and run it locally as an interactive chatbot. No expensive GPU hardware required.

What You'll Build

A custom fine-tuned language model trained on 15,000 instruction-following examples, exported and deployed locally as a working chatbot via Ollama.

By the end of this project, you'll have:

  • A Google Colab notebook with the complete SFT training pipeline using Unsloth and a free T4 GPU.
  • A LoRA adapter (a small ~100MB file) that specializes an 8-billion parameter model using QLoRA 4-bit quantization.
  • A GGUF-quantized model exported for efficient local inference and running as an interactive chatbot via Ollama.
  • Secret Mission: Fine-tune on a custom dataset you create yourself, teaching the model a unique persona or specialized domain knowledge.

Are there any prerequisites?

Basic Python familiarity is all you need. You do not need a GPU or any special local hardware. All training happens for free in Google Colab.

Before We Start

You are about to fine-tune a pre-trained large language model to follow instructions better using Supervised Fine-Tuning (SFT). This is the same technique used to turn raw pre-trained models into helpful assistants like ChatGPT.

To train a model like this, you need a GPU-powered environment and the right libraries. In this step, you will set up Google Colab with a free T4 GPU and install Unsloth, the fine-tuning framework that makes training fast and memory-efficient.

In this step, get ready to:
  • Open Google Colab and switch the runtime to a T4 GPU.
  • Install the Unsloth library and its dependencies.
  • Verify that your GPU and libraries are ready to go.
Open Google Colab and enable GPU

Google Colab gives you free access to a GPU in your browser. No local hardware or drivers required.

  • Navigate to colab.research.google.com in your browser.
  • Sign in with your Google account if prompted.
  • Click New notebook to create a fresh notebook.

By default, Colab runs on a CPU. You need to switch to a GPU runtime for model training.

  • Click Runtime in the top menu bar.
  • Click Change runtime type.
  • In the hardware accelerator dropdown, select T4 GPU.
  • Click Save.

Now verify that Colab actually assigned you a GPU.

  • Confirm your GPU is active by typing this command in the first code cell and pressing Shift+Enter to run it:
!nvidia-smi

What does this command do?

The ! prefix tells Colab to run this as a shell command instead of Python code.

nvidia-smi prints a status table showing your GPU model, driver version, and available memory. You should see Tesla T4 and approximately 15GB of total memory listed.

Don't see a GPU listed?

If the output says no GPU is detected, double-check that you selected T4 GPU in the runtime settings and clicked Save.

Colab's free GPUs are shared. If all T4s are busy, try again in a few minutes or during off-peak hours.

help me get a T4 GPU working in Google Colab

Install Unsloth and dependencies

Unsloth is the fine-tuning framework that makes training fast and memory-efficient on a single GPU. Installing it also pulls in the other libraries you need: TRL, datasets, and bitsandbytes.

  • Click the + Code button to create a new code cell below the nvidia-smi cell.
  • Install Unsloth by running this command:
!pip install unsloth

What does this install?

pip install unsloth installs the Unsloth library along with its key dependencies. These include TRL (the training library), datasets (for loading training data from Hugging Face), and bitsandbytes (for 4-bit quantization).

  • Wait for the installation to complete. This takes 1-2 minutes.
  • If Colab displays a Restart Runtime button after installation, click it to restart.

Seeing dependency errors?

Dependency conflicts can appear if Colab pre-installs an older version of a package. Restarting the runtime usually resolves this.

If errors persist after restarting, try running the install cell again.

help me fix Unsloth installation errors in Google Colab

Verify the installation

A quick import test confirms that Unsloth is installed correctly and can see your GPU.

  • Create a new code cell and verify the installation by running:
from unsloth import FastLanguageModel
import torch

# Check GPU memory
gpu_memory = round(torch.cuda.get_device_properties(0).total_mem / 1e9, 1)
print(f"GPU memory available: {gpu_memory} GB")

What should I see?

If everything is working, the cell runs without errors and prints approximately 15.0 GB of GPU memory.

The import of FastLanguageModel confirms Unsloth is installed. The memory check confirms your T4 GPU is active and ready for training.

  • Confirm the output shows around 15 GB of memory with no error messages.

Seeing an import error?

If you see ModuleNotFoundError, the runtime may not have restarted after installation. Click Runtime then Restart session, then re-run this cell.

help me fix Unsloth import errors in Google Colab

✔️ Awesome, I've got everything!

Great. Double check that you have saved your notebook. Colab autosaves, but you can press Ctrl+S to be sure.

ⓧ I'd like to double check the full code

Here is what your first cell should contain. The other cells (nvidia-smi and the import check) are verification steps that you can leave in or delete.

# Cell 1: Install dependencies
# !pip install unsloth
# After installation, restart the runtime if prompted

Your GPU environment is ready and Unsloth is installed. Next up, you will load a large language model and attach LoRA adapters so you can fine-tune it efficiently.

Load the Base Model with 4-bit Quantization

Your environment is set up with GPU access and all the right libraries installed. Now it's time to load the actual language model you'll be fine-tuning.

Loading a full 8-billion parameter model would normally require ~16GB of VRAM. With 4-bit quantization, you can compress those weights down to ~5GB. That leaves plenty of room on the T4 GPU for training.

In this step, get ready to:
  • Load the base Llama 3.1 8B model using 4-bit quantization.
  • Attach LoRA adapters for parameter-efficient training.
  • Verify that only ~1% of parameters are trainable.
Load Llama 3.1 8B in 4-bit

The model you'll fine-tune is Meta's Llama 3.1 8B Instruct. Unsloth provides a pre-quantized version on Hugging Face that downloads faster and loads directly in 4-bit precision.

  • In your Colab notebook, create a new code cell below your installation cell and paste the following code:
# Cell 2: Imports and model loading
from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

max_seq_length = 2048
dtype = None  # Auto-detect (float16 for T4)
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

What does this code do?

  • The imports bring in FastLanguageModel from Unsloth, plus SFTTrainer and SFTConfig from TRL (which you'll use later for training).
  • max_seq_length=2048 sets the maximum number of tokens the model handles at once.
  • load_in_4bit=True compresses the model weights from 16-bit to 4-bit, cutting VRAM usage by roughly 4x.
  • FastLanguageModel.from_pretrained downloads the pre-quantized model from Hugging Face and returns the model and tokenizer as a tuple.
  • Run the cell by pressing Shift+Enter.
  • Wait for the model to download. You'll see a progress bar as the model weights download, followed by a message like "Unsloth patched 32 layers" confirming the model loaded successfully.

How long does this take?

The first time you run this cell, it downloads approximately 4-5GB of model weights from Hugging Face. This typically takes 1-3 minutes depending on Colab's network speed. Subsequent runs in the same session use cached files and load much faster.

Seeing a download error or timeout?

  • Check that your Colab runtime is still connected. If it disconnected, click Reconnect in the top right.
  • If the download gets stuck at a high percentage, try restarting the runtime (Runtime > Restart runtime) and running the cells again.
  • If you see a CUDA out-of-memory error, make sure your runtime type is set to T4 GPU. Go to Runtime > Change runtime type and confirm T4 GPU is selected.

Still stuck? Help me debug my model loading error in Google Colab with Unsloth

Attach LoRA adapters

With the base model loaded, you need to add the trainable LoRA adapters. These are small layers inserted into the model that get updated during training, while the original 8 billion parameters stay frozen.

  • Create a new code cell below Cell 2 and paste the following code:
# Cell 3: Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    max_seq_length=max_seq_length,
)

# Print trainable parameters
model.print_trainable_parameters()

What does this code do?

  • r=16 sets the rank of the LoRA adapters. Higher rank means more capacity to learn, but uses more memory.
  • target_modules lists all the linear layers inside the model where LoRA adapters get inserted. This covers the attention layers (q, k, v, o) and feed-forward layers (gate, up, down).
  • use_gradient_checkpointing="unsloth" reduces VRAM usage by about 30% during training at the cost of slightly slower computation.
  • model.print_trainable_parameters() outputs the number of parameters that will be updated during training compared to the total.
  • Run the cell by pressing Shift+Enter.
  • You should see output showing the trainable parameter count. It will look something like: trainable params: 41,943,040 || all params: 8,030,261,248 || trainable%: 0.5222.

Not seeing the parameter count output?

  • Make sure you ran Cell 2 first. Cell 3 depends on the model variable created in the previous cell.
  • If you see a NameError mentioning model is not defined, go back and run Cell 2 again.

Still having trouble? Help me fix my LoRA adapter setup in Unsloth

Inspect the model

The output from print_trainable_parameters() tells you how efficient QLoRA is. Out of roughly 8 billion total parameters, only about 40 million are trainable.

What do these numbers mean?

The trainable percentage (around 0.5%) means you're only updating a tiny fraction of the model during training. The remaining 99.5% of the model stays frozen in its original state.

This is the magic of LoRA. You get the benefits of fine-tuning a massive model without needing the VRAM or compute to update all 8 billion parameters. The adapter file you produce at the end will be about 100MB instead of 16GB.

Why target all linear layers?

The target_modules list covers every major computation point in each transformer layer. Targeting all of them gives the model maximum flexibility to adapt its behavior during fine-tuning.

Some tutorials only target the attention layers (q, k, v, o). Including the feed-forward layers (gate, up, down) as well is Unsloth's recommended configuration for better results.

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook so far.

ⓧ I'd like to double check the full code

Here's everything you should have in your notebook up to this point:

# Cell 1: Install dependencies
# !pip install unsloth
# After installation, restart the runtime if prompted

# Cell 2: Imports and model loading
from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

max_seq_length = 2048
dtype = None  # Auto-detect (float16 for T4)
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

# Cell 3: Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    max_seq_length=max_seq_length,
)

# Print trainable parameters
model.print_trainable_parameters()

Your model is loaded and configured with LoRA adapters. Next up, you'll prepare the training dataset that teaches the model how to follow instructions.

Prepare the Training Dataset

Your model is loaded and ready to learn. The next question is: learn from what?

A fine-tuned model is only as good as the data you train it on. In this step, you will load a high-quality instruction-following dataset and format it into the exact chat template your model expects. This is how the model learns to respond like a helpful assistant.

In this step, get ready to:
  • Load the Databricks Dolly 15k dataset from Hugging Face.
  • Format the dataset into the Llama 3.1 chat template with system, user, and assistant roles.
  • Verify the formatting by inspecting a sample example.
Load the Databricks Dolly 15k dataset

The model needs real examples of humans asking questions and getting helpful answers. The Databricks Dolly 15k dataset provides exactly that.

  • In your Colab notebook, create a new code cell below Cell 3.
  • Load the dataset by adding the following code to your new cell:
# Cell 4: Load and format dataset
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")

What is the Databricks Dolly 15k dataset?

This is an open-source dataset of over 15,000 instruction-following examples written entirely by humans. It covers eight categories: brainstorming, classification, closed QA, generation, information extraction, open QA, summarization, and creative writing.

Each example has three fields: an instruction (the question or task), a context (optional reference text), and a response (the human-written answer). The load_dataset function was already imported in Cell 2, so you can use it directly.

  • Run the cell. You should see a download progress bar completing as the dataset loads from Hugging Face.

Seeing a connection error?

Make sure your Colab runtime is still active. If the session timed out, reconnect by clicking **Connect** in the top right, then re-run all previous cells before this one.

help me fix a connection error when loading a Hugging Face dataset in Google Colab

Format the dataset for chat-style SFT

The raw dataset has separate fields for instruction, context, and response. Your model expects a specific chat format with role-tagged messages. You need a function that bridges this gap.

  • In the same cell, add the following formatting function and apply it to the dataset:
def format_dolly(example):
    """Format a Dolly example into Llama 3.1 chat template."""
    instruction = example["instruction"]
    context = example["context"]
    response = example["response"]

    if context:
        user_message = f"{instruction}\n\nContext: {context}"
    else:
        user_message = instruction

    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_message},
        {"role": "assistant", "content": response},
    ]

    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
    return {"text": text}

dataset = dataset.map(format_dolly)

# Print a sample
print(dataset[0]["text"][:500])

What does this code do?

  • The format_dolly function takes each dataset example and builds a list of messages with system, user, and assistant roles.
  • If the example has a context field (like a Wikipedia paragraph), it gets appended to the user message. Otherwise, just the instruction is used.
  • The tokenizer.apply_chat_template() method converts the message list into the exact token format Llama 3.1 expects, including special tokens like role headers.
  • The dataset.map(format_dolly) call applies this function to all 15,000 examples, creating a new text field on each one.
  • Run the cell. The mapping takes a few seconds to process all 15,000 examples, then prints the first 500 characters of a formatted sample.
Inspect a formatted example

The printed output shows you exactly what the model will see during training. Look for the special chat template tokens in the output.

What should I see?

Your output should contain special tokens like <|begin_of_text|> at the start, <|start_header_id|> before each role, and the role names system, user, and assistant tagged within the headers.

These tokens tell the model where each speaker's turn begins and ends. The system message sets the persona, the user message contains the instruction, and the assistant message contains the target response the model should learn to generate.

Seeing a KeyError or empty output?

Check that you typed the field names exactly as instruction, context, and response (all lowercase). These are the column names in the Dolly dataset.

help me fix a KeyError when formatting the Dolly 15k dataset for fine-tuning

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook, then move on to training.

ⓧ I'd like to double check the full code

Here is the complete code for all cells up to this point:

# Cell 1: Install dependencies
# !pip install unsloth
# After installation, restart the runtime if prompted

# Cell 2: Imports and model loading
from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

max_seq_length = 2048
dtype = None  # Auto-detect (float16 for T4)
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

# Cell 3: Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    max_seq_length=max_seq_length,
)

# Print trainable parameters
model.print_trainable_parameters()

# Cell 4: Load and format dataset
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")

def format_dolly(example):
    """Format a Dolly example into Llama 3.1 chat template."""
    instruction = example["instruction"]
    context = example["context"]
    response = example["response"]

    if context:
        user_message = f"{instruction}\n\nContext: {context}"
    else:
        user_message = instruction

    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_message},
        {"role": "assistant", "content": response},
    ]

    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
    return {"text": text}

dataset = dataset.map(format_dolly)

# Print a sample
print(dataset[0]["text"][:500])

Your dataset is ready for training. Next up, you will configure the training loop and watch the model learn from these 15,000 examples.

Configure and Run Training

Your dataset is ready and your model has trainable LoRA adapters attached. Now it is time to actually train the model on those 15,000 instruction-following examples.

Training a model means feeding it data in small batches, calculating how wrong its predictions are (the loss), and nudging the weights to reduce that error. You will configure the training hyperparameters, kick off the run, and then test the result.

In this step, get ready to:
  • Configure the SFTTrainer with optimized training hyperparameters.
  • Run 60 training steps and observe the loss decreasing.
  • Test your fine-tuned model by generating a response to a prompt.
Set up training arguments

The SFTConfig object holds all the hyperparameters that control how training runs. These include batch size, learning rate, how many steps to train, and which optimizer to use.

  • Create a new code cell in your Colab notebook and set up the trainer by pasting the following code:
# Cell 5: Configure training
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=SFTConfig(
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        output_dir="outputs",
        optim="adamw_8bit",
        seed=3407,
    ),
)

What does this code do?

This creates the SFTTrainer with your model, dataset, and a set of training hyperparameters bundled inside SFTConfig.

  • per_device_train_batch_size=2 processes 2 examples at a time. gradient_accumulation_steps=4 simulates a larger effective batch size of 8 without needing extra VRAM.
  • max_steps=60 caps training at 60 optimizer steps. This keeps runtime under 10 minutes on a T4 GPU.
  • learning_rate=2e-4 is the standard rate for QLoRA fine-tuning. optim="adamw_8bit" uses a memory-efficient 8-bit optimizer.
  • The fp16/bf16 lines auto-detect which precision your GPU supports. The T4 does not support bfloat16 natively, so it will use fp16.
  • logging_steps=1 prints the loss after every single step so you can watch it decrease in real time.

Why adamw_8bit instead of standard AdamW?

Standard AdamW stores two extra copies of every parameter (momentum and variance), which doubles memory usage. The 8-bit version quantizes these optimizer states to 8 bits, cutting their memory footprint while maintaining nearly identical training quality.

  • Run the cell. You should see no output errors. The trainer is now ready to start training.

Seeing an error about missing columns?

Make sure your dataset has a text column. This was created when you ran dataset.map(format_dolly) in the previous step. If the column is missing, re-run Cell 4 first.

Help me fix this SFTTrainer configuration error.

Run training

This is the moment of truth. One line of code starts the actual fine-tuning process on your GPU.

  • Create a new code cell and start training by running:
# Cell 6: Train
trainer.train()

What should I see?

A progress bar appears showing steps 1 through 60. After each step, you will see a log line with the current loss value. The loss should start somewhere around 1.0-2.0 and gradually decrease as training progresses.

This takes approximately 5-10 minutes on a T4 GPU. Do not close the tab or interrupt the runtime while it is running.

  • Wait for training to complete. When it finishes, you will see a summary table showing the final training loss and runtime.

Training seems stuck or taking too long?

Confirm your runtime is using a T4 GPU by going to Runtime > Change runtime type and checking the hardware accelerator. If it says CPU or None, switch to T4 GPU and re-run all cells from the beginning.

Help me troubleshoot why training is slow or stuck.

Test the fine-tuned model

Training is done. Now you can switch the model into inference mode and ask it a question to see how it responds after fine-tuning.

  • Create a new code cell and test your model by running the following code:
# Cell 7: Test inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are three creative uses for old newspapers?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

What does this code do?

  • FastLanguageModel.for_inference(model) switches the model out of training mode into optimized inference mode. Unsloth makes this 2x faster than standard inference.
  • The messages list uses the same chat template format you trained on. apply_chat_template converts it into the token IDs the model expects.
  • model.generate() runs the actual text generation. max_new_tokens=256 caps the response length. temperature=0.7 adds a bit of randomness so responses are not repetitive.
  • Run the cell. You should see the model print a coherent, multi-point response about creative uses for old newspapers.

Getting garbled or empty output?

Make sure you ran FastLanguageModel.for_inference(model) before calling model.generate(). Without this, the model may still be in training mode and produce unexpected results.

Also confirm that Cell 6 (trainer.train()) completed successfully before running this cell.

Help me debug my model's inference output.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook so nothing is lost.

ⓧ I'd like to double check the full code

Here is the complete notebook code up to this point:

# Cell 1: Install dependencies
# !pip install unsloth
# After installation, restart the runtime if prompted

# Cell 2: Imports and model loading
from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

max_seq_length = 2048
dtype = None  # Auto-detect (float16 for T4)
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

# Cell 3: Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    max_seq_length=max_seq_length,
)

# Print trainable parameters
model.print_trainable_parameters()

# Cell 4: Load and format dataset
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")

def format_dolly(example):
    """Format a Dolly example into Llama 3.1 chat template."""
    instruction = example["instruction"]
    context = example["context"]
    response = example["response"]

    if context:
        user_message = f"{instruction}\n\nContext: {context}"
    else:
        user_message = instruction

    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_message},
        {"role": "assistant", "content": response},
    ]

    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
    return {"text": text}

dataset = dataset.map(format_dolly)

# Print a sample
print(dataset[0]["text"][:500])

# Cell 5: Configure training
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=SFTConfig(
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        output_dir="outputs",
        optim="adamw_8bit",
        seed=3407,
    ),
)

# Cell 6: Train
trainer.train()

# Cell 7: Test inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are three creative uses for old newspapers?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Your model is now fine-tuned and responding to prompts. Next up, you will export it to a format that runs locally on your own machine and chat with it through Ollama.

Export to GGUF and Run with Ollama

Your model has been trained and you have already seen it respond to a test prompt. The weights you care about exist only inside Colab's temporary memory right now.

In this step, you will save the trained adapter, export the full model in a format optimized for local inference, and launch it as an interactive chatbot using Ollama.

In this step, get ready to:
  • Save the fine-tuned LoRA adapter as a reusable file.
  • Export the model to GGUF format for local inference.
  • Run the model as an interactive chatbot with Ollama.
Save the LoRA adapter

The LoRA adapter is a small set of weights (~100MB) that captures everything your model learned during training. Saving it separately means you can reload, share, or redeploy the fine-tuning later without retraining.

  • Save the LoRA adapter and tokenizer by adding a new code cell (Cell 8) and running this code:
# Cell 8: Save LoRA adapter
model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")

What does this code do?

  • model.save_pretrained("lora_model") writes only the trained LoRA weights to a folder called lora_model. This is much smaller than the full 8B model because it only contains the delta weights from training.
  • tokenizer.save_pretrained("lora_model") saves the tokenizer configuration alongside the adapter so you can reload both together later.
  • Verify the adapter was saved by running this in a new cell:
!ls lora_model

You should see files like adapter_config.json, adapter_model.safetensors, and tokenizer files listed.

Not seeing the lora_model folder?

  • Make sure you ran Cell 8 after training completed. The model must be in memory for save_pretrained to work.
  • Check that you typed the folder name "lora_model" exactly as shown, with quotes.

Help me debug why my LoRA adapter didn't save in Colab.

Export to GGUF format

GGUF is the standard file format for running LLMs locally with tools like Ollama and llama.cpp. The export merges your LoRA weights into the base model and quantizes everything into a single portable file.

  • Export the merged model to GGUF with Q8_0 quantization by adding a new cell (Cell 9) and running:
# Cell 9: Export to GGUF (Q8_0)
model.save_pretrained_gguf("model_gguf", tokenizer, quantization_method="q8_0")

What does this code do?

  • save_pretrained_gguf merges the LoRA adapter weights back into the base Llama 3.1 model, then converts and quantizes the result into a single GGUF file.
  • Q8_0 is an 8-bit quantization method that keeps quality high while making the file small enough to run on consumer hardware.
  • Unsloth also auto-generates a Modelfile inside the output directory. This file tells Ollama how to load and serve the model.

This export step takes several minutes as it merges billions of parameters. Be patient while it runs.

Why Q8_0 instead of a smaller quantization?

Q8_0 preserves the most quality from your fine-tuning. Smaller quantizations like q4_k_m produce smaller files but can lose some of the instruction-following behaviour you just trained. For a first fine-tune, Q8_0 lets you see the clearest results.

  • Verify the export completed by running:
!ls model_gguf

You should see a .gguf file (4-8GB) and a Modelfile in the output.

Export failing or running out of memory?

  • The GGUF export temporarily needs extra RAM. If Colab runs out of memory, go to Runtime > Restart runtime, re-run Cell 2 and Cell 3 to reload the model, then skip straight to Cell 9. You do not need to retrain.
  • Make sure your Colab session is still active. If the session timed out, reconnect and reload the model before exporting.

Help me fix a GGUF export error in Colab.

Run locally with Ollama

Ollama is a tool for running LLMs locally. It reads your GGUF file and serves it as an interactive chatbot you can talk to in the terminal.

  • Install Ollama inside your Colab environment by adding a new cell (Cell 10) and running:
!curl -fsSL https://ollama.com/install.sh | sh

Why install Ollama in Colab?

Running Ollama directly in Colab lets you test the model immediately without downloading a multi-gigabyte file first. Colab's Linux environment supports Ollama natively.

If you prefer to run on your local machine instead, download the model_gguf folder from Colab to your computer, install Ollama from ollama.com, and run the same commands below in your local terminal.

Seeing a connection error during install?

  • This is a network issue inside Colab. Wait a moment and re-run the cell. Colab sessions occasionally have brief connectivity interruptions.

Help me install Ollama in my Colab environment.

Ollama needs its server running in the background before you can create or run models. In Colab, the server does not start automatically after install.

  • Start the Ollama server as a background process by running this in a new cell:
import subprocess
import threading
import os

# Start Ollama server in background
os.environ['OLLAMA_HOST'] = '0.0.0.0:11434'
os.environ['OLLAMA_ORIGINS'] = '*'

def start_ollama_server():
    subprocess.Popen(["ollama", "serve"])

thread = threading.Thread(target=start_ollama_server)
thread.start()

import time
time.sleep(3)  # Give the server a moment to start
print("Ollama server started.")

What does this code do?

Colab blocks the cell if you run ollama serve directly because the server runs indefinitely. Wrapping it in a background thread lets the server run while you continue using other cells.

The 3-second sleep gives the server time to initialize before you issue commands against it.

  • Create your custom model from the GGUF export by running this in a new cell:
!ollama create unsloth_model -f model_gguf/Modelfile

What does this command do?

ollama create registers your GGUF model with Ollama under the name unsloth_model. The -f model_gguf/Modelfile flag points to the auto-generated Modelfile that Unsloth created during the GGUF export. This file tells Ollama the model path, chat template, and parameters.

You should see output confirming the model was created successfully.

  • Start chatting with your fine-tuned model by running:
!ollama run unsloth_model

You should see an interactive prompt where you can type messages and receive responses from your custom fine-tuned chatbot.

What should I see?

Ollama opens an interactive chat session. Type any question or instruction and press Enter. The model responds using the instruction-following behaviour it learned from the Dolly 15k dataset.

To exit the chat, type /bye and press Enter.

Model not responding or showing an error?

  • If you see a connection error, the Ollama server may not be running. Re-run the background server cell from above and wait 3 seconds before trying again.
  • If the create step failed, check that model_gguf/Modelfile exists by running !ls model_gguf. If the Modelfile is missing, re-run the GGUF export cell.
  • If Colab's session disconnected mid-export, you will need to re-run the export. Colab does not persist files across session resets.

Help me get Ollama working with my exported model.

✔️ Awesome, I've got everything!

Great. Double check that you have saved your notebook.

ⓧ I'd like to double check the full code

Here is the complete notebook code up to and including this step:

# Cell 1: Install dependencies
# !pip install unsloth
# After installation, restart the runtime if prompted

# Cell 2: Imports and model loading
from unsloth import FastLanguageModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

max_seq_length = 2048
dtype = None  # Auto-detect (float16 for T4)
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

# Cell 3: Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    max_seq_length=max_seq_length,
)

# Print trainable parameters
model.print_trainable_parameters()

# Cell 4: Load and format dataset
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")

def format_dolly(example):
    """Format a Dolly example into Llama 3.1 chat template."""
    instruction = example["instruction"]
    context = example["context"]
    response = example["response"]

    if context:
        user_message = f"{instruction}\n\nContext: {context}"
    else:
        user_message = instruction

    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_message},
        {"role": "assistant", "content": response},
    ]

    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
    return {"text": text}

dataset = dataset.map(format_dolly)

# Print a sample
print(dataset[0]["text"][:500])

# Cell 5: Configure training
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=SFTConfig(
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        output_dir="outputs",
        optim="adamw_8bit",
        seed=3407,
    ),
)

# Cell 6: Train
trainer.train()

# Cell 7: Test inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are three creative uses for old newspapers?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Cell 8: Save LoRA adapter
model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")

# Cell 9: Export to GGUF (Q8_0)
model.save_pretrained_gguf("model_gguf", tokenizer, quantization_method="q8_0")

# Cell 10: Set up Ollama and run (in Colab or locally)
# !curl -fsSL https://ollama.com/install.sh | sh
# The Modelfile is auto-generated by Unsloth during GGUF export
# !ollama create unsloth_model -f model_gguf/Modelfile
# !ollama run unsloth_model

Secret mission

Train on Your Own Custom Dataset

You trained on someone else's dataset. Now create your own. Teach the model a unique persona or specialized domain knowledge by crafting a custom instruction-following dataset and fine-tuning from scratch.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on Google Colab's free tier and Ollama locally, so there are no ongoing costs. The only consideration is disk space from the GGUF file on your local machine.

Resources you used:

  • Google Colab notebook (stored in Google Drive).
  • Ollama model unsloth_model (stored locally on your machine).
  • GGUF model file in model_gguf/ (~4-8GB, on Colab VM or downloaded locally).
  • LoRA adapter in lora_model/ (~100MB, on Colab VM or downloaded locally).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your fine-tuned model.

  • Your Colab notebook stays in Google Drive at no cost.
  • Your Ollama model stays on your machine until you explicitly remove it.
  • Files on the Colab VM (lora_model/, model_gguf/) disappear when the session ends. Download them to your local machine if you want to keep them.

Pause - I'll come back to this later

Free up the GPU session for others while keeping your files safe.

  • In Google Colab, click Runtime then Disconnect and delete runtime. This frees the GPU but your notebook stays in Google Drive.
  • Your Ollama model remains on your local machine ready to use anytime.
  • Download the GGUF file from Colab before disconnecting if you have not already.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space.

  • Remove the Ollama model by running this command in your local terminal:
ollama rm unsloth_model
  • Delete the GGUF file from your local machine if you downloaded it. On macOS/Linux:
rm -rf model_gguf/
  • On Windows, open File Explorer, navigate to the folder where you saved the GGUF file, and delete it.
  • Delete the Colab notebook from Google Drive. Open Google Drive, find the notebook in Colab Notebooks, right-click it, and select Move to trash.
  • The Colab GPU session auto-terminates after 12 hours of inactivity. No action needed there.

Nice Work!

Nice Work!

You just fine-tuned your own LLM. That puts you ahead of most people talking about AI.

You've learned how to:

  • Build a complete SFT training pipeline in Google Colab using Unsloth and QLoRA to fine-tune an 8B parameter model with only 5GB of VRAM.
  • Train on 15,000 human-generated instruction-following examples from the Databricks Dolly 15k dataset, formatted for the Llama 3.1 chat template.
  • Export the fine-tuned model to GGUF format and deploy it locally as an interactive chatbot with Ollama.
  • Train the model on a custom persona dataset and verify the behavioral change.

Ready to quiz yourself?