Fine-Tune an SLM for IoT with QLoRA

Fine-tune Gemma 3 1B into an IoT expert using QLoRA on a free Colab GPU.

Introduction

30 Second Summary

Smart home sensors, industrial controllers, and connected devices generate endless technical questions that general-purpose AI models struggle to answer with real depth. Ask one about wiring an ESP32 sensor or choosing a low-power mesh protocol, and you get a vague, surface-level response.

In this project, you will fine-tune Google's Gemma 3 1B model into a specialized IoT/electronics expert using QLoRA. You will train it on 6,005 real-world Q&A samples and see it transform from giving generic answers to producing detailed code examples and technical guidance.

What You'll Build

You will ask your fine-tuned model an ESP32 question and watch it respond with production-ready code and protocol-specific details that the base model could never produce.

By the end of this project, you'll have:

  • A side-by-side comparison showing the base Gemma 3 1B giving generic answers versus your fine-tuned version producing detailed technical responses with compilable code examples.
  • Hands-on experience with the full QLoRA fine-tuning pipeline, updating less than 1% of the model's weights on a free Google Colab T4 GPU.
  • A published LoRA adapter on Hugging Face Hub that anyone can download and use for IoT/electronics questions.
  • Secret Mission: Export your model to GGUF format for local inference with Ollama.

Do I need fine-tuning experience?

No prior fine-tuning experience is needed. You should be comfortable reading basic Python and have a general familiarity with machine learning concepts like models and training.

Before We Start

Before diving into any code or setup, let's lock in what you're about to build and why it matters.

Set Up Google Colab and Install Unsloth

Fine-tuning a language model means running thousands of matrix calculations across the model's weights. That kind of workload needs a GPU.

In this step, you'll set up Google Colab to get a free T4 GPU for training. You'll also create a Hugging Face account so you can publish your trained model later.

In this step, get ready to:
  • Create a Hugging Face account and generate a write access token.
  • Open a Google Colab notebook and connect to a T4 GPU.
  • Install Unsloth and authenticate with Hugging Face.
Create a Hugging Face account and generate an access token

Hugging Face is the go-to platform for sharing machine learning models and datasets. Your fine-tuned model will live there so anyone can download and use it.

  • Open your browser and go to huggingface.co.
  • Click Sign Up and create a free account.
  • Sign in to your new account.

Your Hugging Face account needs a write access token. This is how Colab will authenticate when pushing your model later.

  • Click your profile picture in the top-right corner.
  • Click Settings.
  • In the left sidebar, click Access Tokens.
  • Click New token.
  • Enter a name for your token, like iot-fine-tuning.
  • Select the Write preset.
  • Click Create token.
  • Copy the displayed token and save it somewhere secure. You'll paste it into Colab in a few minutes.

Treat this token like a password

Your access token gives write access to your Hugging Face account. Do not share it publicly or commit it to a repository.

Open Google Colab and connect to a T4 GPU

Google Colab gives you a free cloud notebook with access to an NVIDIA T4 GPU. This is the GPU that will power your fine-tuning later.

By default, Colab runs on a CPU. You need to switch it to a GPU runtime.

  • Click Runtime in the top menu bar.
  • Click Change runtime type.
  • In the Hardware accelerator dropdown, select T4 GPU.
  • Click Save.
  • Verify the GPU is active by running this command in a new cell:
!nvidia-smi

You should see a table listing Tesla T4 as the GPU name with 15 GB of memory.

Don't see a GPU listed?

Make sure you selected a GPU runtime. Go back to Runtime > Change runtime type and confirm T4 GPU is selected. If Colab says GPU is unavailable, wait a few minutes and try again. Free GPUs are shared and sometimes all are in use.

Help me troubleshoot why my Colab notebook isn't showing a T4 GPU when I run !nvidia-smi.

Install Unsloth and log in to Hugging Face

Unsloth is a fine-tuning framework built for speed and low memory usage. It is the only framework that fixes Gemma 3's float16 overflow issue on T4 GPUs, so training runs correctly on Colab's free tier.

  • Install Unsloth by running this code in a new cell:
# Cell 1: Install Unsloth
# Run this cell first, then restart the runtime if prompted
!pip install unsloth

What does this command do?

The ! prefix tells Colab to run a shell command inside the notebook cell. pip install unsloth downloads and installs the Unsloth library and all its dependencies.

Installation takes about a minute. You'll see a stream of output as packages download.

Colab may ask you to restart the runtime

If you see a button or message saying to restart the runtime after installation, click Restart runtime. This reloads the Python environment so the new packages take effect. Your notebook cells stay intact. You do not need to re-run the install cell after restarting.

  • Authenticate with Hugging Face by running this code in a new cell:
# Cell 2: Log in to Hugging Face
from huggingface_hub import notebook_login
notebook_login()

What does this code do?

notebook_login() displays a text field directly in the notebook cell output. You paste your Hugging Face access token there, and Colab stores it for the rest of your session.

This authentication is what lets you push your fine-tuned model to Hugging Face Hub later.

  • When the token input field appears, paste the Hugging Face access token you saved earlier.
  • Press Enter or click Login to authenticate.

You should see a success message confirming the login.

Login not working?

Make sure you're pasting the full token string with no extra spaces. If you restarted the runtime earlier, you may need to re-run Cell 2 since the environment reset.

If the token field doesn't appear at all, check that you ran the cell (click the play button or press Shift+Enter).

Help me troubleshoot my Hugging Face notebook_login() in Google Colab.

Before you run the final check, what do you expect to see if Unsloth installed correctly?

  • Verify that Unsloth is working by running this in a new cell:
import unsloth
print(unsloth.__version__)

You should see a version number printed with no error messages. This confirms Unsloth is installed and ready to go.

Seeing a ModuleNotFoundError?

This means the runtime doesn't have Unsloth loaded. If Colab asked you to restart the runtime after the install, make sure you actually restarted it. Then run this verification cell again without re-running the install cell.

Help me fix a ModuleNotFoundError for unsloth in Google Colab after installing it with pip.

✔️ Awesome, I've got everything!

Great. Make sure your notebook is saved. Your Colab notebook should now have two cells (Cell 1 and Cell 2) plus the verification cell.

ⓧ I'd like to double check the full code

Your notebook should have these two cells. The verification cell is just for checking and doesn't need to stay.

# Cell 1: Install Unsloth
# Run this cell first, then restart the runtime if prompted
!pip install unsloth

# Cell 2: Log in to Hugging Face
from huggingface_hub import notebook_login
notebook_login()

Your environment is ready. You have a GPU-powered notebook and a connection to Hugging Face. Next up, you'll load Gemma 3 1B and see what happens when you ask it IoT questions it was never trained on.

Load Gemma 3 1B and Expose Its IoT Blind Spots

Your Colab notebook is ready with a T4 GPU, Unsloth installed, and your Hugging Face account connected. Now it's time to load a model and put it to the test.

Gemma 3 1B is Google's compact, instruction-tuned language model. It handles general questions well, but it has no specialized IoT or electronics training. In this step, you'll load the model and ask it domain-specific questions to see exactly where it falls short.

In this step, get ready to:
  • Load Gemma 3 1B in 4-bit quantization on the free T4 GPU.
  • Test the base model with three IoT questions.
  • Save the base responses for comparison after fine-tuning.
Load the Gemma 3 1B model in 4-bit quantization

4-bit quantization compresses the model's weights from their original precision down to 4 bits per parameter. This shrinks Gemma 3 1B from 2 GB down to about 0.5 GB, making it fit comfortably on Colab's free T4 GPU.

  • Create a new cell in your Colab notebook and paste the following code:
# Cell 3: Load Gemma 3 1B in 4-bit
from unsloth import FastLanguageModel
import torch

max_seq_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-1b-it-unsloth-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=True,
)

print("Model loaded successfully!")

What does this code do?

  • FastLanguageModel.from_pretrained downloads and loads the Gemma 3 1B model with Unsloth's optimizations. The model handles generation and tokenizer converts text to and from tokens.
  • unsloth/gemma-3-1b-it-unsloth-bnb-4bit is Unsloth's pre-quantized version of Gemma 3 1B. The it means instruction-tuned, and bnb-4bit means it uses 4-bit quantization via bitsandbytes.
  • max_seq_length=2048 sets the maximum number of tokens per input/output sequence. This is enough for detailed IoT answers.
  • load_in_4bit=True tells Unsloth to keep the model weights compressed at 4-bit precision, saving VRAM for training later.
  • Run the cell by clicking the play button or pressing Shift+Enter.

You should see download progress bars as the model downloads, followed by Model loaded successfully! printed at the end.

Seeing an error about the runtime?

  • If you see a CUDA-related error, double-check your runtime type. Click Runtime in the menu bar, then Change runtime type. Confirm that T4 GPU is selected.
  • If Colab says no GPU is available, free-tier GPUs may be temporarily at capacity. Wait a few minutes and try reconnecting.
  • help me fix a CUDA or GPU error when loading the Gemma 3 model in Colab
Ask three IoT questions and save the base responses

Now that the model is loaded, you need a way to send it questions and capture the answers. You also need a set of IoT questions that will reveal the model's limitations.

  • Create a new cell and paste the helper function that sends a question to the model:
# Cell 4: Define helper function and test questions
def generate_response(model, tokenizer, question, max_tokens=256):
    messages = [{"role": "user", "content": question}]
    input_ids = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    ).to("cuda")
    with torch.no_grad():
        outputs = model.generate(
            input_ids=input_ids,
            max_new_tokens=max_tokens,
            do_sample=True,
            temperature=0.7,
        )
    response = tokenizer.decode(
        outputs[0][input_ids.shape[-1]:],
        skip_special_tokens=True,
    )
    return response

What does this function do?

  • tokenizer.apply_chat_template wraps the question in the chat format Gemma 3 expects, with the correct role tags and generation prompt.
  • torch.no_grad() disables gradient tracking during generation. This saves memory because you're only asking the model to respond, not training it.
  • model.generate produces the response tokens. The temperature=0.7 adds slight randomness so answers feel natural.
  • tokenizer.decode converts the output tokens back into readable text. The slicing outputs[0][input_ids.shape[-1]:] strips the original question out so you only see the model's answer.
  • Still in the same cell, add the three test questions below the function:
test_questions = [
    "How do I set up an ESP32 MQTT client with TLS encryption?",
    "What is the best protocol for low-power IoT sensor networks?",
    "Explain how to integrate a DHT22 temperature sensor with Home Assistant using ESPHome.",
]

Why these three questions?

Each question targets a different area of IoT expertise. The first asks about ESP32 firmware with MQTT and TLS security. The second asks about protocol selection for sensor networks. The third asks about integrating hardware with Home Assistant.

A domain expert would answer with specific register names, compilable code, and protocol comparisons. A general-purpose model will give vague, surface-level advice.

  • Run the cell. The function and questions are now defined in memory for the next cell to use.

Now it's time for the real test. Before you run the base model, think about this: what kind of answer do you expect a general-purpose 1B parameter model to give when asked about ESP32 MQTT configuration with TLS?

  • Create a new cell and paste the code that generates and saves the base model's responses:
# Cell 5: Generate and save base model responses
base_responses = []
print("=" * 60)
print("BASE MODEL RESPONSES (before fine-tuning)")
print("=" * 60)
for q in test_questions:
    response = generate_response(model, tokenizer, q)
    base_responses.append(response)
    print(f"\nQ: {q}")
    print(f"A: {response}")
    print("-" * 60)

What does this code do?

  • The loop sends each question in test_questions to generate_response and prints the question-answer pair.
  • base_responses stores each answer in a list. You'll use this list in a later step to compare side-by-side with the fine-tuned model's answers.
  • Run the cell. Each response takes a few seconds to generate.

You should see all three questions printed with the model's answers. Read through them carefully.

The answers are grammatically correct and topically relevant, but notice what's missing. The ESP32 MQTT answer likely lacks compilable code or specific library references. The protocol comparison skips concrete tradeoffs like power consumption numbers. The Home Assistant answer stays high-level without actual ESPHome YAML configuration.

Why are the answers generic?

Gemma 3 1B was trained on broad internet text. It knows IoT exists, but it was never trained on detailed, expert-level Q&A pairs about specific protocols, hardware, and integrations.

This gap is exactly what fine-tuning will fix. In the next steps, you'll train the model on 6,005 real-world IoT Q&A samples and then re-ask these same three questions to see the difference.

Seeing an empty or garbled response?

  • If a response looks cut off, the model hit the max_tokens=256 limit. This is expected for the base model test. You'll increase this limit when comparing with the fine-tuned model later.
  • If you see a CUDA out-of-memory error, click Runtime then Restart session and re-run all cells from Cell 1.
  • help me debug an issue with generating responses from Gemma 3 1B in Colab

✔️ Awesome, I've got everything!

Great. Make sure you've run all three cells (Cell 3, Cell 4, and Cell 5) and that base_responses is saved in memory. You'll need it later for the side-by-side comparison.

ⓧ I'd like to double check the full code

Scroll through your Colab notebook and confirm these cells are present, in order:

  • Cell 1: !pip install unsloth (from the previous step).
  • Cell 2: notebook_login() (from the previous step).
  • Cell 3: Imports FastLanguageModel and torch, loads the model with from_pretrained, and prints Model loaded successfully!.
  • Cell 4: Defines the generate_response function and the test_questions list with three IoT questions.
  • Cell 5: Creates the base_responses list, loops through test_questions, and prints each question-answer pair.

Those generic answers are the baseline. In the next step, you'll load a training dataset packed with 6,005 expert-level IoT Q&A pairs that will teach the model what a real answer looks like.

Load and Format the IoT Training Dataset

In the last step, you asked Gemma 3 1B three IoT questions and saw how generic its answers were. The model gave surface-level responses that lacked the technical depth an engineer would need.

The fix is domain-specific training data. In this step, you'll load a dataset of 6,005 expert-level IoT Q&A pairs and format them so the model can learn from real-world answers.

In this step, get ready to:
  • Load the Mascarade IoT dataset from Hugging Face Hub.
  • Format conversations from ShareGPT style into the tokenizer's chat template.
Load the dataset from Hugging Face

The Mascarade IoT dataset on Hugging Face Hub contains 6,005 expert-level Q&A pairs covering ESP32, MQTT, BLE, LoRaWAN, and Home Assistant.

Each sample uses the ShareGPT conversation format, where each turn is labeled with a role like system, human, or gpt.

  • Create a new code cell in your Colab notebook and load the dataset by pasting this code:
# Cell 6: Load the IoT dataset
from datasets import load_dataset

dataset = load_dataset(
    "electron-rare/mascarade-iot-dataset",
    split="train",
)
print(f"Dataset size: {len(dataset)} samples")
print(f"\nFirst conversation preview:")
for turn in dataset[0]["conversations"]:
    role = turn["from"]
    content = turn["value"][:100]
    print(f"  [{role}]: {content}...")

What does this code do?

  • load_dataset("electron-rare/mascarade-iot-dataset", split="train") downloads all 6,005 training samples from Hugging Face Hub.
  • The loop prints the first conversation's turns, showing each turn's from role and a 100-character preview of the content.
  • Run the cell.
  • You should see Dataset size: 6005 samples followed by a preview of the first conversation showing [system], [human], and [gpt] role labels.

Seeing a ModuleNotFoundError?

  • If you see an error about the datasets module not being found, it wasn't installed with Unsloth. Run !pip install datasets in a new cell and try again.
  • If the download is slow or times out, run the cell again. Hugging Face datasets sometimes need a retry on the first attempt.
  • Still stuck? Help me troubleshoot loading the mascarade-iot-dataset in Google Colab.

What is ShareGPT format?

ShareGPT is a popular format for storing multi-turn conversations as training data. Each sample contains a conversations array where every turn has a from field (the role) and a value field (the message text).

Many fine-tuning datasets use this format because it cleanly separates each speaker's turn, making it easy to convert into the chat template any model expects.

Format conversations for training

The dataset uses ShareGPT role names like human and gpt, but the Gemma tokenizer expects standard chat roles like user and assistant.

You need to map these roles and then apply the tokenizer's chat template to produce the exact format that SFTTrainer expects for training.

  • Create a new code cell and define the formatting function by pasting this code:
# Cell 7: Format dataset for training
def format_conversation(example):
    role_map = {"system": "system", "human": "user", "gpt": "assistant"}
    messages = []
    for turn in example["conversations"]:
        messages.append({
            "role": role_map.get(turn["from"], turn["from"]),
            "content": turn["value"],
        })
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": text}

dataset = dataset.map(format_conversation)
print("Formatted sample preview:")
print(dataset[0]["text"][:500])

What does this code do?

  • role_map translates ShareGPT labels to standard chat roles: human becomes user, and gpt becomes assistant.
  • tokenizer.apply_chat_template() wraps each conversation in the special tokens that Gemma expects during training.
  • dataset.map(format_conversation) applies this function to every sample, adding a text field that SFTTrainer reads directly.

Before you run this cell, think about what the formatted output will look like. How would Gemma's chat template structure a conversation with system, user, and assistant turns?

  • Run the cell.
  • You should see a formatted sample preview showing the conversation wrapped in Gemma's special tokens, with user and assistant role labels instead of the original human and gpt labels.

Seeing a KeyError or empty output?

  • Make sure you ran Cell 6 first. The dataset variable must already be loaded before you can map it.
  • Check that Cell 3 ran successfully. The tokenizer variable from that cell is used inside format_conversation().
  • Still stuck? Help me debug my dataset formatting for Gemma fine-tuning.

✔️ Awesome, I've got everything!

Great. Double check you've run both Cell 6 and Cell 7 successfully before moving on.

ⓧ I'd like to double check the full code

Scroll through your notebook and confirm these cells are present and have run successfully, in order:

  • Cells 1-5 from previous steps (install Unsloth, log in to Hugging Face, load the model, define the helper function, generate base responses).
  • Cell 6 loads the IoT dataset and prints the dataset size and a conversation preview.
  • Cell 7 defines format_conversation(), maps the dataset, and prints a formatted sample preview.

Your dataset is loaded and formatted into 6,005 training-ready conversations. Next up, you'll attach LoRA adapters to the model and run the actual fine-tuning.

Fine-Tune with QLoRA

Your IoT dataset is formatted and ready for training. Now comes the core of this project: teaching Gemma 3 1B to be an IoT specialist.

Full fine-tuning would update all 1 billion parameters, requiring far more GPU memory than a free Colab T4 provides. QLoRA solves this by freezing the base model at 4-bit quantization and training only small adapter matrices that update less than 1% of the model's weights.

In this step, get ready to:
  • Attach LoRA adapters to the model's attention and MLP layers.
  • Run supervised fine-tuning for 60 training steps and watch the loss decrease.
Attach LoRA adapters to the model

LoRA works by injecting small, trainable matrices into specific layers of the frozen model. Instead of updating all 1 billion parameters, you only train these lightweight adapters.

  • Create a new code cell in your Colab notebook (Cell 8) and paste the following code:
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
    use_rslora=False,
    loftq_config=None,
)

What does this code do?

  • r=16 sets the rank of each LoRA matrix. This controls how much new information each adapter can capture. Rank 16 is the standard recommendation for QLoRA fine-tuning.
  • target_modules lists the seven attention and MLP projection layers that receive adapters. These are the layers where domain knowledge is most effectively encoded.
  • use_gradient_checkpointing="unsloth" trades a small amount of speed for significant VRAM savings during training.
  • Run the cell by pressing the play button or Shift+Enter.

You should see a summary showing the number of trainable parameters versus total parameters. The trainable count is a tiny fraction of the total.

Why is the trainable parameter count so small?

That small number is the whole point of PEFT (Parameter-Efficient Fine-Tuning). The base model's billion parameters stay frozen at 4-bit precision. Only the LoRA adapter weights are updated during training, keeping VRAM usage low enough for a free T4 GPU.

Seeing an out-of-memory error?

  • Make sure your runtime is set to T4 GPU. Click Runtime then Change runtime type to confirm.
  • If you ran other notebooks before this one, your GPU memory may be full. Click Runtime then Disconnect and delete runtime, reconnect, and re-run all previous cells.

Why am I getting an out-of-memory error when attaching LoRA adapters?

Configure and run training with SFTTrainer

SFTTrainer from the trl library handles the supervised fine-tuning loop. It feeds your formatted dataset through the model, computes loss, and updates only the LoRA adapter weights.

  • Create a new code cell (Cell 9) and paste the following code:
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)

trainer_stats = trainer.train()

What does this code do?

  • per_device_train_batch_size=2 with gradient_accumulation_steps=4 gives an effective batch size of 8. Gradient accumulation simulates a larger batch without needing the VRAM to hold all 8 samples at once.
  • max_steps=60 caps training at 60 steps. This processes roughly 480 samples from the dataset.
  • learning_rate=2e-4 is the recommended starting rate for QLoRA fine-tuning.
  • logging_steps=10 prints the training loss every 10 steps so you can watch progress in real time.
  • trainer.train() kicks off the training loop and stores the results in trainer_stats.

Before you run this cell, think about what you expect to happen to the training loss over 60 steps. Should it go up, go down, or stay flat?

  • Run the cell. Training takes a few minutes on a T4 GPU.

You should see a training log with loss values printed every 10 steps. The loss should decrease noticeably from the first logged value to the last. That falling loss means the model is learning the IoT domain patterns from your dataset.

Training seems stuck or loss isn't decreasing?

  • Check that you ran Cell 7 (the dataset formatting cell) before this one. If the text field is missing from the dataset, the trainer has nothing to learn from.
  • If you see a CUDA error, your runtime may have disconnected. Click Runtime then Reconnect, and re-run all cells from Cell 3 onwards.
  • A flat loss after all 60 steps can happen if the LoRA adapters were not attached. Confirm Cell 8 ran without errors before running Cell 9.

My training loss isn't decreasing after 60 steps. What should I check?

✔️ Awesome, I've got everything!

Double check that you've saved your notebook. Both Cell 8 and Cell 9 should have run without errors, and your training log should show decreasing loss values.

ⓧ I'd like to double check the full code

Your notebook should now have 9 cells. Scroll through and confirm each one is present in this order:

  • Cell 1: !pip install unsloth to install Unsloth.
  • Cell 2: notebook_login() for Hugging Face authentication.
  • Cell 3: FastLanguageModel.from_pretrained to load Gemma 3 1B in 4-bit.
  • Cell 4: generate_response() helper function and test_questions list.
  • Cell 5: Base model response generation, saved in base_responses.
  • Cell 6: load_dataset to load the IoT dataset.
  • Cell 7: format_conversation() to format the dataset for training.
  • Cell 8: FastLanguageModel.get_peft_model to attach LoRA adapters.
  • Cell 9: SFTTrainer configuration and trainer.train() to run fine-tuning.

Your model just learned from thousands of IoT conversations. Next up, you'll ask it the same three questions from earlier and compare the before-and-after responses side by side. Then you'll publish your fine-tuned model to Hugging Face Hub.

Compare Results and Publish to Hugging Face Hub

Your training run is complete and the loss decreased over those 60 steps. But a dropping loss number only tells you the model got better at predicting the training data.

The real question is whether the model can now answer IoT questions like a domain expert. In this step, you'll ask the same three questions from earlier and compare the fine-tuned responses side-by-side with the base model's vague originals. Then you'll publish the trained LoRA adapter to Hugging Face Hub so anyone can download and use it.

In this step, get ready to:
  • Compare fine-tuned model responses with the base model's original answers.
  • Publish your trained LoRA adapter to Hugging Face Hub.
Merge LoRA weights and compare responses side-by-side

During training, the LoRA adapter weights lived separately from the base model. Before you can generate text with the fine-tuned model, those adapter weights need to be merged back in for optimized inference.

Before you run this cell, think about what you expect. Will the fine-tuned model's IoT answers look noticeably different from the base model's? What specific improvements would you look for?

  • Merge the LoRA weights and run the side-by-side comparison by creating a new code cell and pasting this code:
# Cell 10: Compare fine-tuned vs base model
FastLanguageModel.for_inference(model)

print("=" * 60)
print("COMPARISON: BASE vs FINE-TUNED MODEL")
print("=" * 60)
for i, q in enumerate(test_questions):
    finetuned_response = generate_response(model, tokenizer, q, max_tokens=512)
    print(f"\nQ: {q}")
    print(f"\n--- BASE MODEL ---")
    print(base_responses[i])
    print(f"\n--- FINE-TUNED MODEL ---")
    print(finetuned_response)
    print("=" * 60)

What does this code do?

  • FastLanguageModel.for_inference(model) merges the trained LoRA adapter weights back into the base model so it can generate text at full speed.
  • The loop asks the same three IoT questions from Step 2, but this time with max_tokens=512 (doubled from 256) to give the fine-tuned model room for longer, more detailed answers.
  • Each question prints the base model's original response next to the fine-tuned response so you can see the difference directly.

The output prints each question followed by both the base model's response and the fine-tuned model's response.

Look for concrete improvements in the fine-tuned answers. The base model likely gave surface-level definitions, while the fine-tuned model should include specific protocol details, register names, or compilable code examples.

Seeing an error or identical responses?

  • If you see a CUDA out of memory error, click **Runtime**, then **Restart runtime**, and re-run Cells 3 through 10 in order. The T4 GPU has limited VRAM and stale tensors from earlier cells can fill it up.
  • If the fine-tuned answers look nearly identical to the base model, confirm Cell 9 completed all 60 training steps. Check the last logged loss value in Cell 9's output.
  • If you see a NameError for base_responses, you need to re-run Cell 5 first. The base responses must be stored in memory before you can compare them.

Why are my fine-tuned model responses not improving over the base model?

Save and push the LoRA adapter to Hugging Face Hub

Your fine-tuned model currently only exists in Colab's temporary memory. If the runtime disconnects, you lose everything. Saving the LoRA adapter locally and pushing it to Hugging Face Hub gives you a permanent, shareable copy.

  • Before running the next cell, note your Hugging Face username: your Hugging Face username
  • Save the model locally and push it to Hugging Face Hub by creating a new code cell and pasting this code:
# Cell 11: Save and push to Hugging Face Hub
model.save_pretrained("iot-expert-gemma3-1b-lora")
tokenizer.save_pretrained("iot-expert-gemma3-1b-lora")

# Replace "your-hf-username" with your actual Hugging Face username
model.push_to_hub("your-hf-username/iot-expert-gemma3-1b-lora")
tokenizer.push_to_hub("your-hf-username/iot-expert-gemma3-1b-lora")
print("Model pushed to Hugging Face Hub!")
  • Replace both instances of your-hf-username in the push_to_hub lines with your actual Hugging Face username.
  • Run the cell.

What does this code do?

  • save_pretrained writes the LoRA adapter weights and tokenizer files to a local folder called iot-expert-gemma3-1b-lora inside the Colab runtime.
  • push_to_hub uploads the adapter and tokenizer to a new repository on your Hugging Face profile. Only the small LoRA adapter weights are uploaded, not the full 1B parameter model.

The cell should finish with Model pushed to Hugging Face Hub! printed in the output.

  • Open a new browser tab and go to huggingface.co.
  • Click your profile icon in the top right corner.
  • Click **Profile**.
  • You should see a new repository called iot-expert-gemma3-1b-lora listed on your profile. Click it to confirm the adapter files are there.

Don't see the repository on your profile?

  • Double-check that you replaced your-hf-username with your actual Hugging Face username in both push_to_hub lines. A typo creates the repo under a non-existent user and the push silently fails.
  • Make sure your Hugging Face token has **Write** permission. If you created a read-only token in Step 1, go back to **Settings**, then **Access Tokens**, and create a new one with Write permission.
  • If you see an authentication error, re-run Cell 2 (notebook_login()) and paste a fresh token with Write permission.

Why is my model not appearing on Hugging Face Hub after push_to_hub?

✔️ Awesome, I've got everything!

Great. Confirm you have saved your Colab notebook. Your fine-tuned model is live on Hugging Face Hub and ready for anyone to download.

ⓧ I'd like to double check the full code

Scroll through your Colab notebook and confirm you have Cells 1 through 11 in order. Each cell's code should match the code you pasted in its respective step.

  • Cell 1: !pip install unsloth to install Unsloth.
  • Cell 2: notebook_login() to authenticate with Hugging Face.
  • Cell 3: Load Gemma 3 1B with FastLanguageModel.from_pretrained.
  • Cell 4: Define generate_response and test_questions.
  • Cell 5: Generate and save base model responses into base_responses.
  • Cell 6: Load the IoT dataset with load_dataset.
  • Cell 7: Format the dataset with format_conversation and apply_chat_template.
  • Cell 8: Attach LoRA adapters with get_peft_model.
  • Cell 9: Train with SFTTrainer for 60 steps.
  • Cell 10: Merge LoRA weights with for_inference and print the side-by-side comparison.
  • Cell 11: Save locally and push to Hugging Face Hub with push_to_hub.

Secret mission

Export to GGUF for Local Inference

Your fine-tuned model lives on Hugging Face Hub, but it still needs Python and a GPU to run. In this Secret Mission, you will export it to GGUF format so anyone can run it locally with Ollama. No Python, no GPU, just a single terminal command.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on Google Colab's free tier and Hugging Face's free tier, so there are no ongoing costs. Decide whether to keep your resources available, pause them to come back later, or delete them entirely.

Resources you used:

  • Google Colab notebook (stored in Google Drive).
  • Hugging Face Hub repository: your-hf-username/iot-expert-gemma3-1b-lora.
  • Hugging Face Hub repository: your-hf-username/iot-expert-gemma3-1b-gguf (Secret Mission).
  • Hugging Face access token with Write permission.
  • Local Ollama model (Secret Mission).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your fine-tuned model or want to share it publicly on Hugging Face Hub.

  • Your Hugging Face Hub repositories will remain publicly accessible for anyone to download and use.
  • Your Google Colab notebook stays in Google Drive for future reference.
  • Your Hugging Face access token remains active.
  • Your local Ollama model stays available for inference (Secret Mission only).

Pause - I'll come back to this later

Free up local resources while keeping your cloud artifacts intact. Your Colab runtime disconnects automatically after inactivity, so all local files created during training are already gone.

  • Your Hugging Face Hub repositories stay available. No action needed there.
  • If you completed the Secret Mission and want to free disk space, remove the local Ollama model by running:
ollama rm your-hf-username/iot-expert-gemma3-1b-gguf
  • You can re-download it later with ollama run when you need it again.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each item below to fully clean up.

  • Delete your Google Colab notebook from Google Drive.
  • Open Google Drive at drive.google.com.
  • Find the notebook in the Colab Notebooks folder.
  • Right-click the notebook and select Move to trash.
  • Delete your Hugging Face Hub LoRA repository.
  • Go to huggingface.co/your-hf-username/iot-expert-gemma3-1b-lora.
  • Click the Settings tab.
  • Scroll to the bottom and click Delete this repository. Confirm when prompted.
  • Delete your Hugging Face Hub GGUF repository (Secret Mission only).
  • Go to huggingface.co/your-hf-username/iot-expert-gemma3-1b-gguf.
  • Click the Settings tab.
  • Scroll to the bottom and click Delete this repository. Confirm when prompted.
  • Revoke your Hugging Face access token.
  • Go to huggingface.co and click your profile icon in the top right.
  • Select Settings, then Access Tokens.
  • Find the token you created for this project and click Revoke.
  • Remove the local Ollama model (Secret Mission only).
  • Run this in your terminal:
ollama rm your-hf-username/iot-expert-gemma3-1b-gguf

Nice Work!

Nice Work!

You just took a general-purpose language model and turned it into a domain specialist. Your fine-tuned Gemma 3 1B model now gives detailed, technical IoT answers with code examples and protocol-specific guidance.

You've learned how to:

  • Fine-tuned Gemma 3 1B on 6,005 IoT/electronics Q&A samples using QLoRA, training less than 1% of the model's parameters in 4-bit precision on a free Google Colab T4 GPU.
  • Compared base model vs fine-tuned model responses side-by-side, seeing the transformation from generic, surface-level answers to detailed technical responses with compilable code examples.
  • Published the fine-tuned LoRA adapter to Hugging Face Hub for anyone to download and use.
  • Secret Mission: Exported the fine-tuned model to GGUF format with Q4_K_M quantization and tested it locally with Ollama for fast inference without Python or a GPU.

Ready to quiz yourself?