Fine-Tune an SLM with QLoRA
Fine-tune SmolLM3-3B with QLoRA on real sanctions data for entity resolution.
Introduction
30 Second Summary
When banks screen transactions against sanctions lists or investigators trace corporate ownership across borders, they face a deceptively hard problem: determining whether two records with slightly different names, scripts, or missing fields refer to the same real-world entity. Getting this wrong means either missing sanctioned parties or flooding analysts with false positives.
In this project, you will fine-tune a Small Language Model on real sanctions data to classify entity pairs as matches or non-matches, build a tiered evaluation framework, and package everything into a production-ready GitHub repository.
What You'll Build
A complete end-to-end entity resolution system powered by a QLoRA-fine-tuned SmolLM3-3B model, trained locally on your GPU using the OpenSanctions Pairs dataset.
By the end of this project, you'll have:
- A QLoRA-fine-tuned adapter for SmolLM3-3B that classifies entity pairs from real sanctions and corporate ownership data spanning 31 countries.
- A data preparation pipeline that processes messy, multilingual entity records into chat-format training examples with stratified train/eval splits.
- A tiered evaluation framework measuring precision, recall, and F1 across easy, medium, and hard difficulty tiers with baseline comparisons.
- A production-ready GitHub repository with a comprehensive README documenting model selection rationale, hyperparameter choices, and results.
- Secret Mission: Compare QLoRA vs full LoRA training and document the trade-offs with empirical evidence.
Are there any prerequisites?
This project requires an NVIDIA RTX 4070 GPU (16 GB VRAM), 32 GB system RAM, and Python installed on your machine. You should have basic familiarity with PyTorch and the HuggingFace ecosystem (Transformers, tokenizers, datasets).
Set Up Your Environment
Before you can fine-tune a model, you need a solid local environment. This step gets your machine ready with the exact library versions, dataset, and project structure needed for the rest of the project.
You will create a Python virtual environment, install the ML stack (PyTorch, HuggingFace Transformers, PEFT), download the OpenSanctions Pairs dataset, and verify that your GPU is accessible for training.
In this step, get ready to:
- Install all required Python packages with exact version pins and verify GPU detection.
- Download the OpenSanctions Pairs dataset and create the project directory structure.
- Verify HuggingFace model access and initialize a Git repository.
Install the required Python packages
The fine-tuning stack requires specific library versions to work together. QLoRA relies on bitsandbytes for 4-bit quantization, peft for adapter configuration, and trl for supervised fine-tuning. Pinning versions avoids compatibility headaches.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir entity-resolution-qlora
cd entity-resolution-qlora- Create a Python virtual environment and activate it by running:
python3 -m venv .venv
source .venv/bin/activateWhat is a virtual environment?
A virtual environment isolates your project's Python packages from the rest of your system. This means you can pin exact versions for this project without conflicting with other projects on your machine.
- Create the requirements.txt file by running:
cat > requirements.txt << 'EOF'
torch>=2.4
transformers>=4.53.0
peft==0.19.1
trl>=0.12.0
bitsandbytes>=0.45.0
datasets>=2.15.0
accelerate>=1.1.0
scikit-learn>=1.4.0
numpy>=1.17
EOFWhat do these packages do?
- torch is the deep learning framework that powers model training on your GPU.
- transformers provides the SmolLM3-3B model and tokenizer. Version 4.53.0+ is required for SmolLM3 support.
- peft implements LoRA and QLoRA adapters for parameter-efficient fine-tuning.
- trl provides the SFTTrainer for supervised fine-tuning with chat-format data.
- bitsandbytes handles 4-bit quantization to fit the model in your 16GB VRAM.
- Install all packages by running:
pip install -r requirements.txtThis install takes a few minutes
PyTorch alone is over 2GB. Grab a coffee while pip downloads and installs everything.
- Verify your GPU is detected by running:
python3 -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'Device: {torch.cuda.get_device_name(0)}')"- You should see output showing CUDA available: True and your GPU name (e.g. NVIDIA GeForce RTX 4070).
Seeing CUDA available: False?
- Make sure you have NVIDIA drivers installed. Run nvidia-smi to check.
- Confirm you installed the CUDA-enabled version of PyTorch. Visit pytorch.org for the correct install command for your system.
Need more help? Help me debug why CUDA is not detected on my RTX 4070
Download the OpenSanctions Pairs dataset
The OpenSanctions Pairs dataset contains 755K+ expert-labeled entity pairs from real sanctions and compliance data. Each pair has a left entity, a right entity, and a judgement (positive or negative) indicating whether they refer to the same real-world entity.
- Create the project directory structure by running:
mkdir -p src data/raw data/processed checkpoints adapters reports- Confirm the directories exist by running:
ls- You should see src, data, checkpoints, adapters, reports, and requirements.txt in the output.
- Download the OpenSanctions Pairs dataset by running:
wget -P data/raw/ https://data.opensanctions.org/contrib/training/pairs-20251209.json.gzHow big is this download?
The compressed file is approximately 390MB. The full dataset contains 755K+ pairs, but your code will only load the first 5000 pairs for training. Downloading the full file gives you the flexibility to scale up later.
- Decompress the file by running:
gunzip data/raw/pairs-20251209.json.gz- Verify the file exists by running:
ls data/raw/- You should see pairs-20251209.json in the output.
wget not found?
- On macOS, install wget with brew install wget. Alternatively, use curl -L -o data/raw/pairs-20251209.json.gz https://data.opensanctions.org/contrib/training/pairs-20251209.json.gz instead.
- On Windows with Git Bash, curl is available by default. Use the curl alternative above.
Need more help? Help me download the OpenSanctions dataset
Verify HuggingFace access and initialize Git
Before you start training, confirm that your environment can load the SmolLM3-3B tokenizer from HuggingFace. This catches authentication or network issues early, before you commit to a long training run.
- Verify access to the SmolLM3-3B tokenizer by running:
python3 -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('HuggingFaceTB/SmolLM3-3B'); print(f'Tokenizer loaded. Vocab size: {t.vocab_size}')"- You should see a message confirming the tokenizer loaded with a vocab size of 128256.
Why SmolLM3-3B?
SmolLM3-3B is HuggingFace's current state-of-the-art model at the 3B parameter scale. It uses Grouped Query Attention (4 key-value heads) which makes it memory-efficient for fine-tuning.
At 3B parameters with QLoRA, it fits comfortably within your 16GB VRAM. Larger models would need more expensive hardware or aggressive quantization that hurts quality.
Tokenizer download failing?
- Check your internet connection. The tokenizer files are downloaded from HuggingFace Hub.
- If you are behind a corporate proxy, set HF_HOME and configure proxy settings for requests.
Need more help? Help me load the SmolLM3-3B tokenizer from HuggingFace
- Initialize a Git repository by running:
git init- Create a README skeleton by running:
cat > README.md << 'EOF'
# Entity Resolution with QLoRA Fine-Tuned SmolLM3-3B
Fine-tuning SmolLM3-3B using QLoRA for entity resolution on OpenSanctions Pairs data.
## Overview
- **Model**: HuggingFaceTB/SmolLM3-3B
- **Method**: QLoRA (4-bit quantization + LoRA adapters)
- **Dataset**: OpenSanctions Pairs (755K+ labeled entity pairs)
- **Hardware**: RTX 4070 (16GB VRAM)
## Project Structure
- `src/` - Training and evaluation scripts
- `data/` - Raw and processed datasets
- `adapters/` - Saved LoRA adapter checkpoints
- `reports/` - Evaluation reports
EOF- Stage and commit your initial files by running:
git add requirements.txt README.md
git commit -m "Initial commit: requirements and README skeleton"- You should see a commit confirmation message listing 2 files changed.
✔️ Awesome, I've got everything!
Great. Make sure you have saved requirements.txt and confirmed your GPU is detected.
ⓧ I'd like to double check the full code
Here is the complete requirements.txt for comparison:
torch>=2.4
transformers>=4.53.0
peft==0.19.1
trl>=0.12.0
bitsandbytes>=0.45.0
datasets>=2.15.0
accelerate>=1.1.0
scikit-learn>=1.4.0
numpy>=1.17Your environment is ready and battle-tested. Next up, you will build the data preparation pipeline that transforms raw entity pairs into the chat-format training examples your model needs.
Prepare Your Data
Your environment is locked and loaded. Now you need training data that teaches your model what entity resolution actually looks like in the real world.
The OpenSanctions Pairs dataset you downloaded contains messy, multilingual entity records from real sanctions and corporate ownership data. In this step, you will build a data preparation pipeline that transforms raw entity pairs into chat-format training examples, creates stratified train/eval splits, and tags evaluation samples with difficulty tiers.
In this step, get ready to:
- Explore the raw entity pair data and understand its schema.
- Build a serialization pipeline that converts entity pairs into SFT chat-format examples.
- Create difficulty tiers and generate train/eval splits for your model.
Load and explore the OpenSanctions Pairs data
Before writing any transformation code, you need to understand what your raw data looks like. Each line in pairs-20251209.json is a JSON object with three top-level fields.
What does the data look like?
Each record has a left entity, a right entity, and a judgement field (positive for a match, negative for non-match).
Each entity object contains an id, a caption (display name), a schema (Person, Company, or Organization), and a properties dictionary mapping attribute names to lists of string values.
Common property fields include alias, birthDate, nationality, country, address, and idNumber. The dataset is 76.9% positive (matches) and 23.1% negative (non-matches).
- Create the file src/data_prep.py and add the imports and data loading function by pasting this code:
import json
import random
from pathlib import Path
from sklearn.model_selection import train_test_split
def load_pairs(file_path: str, max_pairs: int = 5000) -> list[dict]:
"""Load entity pairs from line-based JSON file."""
pairs = []
with open(file_path, "r", encoding="utf-8") as f:
for i, line in enumerate(f):
if i >= max_pairs:
break
pair = json.loads(line.strip())
pairs.append(pair)
return pairsWhat does this code do?
The imports pull in json for parsing, random for shuffling, Path for file system operations, and train_test_split from scikit-learn for stratified splitting.
The load_pairs() function reads the line-based JSON file one line at a time, capping at max_pairs to keep training manageable on a single GPU. Each line is a complete JSON object representing one entity pair.
- Save src/data_prep.py.
Import error on scikit-learn?
If you see ModuleNotFoundError: No module named 'sklearn', make sure your virtual environment is activated and that you installed the packages from requirements.txt in Step 1.
help me debug this scikit-learn import error
Build the entity serialization and chat formatting pipeline
Your model needs structured text input, not raw nested JSON. You will serialize each entity's properties into a readable text block, then wrap pairs of entities into a chat-format prompt suitable for Supervised Fine-Tuning (SFT).
- Add the serialize_entity() function below load_pairs() in src/data_prep.py:
def serialize_entity(entity: dict) -> str:
"""Serialize an entity's properties into a readable text block."""
parts = [f"Name: {entity.get('caption', 'Unknown')}"]
props = entity.get("properties", {})
if "alias" in props:
parts.append(f"Aliases: {', '.join(props['alias'])}")
if "birthDate" in props:
parts.append(f"Birth Date: {', '.join(props['birthDate'])}")
if "nationality" in props:
parts.append(f"Nationality: {', '.join(props['nationality'])}")
if "country" in props:
parts.append(f"Country: {', '.join(props['country'])}")
if "idNumber" in props:
parts.append(f"ID Numbers: {', '.join(props['idNumber'])}")
if "address" in props:
parts.append(f"Address: {', '.join(props['address'][:2])}")
schema = entity.get("schema", "Unknown")
parts.append(f"Type: {schema}")
return "\n".join(parts)What does this code do?
This function extracts the most informative fields from an entity's properties dictionary and formats them as labeled lines. It pulls the caption as the primary name, then conditionally adds aliases, birth dates, nationalities, countries, ID numbers, and addresses.
Addresses are capped at 2 entries to avoid overwhelming the model's context window. The schema field (Person, Company, or Organization) is always included to give the model entity-type context.
- Add the format_pair_as_chat() function below serialize_entity():
def format_pair_as_chat(pair: dict) -> dict:
"""Convert an entity pair into chat-format training example."""
left = serialize_entity(pair["left"])
right = serialize_entity(pair["right"])
judgement = pair["judgement"]
label = "MATCH" if judgement == "positive" else "NO_MATCH"
system_msg = (
"You are an entity resolution expert. Given two entity records, "
"determine if they refer to the same real-world entity. "
"Respond with MATCH or NO_MATCH followed by a brief explanation."
)
user_msg = f"Entity A:\n{left}\n\nEntity B:\n{right}\n\nDo these refer to the same entity?"
assistant_msg = f"{label}. "
if label == "MATCH":
assistant_msg += "These records share sufficient identifying information to conclude they refer to the same entity."
else:
assistant_msg += "These records contain conflicting information or insufficient overlap to confirm they are the same entity."
return {
"messages": [
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg},
{"role": "assistant", "content": assistant_msg},
]
}Why chat format for SFT?
The SFTTrainer from TRL expects training examples in chat-message format (system, user, assistant). This mirrors how the model will be used at inference time.
The system message establishes the model's role. The user message presents both serialized entities. The assistant message provides the ground-truth classification with a brief rationale, teaching the model to explain its reasoning.
- Save src/data_prep.py.
Unsure about the prompt design?
The prompt template is deliberately simple. At inference time, you will strip the assistant message and let the model generate its own classification. The system prompt anchors the model's behavior, while the user prompt provides a consistent structure it can learn to parse.
help me understand why this chat format works for entity resolution fine-tuning
Create difficulty tiers and run the pipeline
A single F1 score hides important nuance. Some entity pairs are trivially easy (identical names, many shared fields) while others are genuinely hard (cross-script transliterations, missing attributes). Stratifying your evaluation by difficulty will reveal where your model actually struggles.
- Add the difficulty scoring functions below format_pair_as_chat() in src/data_prep.py:
def compute_name_similarity(name_a: str, name_b: str) -> float:
"""Simple Jaccard similarity on character trigrams."""
def trigrams(s):
s = s.lower().strip()
return set(s[i:i+3] for i in range(len(s) - 2))
tri_a = trigrams(name_a)
tri_b = trigrams(name_b)
if not tri_a or not tri_b:
return 0.0
return len(tri_a & tri_b) / len(tri_a | tri_b)
def assign_difficulty(pair: dict) -> str:
"""Assign difficulty tier based on name similarity and property completeness."""
left_name = pair["left"].get("caption", "")
right_name = pair["right"].get("caption", "")
sim = compute_name_similarity(left_name, right_name)
left_props = pair["left"].get("properties", {})
right_props = pair["right"].get("properties", {})
shared_keys = set(left_props.keys()) & set(right_props.keys())
if sim > 0.7 and len(shared_keys) >= 3:
return "easy"
elif sim > 0.3 or len(shared_keys) >= 2:
return "medium"
else:
return "hard"How do the difficulty tiers work?
- Easy: name trigram similarity above 0.7 AND at least 3 shared property keys. These are near-duplicates with plenty of matching evidence.
- Medium: name similarity above 0.3 OR at least 2 shared property keys. These have partial overlap like abbreviations or minor spelling differences.
- Hard: everything else. Cross-script transliterations, missing fields, and conflicting attributes land here.
Character trigram Jaccard similarity is a fast, language-agnostic proxy for name overlap. It captures substring-level matches without needing tokenization.
- Add the main pipeline function and entry point below assign_difficulty():
def prepare_dataset(input_path: str, output_dir: str, max_pairs: int = 5000):
"""Full data preparation pipeline."""
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
pairs = load_pairs(input_path, max_pairs)
random.seed(42)
random.shuffle(pairs)
entity_ids = set()
for p in pairs:
entity_ids.add(p["left"]["id"])
entity_ids.add(p["right"]["id"])
train_pairs, eval_pairs = train_test_split(
pairs, test_size=0.2, random_state=42,
stratify=[p["judgement"] for p in pairs]
)
train_formatted = [format_pair_as_chat(p) for p in train_pairs]
eval_formatted = [format_pair_as_chat(p) for p in eval_pairs]
eval_with_tiers = []
for p in eval_pairs:
formatted = format_pair_as_chat(p)
formatted["difficulty"] = assign_difficulty(p)
formatted["judgement"] = p["judgement"]
eval_with_tiers.append(formatted)
with open(output_path / "train.jsonl", "w", encoding="utf-8") as f:
for item in train_formatted:
f.write(json.dumps(item) + "\n")
with open(output_path / "eval.jsonl", "w", encoding="utf-8") as f:
for item in eval_with_tiers:
f.write(json.dumps(item) + "\n")
print(f"Train: {len(train_formatted)} examples")
print(f"Eval: {len(eval_with_tiers)} examples")
tier_counts = {}
for item in eval_with_tiers:
tier = item["difficulty"]
tier_counts[tier] = tier_counts.get(tier, 0) + 1
print(f"Eval tiers: {tier_counts}")
if __name__ == "__main__":
prepare_dataset(
input_path="data/raw/pairs-20251209.json",
output_dir="data/processed",
max_pairs=5000,
)What does this code do?
The prepare_dataset() function orchestrates the full pipeline. It loads pairs, shuffles them with a fixed seed for reproducibility, then creates a stratified 80/20 train/eval split that preserves the positive/negative ratio.
The eval set gets extra metadata: each record is tagged with its difficulty tier and original judgement label. This lets you slice evaluation metrics by difficulty later. Both sets are written as JSONL (one JSON object per line).
- Save src/data_prep.py.
- Run the data preparation pipeline by running this command:
python src/data_prep.py- You should see output showing the train/eval split counts and the tier distribution:
Train: 4000 examples
Eval: 1000 examples
Eval tiers: {'easy': ..., 'medium': ..., 'hard': ...}- Confirm the output files exist by running:
ls data/processed/- You should see train.jsonl and eval.jsonl in the output.
FileNotFoundError on the input path?
Make sure you are running the command from the project root directory (the folder containing src/ and data/). The script expects data/raw/pairs-20251209.json to exist relative to where you run it.
help me fix this FileNotFoundError when running my data prep script
✔️ Awesome, I've got everything!
Great. Double check that you have saved src/data_prep.py and that both train.jsonl and eval.jsonl were generated in data/processed/.
ⓧ I'd like to double check the full code
Here is the complete src/data_prep.py file:
import json
import random
from pathlib import Path
from sklearn.model_selection import train_test_split
def load_pairs(file_path: str, max_pairs: int = 5000) -> list[dict]:
"""Load entity pairs from line-based JSON file."""
pairs = []
with open(file_path, "r", encoding="utf-8") as f:
for i, line in enumerate(f):
if i >= max_pairs:
break
pair = json.loads(line.strip())
pairs.append(pair)
return pairs
def serialize_entity(entity: dict) -> str:
"""Serialize an entity's properties into a readable text block."""
parts = [f"Name: {entity.get('caption', 'Unknown')}"]
props = entity.get("properties", {})
if "alias" in props:
parts.append(f"Aliases: {', '.join(props['alias'])}")
if "birthDate" in props:
parts.append(f"Birth Date: {', '.join(props['birthDate'])}")
if "nationality" in props:
parts.append(f"Nationality: {', '.join(props['nationality'])}")
if "country" in props:
parts.append(f"Country: {', '.join(props['country'])}")
if "idNumber" in props:
parts.append(f"ID Numbers: {', '.join(props['idNumber'])}")
if "address" in props:
parts.append(f"Address: {', '.join(props['address'][:2])}")
schema = entity.get("schema", "Unknown")
parts.append(f"Type: {schema}")
return "\n".join(parts)
def format_pair_as_chat(pair: dict) -> dict:
"""Convert an entity pair into chat-format training example."""
left = serialize_entity(pair["left"])
right = serialize_entity(pair["right"])
judgement = pair["judgement"]
label = "MATCH" if judgement == "positive" else "NO_MATCH"
system_msg = (
"You are an entity resolution expert. Given two entity records, "
"determine if they refer to the same real-world entity. "
"Respond with MATCH or NO_MATCH followed by a brief explanation."
)
user_msg = f"Entity A:\n{left}\n\nEntity B:\n{right}\n\nDo these refer to the same entity?"
assistant_msg = f"{label}. "
if label == "MATCH":
assistant_msg += "These records share sufficient identifying information to conclude they refer to the same entity."
else:
assistant_msg += "These records contain conflicting information or insufficient overlap to confirm they are the same entity."
return {
"messages": [
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg},
{"role": "assistant", "content": assistant_msg},
]
}
def compute_name_similarity(name_a: str, name_b: str) -> float:
"""Simple Jaccard similarity on character trigrams."""
def trigrams(s):
s = s.lower().strip()
return set(s[i:i+3] for i in range(len(s) - 2))
tri_a = trigrams(name_a)
tri_b = trigrams(name_b)
if not tri_a or not tri_b:
return 0.0
return len(tri_a & tri_b) / len(tri_a | tri_b)
def assign_difficulty(pair: dict) -> str:
"""Assign difficulty tier based on name similarity and property completeness."""
left_name = pair["left"].get("caption", "")
right_name = pair["right"].get("caption", "")
sim = compute_name_similarity(left_name, right_name)
left_props = pair["left"].get("properties", {})
right_props = pair["right"].get("properties", {})
shared_keys = set(left_props.keys()) & set(right_props.keys())
if sim > 0.7 and len(shared_keys) >= 3:
return "easy"
elif sim > 0.3 or len(shared_keys) >= 2:
return "medium"
else:
return "hard"
def prepare_dataset(input_path: str, output_dir: str, max_pairs: int = 5000):
"""Full data preparation pipeline."""
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
pairs = load_pairs(input_path, max_pairs)
random.seed(42)
random.shuffle(pairs)
entity_ids = set()
for p in pairs:
entity_ids.add(p["left"]["id"])
entity_ids.add(p["right"]["id"])
train_pairs, eval_pairs = train_test_split(
pairs, test_size=0.2, random_state=42,
stratify=[p["judgement"] for p in pairs]
)
train_formatted = [format_pair_as_chat(p) for p in train_pairs]
eval_formatted = [format_pair_as_chat(p) for p in eval_pairs]
eval_with_tiers = []
for p in eval_pairs:
formatted = format_pair_as_chat(p)
formatted["difficulty"] = assign_difficulty(p)
formatted["judgement"] = p["judgement"]
eval_with_tiers.append(formatted)
with open(output_path / "train.jsonl", "w", encoding="utf-8") as f:
for item in train_formatted:
f.write(json.dumps(item) + "\n")
with open(output_path / "eval.jsonl", "w", encoding="utf-8") as f:
for item in eval_with_tiers:
f.write(json.dumps(item) + "\n")
print(f"Train: {len(train_formatted)} examples")
print(f"Eval: {len(eval_with_tiers)} examples")
tier_counts = {}
for item in eval_with_tiers:
tier = item["difficulty"]
tier_counts[tier] = tier_counts.get(tier, 0) + 1
print(f"Eval tiers: {tier_counts}")
if __name__ == "__main__":
prepare_dataset(
input_path="data/raw/pairs-20251209.json",
output_dir="data/processed",
max_pairs=5000,
)Your data pipeline is ready. You have 4000 training examples and 1000 evaluation examples tagged with difficulty tiers. Next up, you will load the SmolLM3-3B model with 4-bit quantization and configure the QLoRA adapter for fine-tuning.
Configure QLoRA and Load the Model
Your training data is ready. You have 4000 entity pairs formatted as chat messages in train.jsonl and 1000 tiered evaluation pairs in eval.jsonl.
Now it's time to load the model that will learn from this data. You'll load SmolLM3-3B in 4-bit precision using QLoRA, attach lightweight trainable adapters, and wire everything together into a training pipeline.
In this step, get ready to:
- Load SmolLM3-3B with 4-bit quantization to fit within 16GB VRAM.
- Attach a LoRA adapter targeting attention and MLP layers.
- Configure the SFTTrainer with training hyperparameters and evaluation strategy.
Load SmolLM3-3B with 4-bit quantization
A 3B-parameter model in full bf16 precision needs roughly 6GB just for weights. QLoRA solves this by quantizing the base model to 4-bit NF4 format, reducing its memory footprint to around 1.5GB and leaving ample room for gradients and optimizer states on your RTX 4070.
- Create a new file called src/train.py in your project's src/ folder.
- Add the imports, quantization config, and model loading code:
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
)
from peft import LoraConfig, TaskType, get_peft_model
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
def main():
# Target model: HuggingFace's SmolLM3-3B
model_id = "HuggingFaceTB/SmolLM3-3B"
# 4-bit quantization config for QLoRA
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load tokenizer and set pad token
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Load the model with 4-bit quantization
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quantization_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
# Disable KV cache (incompatible with gradient checkpointing)
model.config.use_cache = False
# Required for QLoRA: enable gradients on input embeddings
model.enable_input_require_grads()What does this code do?
BitsAndBytesConfig tells the model loader to compress all base weights into 4-bit NF4 format. The bnb_4bit_use_double_quant=True flag quantizes the quantization constants themselves, squeezing out a bit more memory savings.
Setting use_cache = False disables the key-value cache that speeds up inference but conflicts with gradient checkpointing during training. The enable_input_require_grads() call ensures gradients flow through the frozen quantized layers to reach the trainable LoRA adapters.
Why QLoRA instead of full-precision LoRA?
Standard LoRA keeps the base model in bf16 (~6GB for 3B params). With optimizer states, activations, and gradients on top, that pushes close to your 16GB VRAM limit. QLoRA compresses the base to ~1.5GB, leaving comfortable headroom for training. The trade-off is a potential 0.5-1% accuracy drop from quantization noise. You'll measure this empirically in the Secret Mission.
- Save src/train.py.
You won't see a change yet. The next chunk adds the LoRA adapter that makes this model trainable.
Seeing an import error?
If Python can't find transformers, peft, or bitsandbytes, confirm your virtual environment is activated. The packages were installed in Step 1.
help me debug an import error with transformers or bitsandbytes in my virtual environment
Configure the LoRA adapter
Parameter-Efficient Fine-Tuning works by freezing the entire base model and injecting small trainable matrices (adapters) into specific layers. With rank 16, each adapter matrix has only 16 rows or columns. That means you train less than 0.2% of the total parameters while still teaching the model your entity resolution task.
- In src/train.py, add the following code directly below the model.enable_input_require_grads() line:
# Configure LoRA: target attention + MLP layers
peft_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
# Apply LoRA adapter and print trainable parameter count
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()What does this code do?
LoraConfig defines which layers receive trainable adapters. The target_modules list covers all four attention projections (Q, K, V, O) plus the three MLP gate layers. Targeting both attention and MLP gives the model enough expressive capacity to learn entity matching patterns.
The rank r=16 controls the dimensionality of each adapter. A higher rank means more parameters and capacity but also more memory. With lora_alpha=32 (2x the rank), the adapter updates are scaled appropriately during training.
get_peft_model() wraps the frozen base model with the LoRA layers. The print_trainable_parameters() call will confirm that only ~0.1-0.2% of total parameters are trainable.
- Save src/train.py.
You won't see output yet since this code runs inside the main() function. The next chunk completes the training setup.
Not sure which lines to target?
SmolLM3-3B uses Grouped Query Attention (GQA) with 4 key-value heads. The target module names (q_proj, k_proj, etc.) match the layer names inside the model's architecture. You can verify them by running print(model) before applying PEFT.
help me understand which target_modules to use for LoRA with SmolLM3-3B
Set up the SFTTrainer and training configuration
The SFTTrainer from TRL handles the full supervised fine-tuning loop. It reads your chat-format JSONL, applies the chat template, and manages gradient accumulation so an effective batch size of 8 fits within your GPU memory.
- In src/train.py, add the following code directly below the model.print_trainable_parameters() line:
# Load the processed training and eval data
dataset = load_dataset(
"json",
data_files={
"train": "data/processed/train.jsonl",
"eval": "data/processed/eval.jsonl",
},
)
# Training hyperparameters
training_args = SFTConfig(
output_dir="./checkpoints/entity-resolution-qlora",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
gradient_checkpointing=True,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.1,
weight_decay=0.01,
bf16=True,
logging_steps=10,
eval_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=50,
save_total_limit=3,
max_seq_length=2048,
packing=False,
report_to="none",
)
# Wire up the trainer with model, data, and config
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["eval"],
processing_class=tokenizer,
peft_config=peft_config,
)
# Launch training and save the adapter
trainer.train()
trainer.model.save_pretrained("./adapters/entity-resolution-qlora")
tokenizer.save_pretrained("./adapters/entity-resolution-qlora")
print("Training complete. Adapter saved to ./adapters/entity-resolution-qlora")
if __name__ == "__main__":
main()What does this code do?
load_dataset reads your processed JSONL files into a HuggingFace Dataset object that the trainer can iterate over.
The SFTConfig sets key hyperparameters. With per_device_train_batch_size=2 and gradient_accumulation_steps=4, the effective batch size is 8. The cosine learning rate scheduler with warmup prevents early instability. Gradient checkpointing trades compute for memory by recomputing activations during the backward pass.
After training completes, save_pretrained writes only the small LoRA adapter weights (tens of MB) to disk. The base model weights are never modified or saved.
- Save src/train.py.
- Confirm the file exists by running:
ls src/train.pyYou should see src/train.py listed in the output.
Seeing a syntax error when you check the file?
Make sure all the code you added is indented inside the def main(): function. Each block after the function definition should be indented by 4 spaces. The only lines at the top level are the imports and the if __name__ == "__main__": guard at the bottom.
help me fix indentation issues in my train.py file
✔️ Awesome, I've got everything!
Great. Double check you've saved src/train.py and move on.
ⓧ I'd like to double check the full code
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
)
from peft import LoraConfig, TaskType, get_peft_model
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
def main():
model_id = "HuggingFaceTB/SmolLM3-3B"
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quantization_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
model.config.use_cache = False
model.enable_input_require_grads()
peft_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
dataset = load_dataset(
"json",
data_files={
"train": "data/processed/train.jsonl",
"eval": "data/processed/eval.jsonl",
},
)
training_args = SFTConfig(
output_dir="./checkpoints/entity-resolution-qlora",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
gradient_checkpointing=True,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.1,
weight_decay=0.01,
bf16=True,
logging_steps=10,
eval_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=50,
save_total_limit=3,
max_seq_length=2048,
packing=False,
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["eval"],
processing_class=tokenizer,
peft_config=peft_config,
)
trainer.train()
trainer.model.save_pretrained("./adapters/entity-resolution-qlora")
tokenizer.save_pretrained("./adapters/entity-resolution-qlora")
print("Training complete. Adapter saved to ./adapters/entity-resolution-qlora")
if __name__ == "__main__":
main()Your training script is fully configured. Next up, you'll launch the training run and monitor how the model learns to resolve entities.
Train the Model
Your training script is ready and your data is prepared. Now it is time to launch the actual fine-tuning run.
The training will process ~4000 entity pairs over 3 epochs, updating only the LoRA adapter weights while keeping the quantized base model frozen. On an RTX 4070, this takes approximately 15-25 minutes.
In this step, get ready to:
- Launch the QLoRA training run and monitor GPU memory usage.
- Verify the saved LoRA adapter and confirm checkpoint size.
Launch the training run
Before you start training, open a second terminal window so you can monitor GPU memory in real time while the training runs.
- Open a second terminal window by pressing Cmd+Space (macOS) or the Windows key (Windows) and typing Terminal.
- In your new terminal, start GPU monitoring by running this command:
nvidia-smi --loop=5What does this command do?
This runs nvidia-smi in a loop every 5 seconds, showing you live GPU memory usage. You will use this to confirm that QLoRA keeps your 16GB VRAM from overflowing during training.
- Switch back to your original terminal where your virtual environment is active.
- Make sure you are in the project root directory (the folder containing src/ and data/).
- Launch the training run by running this command:
python src/train.pyWhat should I see?
The script first prints the trainable parameter count (you should see something like 0.1-0.2% of total parameters). Then training begins, logging the loss every 10 steps and running evaluation every 50 steps.
In your GPU monitoring terminal, watch the memory usage. With QLoRA and gradient checkpointing enabled, peak VRAM usage should stay well under 16GB (typically 4-8GB for the quantized model plus activations).
Training takes 15-25 minutes
This is a good time to watch the training loss decrease and the eval loss stabilize. If you notice eval loss climbing consistently for several evaluation rounds, you can interrupt with Ctrl+C to stop early. The trainer automatically saves checkpoints every 50 steps, so you will not lose progress.
Seeing a CUDA out of memory error?
If training crashes with a memory error, make sure no other GPU processes are running. Close other applications that use the GPU (games, other notebooks, browser hardware acceleration). Run nvidia-smi to check for lingering processes.
If the error persists, reduce per_device_train_batch_size from 2 to 1 in src/train.py and increase gradient_accumulation_steps to 8 to maintain the same effective batch size.
Help me debug my CUDA out of memory error during QLoRA training
When training completes successfully, you should see the final message:
Training complete. Adapter saved to ./adapters/entity-resolution-qloraVerify the saved adapter and record training metadata
A key advantage of LoRA is that only the adapter weights are saved, not the full model. This means your checkpoint is tens of megabytes instead of 6GB+.
- Confirm the adapter files were created by running:
ls ./adapters/entity-resolution-qlora/You should see adapter_config.json, adapter_model.safetensors, and tokenizer files listed.
- Check the adapter size to confirm it is small by running:
du -sh ./adapters/entity-resolution-qlora/The total size should be approximately 50-100MB. Compare that to the ~6GB full model weights you downloaded earlier.
Why is the adapter so small?
QLoRA only trains the low-rank decomposition matrices (rank 16) attached to each target module. The full 3B parameter base model stays frozen and quantized. You only save the delta, which is a tiny fraction of the total parameter count.
At inference time, you load the base model and merge this adapter on top to get the fine-tuned behavior. This makes adapters cheap to store, share, and version-control.
- Record the key training metadata from your terminal output. Note down:
- Stop the GPU monitoring in your second terminal by pressing Ctrl+C.
What training loss values should I expect?
With this dataset size and 3 epochs, you should see training loss decrease from around 1.0-2.0 down to 0.3-0.6 by the end. Eval loss should follow a similar trend but may plateau or slightly increase in the final epoch, which is normal.
If your eval loss starts climbing significantly while training loss keeps dropping, that signals overfitting. The checkpoints saved every 50 steps let you pick an earlier checkpoint if needed.
Training finished but adapter folder is empty?
Check that the ./adapters/entity-resolution-qlora/ path matches what is in src/train.py. The save_pretrained call at the end of the script creates this directory. If you interrupted training early with Ctrl+C, the final save may not have run.
In that case, look in ./checkpoints/entity-resolution-qlora/ for the latest checkpoint-* folder. You can use that checkpoint as your adapter path in the evaluation step instead.
Help me find my saved adapter checkpoint after training
Your fine-tuned adapter is saved and verified. Next up, you will build an evaluation framework that measures how well your model actually performs on entity pairs of varying difficulty.
Evaluate Your Model
In the previous step, you trained a QLoRA adapter and saved it to disk. Now it is time to answer the real question: does your fine-tuned model actually work on entity pairs it has never seen before?
A single F1 score is not enough to understand model behavior in production. You need to know where the model fails. In this step, you will build an evaluation framework that stratifies performance by difficulty tier, revealing whether your model struggles with easy exact matches, medium abbreviation variants, or hard cross-script transliterations.
In this step, get ready to:
- Load the base model and merge the trained LoRA adapter for inference.
- Run predictions on the held-out eval set and compute per-tier precision, recall, and F1.
- Generate a markdown evaluation report saved to your repository.
Load the base model and merge the trained adapter
During training, QLoRA kept the base model frozen in 4-bit and only trained tiny adapter weights. For evaluation, you will load the base model in full bfloat16 precision and merge the adapter directly into the model weights. This gives you a single, standard model that runs inference without any PEFT overhead.
- Create the evaluation script by creating a new file src/evaluate.py and adding the imports and model loading function:
import json
import torch
from pathlib import Path
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from sklearn.metrics import (
precision_score,
recall_score,
f1_score,
confusion_matrix,
classification_report,
)
def load_model(base_model_id: str, adapter_path: str):
"""Load base model and merge LoRA adapter."""
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_path)
model = model.merge_and_unload()
model.eval()
return model, tokenizerWhat does this code do?
The imports bring in PeftModel from the PEFT library for loading your adapter, and scikit-learn's classification metrics for computing precision, recall, and F1.
The load_model function loads SmolLM3-3B in full bfloat16 (no quantization), attaches the LoRA adapter via PeftModel.from_pretrained(), then calls merge_and_unload() to fold the adapter weights directly into the base model. This produces a standard model with no adapter overhead at inference time.
- Add the predict function below load_model to run inference on a single eval example:
def predict(model, tokenizer, messages: list[dict]) -> str:
"""Run inference on a single example (system + user only)."""
prompt_messages = [m for m in messages if m["role"] != "assistant"]
input_text = tokenizer.apply_chat_template(
prompt_messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.1,
do_sample=False,
)
generated = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True,
)
return generated.strip()What does this code do?
The predict function strips out the assistant message from the training example (keeping only system + user), formats it using the tokenizer's chat template, and generates a response. It uses do_sample=False for deterministic greedy decoding, which ensures reproducible evaluation results.
The slicing on outputs[0][inputs["input_ids"].shape[1]:] extracts only the newly generated tokens, discarding the prompt echo.
- Add the extract_prediction function below predict to parse the model's free-text response into a classification label:
def extract_prediction(response: str) -> str:
"""Extract MATCH or NO_MATCH from model response."""
response_upper = response.upper()
if "NO_MATCH" in response_upper or "NO MATCH" in response_upper:
return "negative"
elif "MATCH" in response_upper:
return "positive"
return "unknown"Why check for NO_MATCH first?
The string "MATCH" appears inside "NO_MATCH", so checking for the negative case first prevents false positives. If neither pattern is found, the function returns "unknown" so you can track parse failures separately in your metrics.
Build the evaluation pipeline with tiered metrics
The core evaluation function ties everything together. It loads the eval data, runs predictions on every example, computes overall and per-tier metrics, and generates a formatted report.
- Add the evaluate function and the __main__ block below extract_prediction:
def evaluate(base_model_id: str, adapter_path: str, eval_path: str):
"""Run full evaluation with tiered metrics."""
model, tokenizer = load_model(base_model_id, adapter_path)
eval_data = []
with open(eval_path, "r", encoding="utf-8") as f:
for line in f:
eval_data.append(json.loads(line.strip()))
predictions = []
labels = []
tiers = []
for i, item in enumerate(eval_data):
if i % 50 == 0:
print(f"Evaluating {i}/{len(eval_data)}...")
response = predict(model, tokenizer, item["messages"])
pred = extract_prediction(response)
predictions.append(pred)
labels.append(item["judgement"])
tiers.append(item.get("difficulty", "unknown"))
valid_mask = [p != "unknown" for p in predictions]
valid_preds = [p for p, v in zip(predictions, valid_mask) if v]
valid_labels = [l for l, v in zip(labels, valid_mask) if v]
valid_tiers = [t for t, v in zip(tiers, valid_mask) if v]
report = []
report.append("# Entity Resolution Evaluation Report\n")
report.append(f"Model: {base_model_id} + {adapter_path}")
report.append(f"Eval samples: {len(eval_data)}")
report.append(f"Valid predictions: {len(valid_preds)}")
report.append(f"Parse failures: {len(eval_data) - len(valid_preds)}\n")
report.append("## Overall Metrics\n")
report.append(classification_report(
valid_labels, valid_preds, target_names=["negative", "positive"]
))
report.append("\n## Confusion Matrix\n")
cm = confusion_matrix(valid_labels, valid_preds, labels=["negative", "positive"])
report.append(f" Pred Neg Pred Pos")
report.append(f"Actual Neg {cm[0][0]:>8} {cm[0][1]:>8}")
report.append(f"Actual Pos {cm[1][0]:>8} {cm[1][1]:>8}")
report.append("\n## Per-Tier Metrics\n")
for tier in ["easy", "medium", "hard"]:
tier_preds = [p for p, t in zip(valid_preds, valid_tiers) if t == tier]
tier_labels = [l for l, t in zip(valid_labels, valid_tiers) if t == tier]
if tier_preds:
f1 = f1_score(tier_labels, tier_preds, pos_label="positive")
prec = precision_score(tier_labels, tier_preds, pos_label="positive")
rec = recall_score(tier_labels, tier_preds, pos_label="positive")
report.append(f"### {tier.capitalize()} (n={len(tier_preds)})")
report.append(f" Precision: {prec:.4f}")
report.append(f" Recall: {rec:.4f}")
report.append(f" F1: {f1:.4f}\n")
report_text = "\n".join(report)
print(report_text)
output_path = Path("reports/evaluation_report.md")
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(report_text, encoding="utf-8")
print(f"\nReport saved to {output_path}")
if __name__ == "__main__":
evaluate(
base_model_id="HuggingFaceTB/SmolLM3-3B",
adapter_path="./adapters/entity-resolution-qlora",
eval_path="data/processed/eval.jsonl",
)What does this code do?
- The prediction loop iterates over every eval example, printing progress every 50 samples. Each response is parsed into positive, negative, or unknown.
- The valid_mask filters out parse failures so they do not corrupt your metrics. Parse failure count is tracked separately in the report.
- Overall metrics use scikit-learn's classification_report for a full precision/recall/F1 table.
- Per-tier metrics compute F1, precision, and recall for each difficulty bucket (easy, medium, hard) independently. This reveals where your model is strong and where it struggles.
- The report is saved to reports/evaluation_report.md as a markdown file you can commit to your repository.
Why stratify by difficulty tier?
A single F1 score hides critical failure patterns. A model might achieve 95% overall by acing easy pairs while completely failing on hard cross-script transliterations. In production, those hard pairs are exactly the ones a human analyst would escalate. Per-tier metrics let you report where the model adds value and where it needs a human in the loop.
Run the evaluation and review the report
- Save src/evaluate.py.
- Run the evaluation script from your project root:
python src/evaluate.pyWhat should I see?
The script will print progress updates every 50 samples as it runs inference. This takes several minutes depending on your GPU. Once complete, it prints the full evaluation report to the terminal and saves it to reports/evaluation_report.md.
You should see overall precision, recall, and F1 in the classification report, followed by a confusion matrix and per-tier breakdowns for easy, medium, and hard pairs. The OpenSanctions paper reports a rule-based baseline of 91.33% F1. Your fine-tuned model should beat this baseline, with easy pairs scoring highest and hard pairs scoring lowest.
- Confirm the report was saved by running:
ls reports/- You should see evaluation_report.md in the output.
- Open reports/evaluation_report.md in your editor to review the full report.
Seeing a CUDA out of memory error?
The evaluation script loads SmolLM3-3B in full bfloat16 (not 4-bit), which requires more VRAM than training did. Make sure no other GPU processes are running. Close any notebooks or training scripts that might still hold GPU memory.
If memory is still tight, restart your Python environment to release cached CUDA tensors before running the evaluation.
Why is my evaluation script running out of GPU memory?
✔️ Awesome, I've got everything!
Great. Double check you have saved src/evaluate.py and that reports/evaluation_report.md exists in your project.
ⓧ I'd like to double check the full code
Here is the complete src/evaluate.py for comparison:
import json
import torch
from pathlib import Path
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from sklearn.metrics import (
precision_score,
recall_score,
f1_score,
confusion_matrix,
classification_report,
)
def load_model(base_model_id: str, adapter_path: str):
"""Load base model and merge LoRA adapter."""
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_path)
model = model.merge_and_unload()
model.eval()
return model, tokenizer
def predict(model, tokenizer, messages: list[dict]) -> str:
"""Run inference on a single example (system + user only)."""
prompt_messages = [m for m in messages if m["role"] != "assistant"]
input_text = tokenizer.apply_chat_template(
prompt_messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.1,
do_sample=False,
)
generated = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True,
)
return generated.strip()
def extract_prediction(response: str) -> str:
"""Extract MATCH or NO_MATCH from model response."""
response_upper = response.upper()
if "NO_MATCH" in response_upper or "NO MATCH" in response_upper:
return "negative"
elif "MATCH" in response_upper:
return "positive"
return "unknown"
def evaluate(base_model_id: str, adapter_path: str, eval_path: str):
"""Run full evaluation with tiered metrics."""
model, tokenizer = load_model(base_model_id, adapter_path)
eval_data = []
with open(eval_path, "r", encoding="utf-8") as f:
for line in f:
eval_data.append(json.loads(line.strip()))
predictions = []
labels = []
tiers = []
for i, item in enumerate(eval_data):
if i % 50 == 0:
print(f"Evaluating {i}/{len(eval_data)}...")
response = predict(model, tokenizer, item["messages"])
pred = extract_prediction(response)
predictions.append(pred)
labels.append(item["judgement"])
tiers.append(item.get("difficulty", "unknown"))
valid_mask = [p != "unknown" for p in predictions]
valid_preds = [p for p, v in zip(predictions, valid_mask) if v]
valid_labels = [l for l, v in zip(labels, valid_mask) if v]
valid_tiers = [t for t, v in zip(tiers, valid_mask) if v]
report = []
report.append("# Entity Resolution Evaluation Report\n")
report.append(f"Model: {base_model_id} + {adapter_path}")
report.append(f"Eval samples: {len(eval_data)}")
report.append(f"Valid predictions: {len(valid_preds)}")
report.append(f"Parse failures: {len(eval_data) - len(valid_preds)}\n")
report.append("## Overall Metrics\n")
report.append(classification_report(
valid_labels, valid_preds, target_names=["negative", "positive"]
))
report.append("\n## Confusion Matrix\n")
cm = confusion_matrix(valid_labels, valid_preds, labels=["negative", "positive"])
report.append(f" Pred Neg Pred Pos")
report.append(f"Actual Neg {cm[0][0]:>8} {cm[0][1]:>8}")
report.append(f"Actual Pos {cm[1][0]:>8} {cm[1][1]:>8}")
report.append("\n## Per-Tier Metrics\n")
for tier in ["easy", "medium", "hard"]:
tier_preds = [p for p, t in zip(valid_preds, valid_tiers) if t == tier]
tier_labels = [l for l, t in zip(valid_labels, valid_tiers) if t == tier]
if tier_preds:
f1 = f1_score(tier_labels, tier_preds, pos_label="positive")
prec = precision_score(tier_labels, tier_preds, pos_label="positive")
rec = recall_score(tier_labels, tier_preds, pos_label="positive")
report.append(f"### {tier.capitalize()} (n={len(tier_preds)})")
report.append(f" Precision: {prec:.4f}")
report.append(f" Recall: {rec:.4f}")
report.append(f" F1: {f1:.4f}\n")
report_text = "\n".join(report)
print(report_text)
output_path = Path("reports/evaluation_report.md")
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(report_text, encoding="utf-8")
print(f"\nReport saved to {output_path}")
if __name__ == "__main__":
evaluate(
base_model_id="HuggingFaceTB/SmolLM3-3B",
adapter_path="./adapters/entity-resolution-qlora",
eval_path="data/processed/eval.jsonl",
)You now have a rigorous evaluation framework that goes beyond a single accuracy number. Your per-tier breakdown reveals exactly where your model adds value and where it needs support. Next up, you will tackle the Secret Mission and compare QLoRA against standard LoRA to document the empirical trade-offs.
Secret mission
LoRA vs QLoRA Trade-off Analysis
You built a QLoRA-fine-tuned model. But can you prove it was the right choice? Train a second adapter with standard LoRA and build an empirical comparison that documents the trade-offs with concrete metrics.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources on disk, free up space by removing cached model weights, or delete everything entirely.
Resources you used:
- Project directory on your Desktop (source code, adapters, training data, checkpoints, evaluation reports).
- HuggingFace cached model weights for SmolLM3-3B (~6GB stored in your local HuggingFace cache).
- Python virtual environment (inside your project folder).
Keep everything running
No action needed. Choose this if you plan to continue experimenting with your fine-tuned adapters or running additional evaluations.
- All artifacts remain on your local disk.
- There are no ongoing costs or subscriptions to worry about.
- Your adapters in adapters/entity-resolution-qlora/ and adapters/entity-resolution-lora/ are ready to load anytime.
Pause - I'll come back to this later
Free up disk space by removing the large cached model weights and training checkpoints while keeping your final adapters and code intact.
- Delete the training checkpoints to reclaim disk space by running:
- Your final adapters and source code remain untouched for when you return.
rm -rf checkpoints/- Delete the cached SmolLM3-3B model weights (~6GB) by running:
macOS
rm -rf ~/.cache/huggingface/hub/models--HuggingFaceTB--SmolLM3-3BWindows
rmdir /s /q %USERPROFILE%\.cache\huggingface\hub\models--HuggingFaceTB--SmolLM3-3BWhen you return, the model weights will re-download automatically the next time you run your training or evaluation scripts.
Delete - I don't want to use this again
Remove all project resources and cached weights to start completely fresh.
- Delete the cached SmolLM3-3B model weights from the HuggingFace cache:
macOS
rm -rf ~/.cache/huggingface/hub/models--HuggingFaceTB--SmolLM3-3BWindows
rmdir /s /q %USERPROFILE%\.cache\huggingface\hub\models--HuggingFaceTB--SmolLM3-3B- Delete your entire project folder from the Desktop:
macOS
cd ~/Desktop
rm -rf entity-resolutionWindows
cd ~/Desktop
rmdir /s /q entity-resolutionThis removes all source code, adapters, processed data, checkpoints, reports, and the virtual environment.
Nice Work!
Nice Work!
You just built a complete entity resolution system powered by a fine-tuned Small Language Model. From raw sanctions data spanning 31 countries to a production-ready adapter with tiered evaluation metrics, this project demonstrates the full applied ML lifecycle.
Here's what you accomplished:
- Fine-tuned SmolLM3-3B using QLoRA on a local RTX 4070, training a parameter-efficient adapter on real OpenSanctions entity pairs with multilingual names, missing fields, and cross-script transliterations.
- Designed a tiered evaluation framework that stratifies model performance by difficulty class (easy, medium, hard) and compares against baselines, going beyond a single F1 number.
- Built a data preparation pipeline that serializes messy entity records into chat-format training examples with stratified sampling and entity-level train/eval isolation.
- Secret Mission: Empirically compared LoRA vs QLoRA trade-offs with concrete metrics on training time, peak GPU memory, and per-tier F1, documenting findings in a portfolio-ready README.
Ready to quiz yourself?