Fine-Tune a Job Skill Extractor

Introduction

30 Second Summary

Every job posting buries dozens of skills inside dense paragraphs. Recruiters scan for them manually, and applicant tracking systems guess at them with basic keyword matching.

In this project, you will fine-tune a pre-trained transformer model to automatically extract skills from job posting text using PyTorch and Hugging Face Transformers.

What You'll Build

You'll feed your model a sentence like "Drive our IT compliance agenda and develop our processes" and watch it highlight the exact skill spans, just like a human annotator would.

By the end of this project, you'll have:

• A fine-tuned DistilBERT model trained on 4,800 real English job posting sentences with human-annotated skill labels.
• A working NER inference pipeline you can point at any job description to get extracted skill spans back instantly.
• Training metrics (precision, recall, F1) showing your model improving epoch by epoch on your local Apple Silicon GPU.
• Secret Mission: Swap DistilBERT for a larger model and compare F1 scores to see how model size affects extraction quality.

Are there any prerequisites?

No prior setup is needed. This guide walks you through installing Python, PyTorch, and every library from scratch.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for the specific resume gap you're closing.

Set Up Your Python ML Environment

To fine-tune a transformer model locally, you need a Python environment with GPU-accelerated deep learning libraries.

In this step, you'll install Python, create an isolated virtual environment, and set up PyTorch with Apple Silicon GPU support alongside the Hugging Face Transformers stack.

In this step, get ready to:

• Install Python 3.10+ and create a virtual environment.
• Install PyTorch with MPS GPU support and the Hugging Face ML stack.
• Verify everything works together with a quick smoke test.

Install Python 3.10+ and create a virtual environment

PyTorch and the Hugging Face libraries require Python 3.10 or higher. macOS ships with Python 3.9 via Xcode tools, so you need to confirm you have the right version.

• Press Cmd+Space to open Spotlight search.
• Type Terminal and press Enter to open it.
• Check your Python version by running this command:

python3 --version

Look at the version number printed and pick the tab that matches what you see.

✔️ I see version 3.10 or higher

Your Python version is compatible. You're ready to create your virtual environment.

ⓧ I see an older version

macOS includes Python 3.9 by default, but this project needs 3.10 or higher. You'll install a newer version using Homebrew.

• If you don't have Homebrew yet, install it by running:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Install Python 3.12 by running:

brew install python@3.12

• Verify the new version is available by running:

python3.12 --version

You should see Python 3.12.x printed. Use python3.12 instead of python3 for the rest of this project to ensure you're using the correct version.

Why not just upgrade the system Python?

The system Python at /usr/bin/python3 is managed by macOS. Replacing it can break system tools. Installing via Homebrew gives you a separate binary that doesn't interfere.

ⓧ Command not found

Python isn't installed on your system yet. You'll install it using Homebrew.

• Install Homebrew (the macOS package manager) by running:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Follow any on-screen instructions to add Homebrew to your PATH.
• Install Python 3.12 by running:

brew install python@3.12

• Verify the installation by running:

python3.12 --version

You should see Python 3.12.x printed. Use python3.12 instead of python3 for the rest of this project.

Now you'll create a project folder on your Desktop and set up an isolated virtual environment inside it.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir skill-extractor
cd skill-extractor

• Create and activate a virtual environment by running:

python3 -m venv skill-extractor && source skill-extractor/bin/activate

Your terminal prompt now starts with (skill-extractor), confirming the virtual environment is active.

What is a virtual environment?

A virtual environment is an isolated Python installation that keeps this project's packages separate from your system Python. Everything you install from here on only exists inside this environment.

Don't see (skill-extractor) in your prompt?

Make sure you ran both parts of the command. If the activation failed, try running just the activation part separately:

source skill-extractor/bin/activate

If you used python3.12 in the tabs above, replace python3 with python3.12 in the venv creation command.

Help me troubleshoot my virtual environment activation.

Install PyTorch and the Hugging Face stack

With your virtual environment active, you'll install the ML libraries. PyTorch provides the deep learning engine, and Hugging Face provides the pre-trained models and training tools.

• Install PyTorch and its companion libraries by running:

pip install torch torchvision torchaudio

What are these three packages?

torch is the core PyTorch deep learning framework. torchvision and torchaudio are companion libraries for vision and audio tasks. They share the same MPS backend, so installing all three keeps your environment consistent.

This download is roughly 200MB and may take a minute or two depending on your connection.

• Install the Hugging Face ML packages by running:

pip install "transformers[torch]" datasets evaluate seqeval

What do these packages do?

transformers gives you access to pre-trained models like DistilBERT. datasets loads training data from Hugging Face Hub. evaluate computes metrics. seqeval is the standard metric library for NER tasks.

Now confirm that your M3 Pro's GPU is accessible to PyTorch. Before you run this, what do you think it will print? You're on Apple Silicon with MPS support built in.

• Check that MPS GPU acceleration is available by running:

python -c "import torch; print(torch.backends.mps.is_available())"

You should see True printed, confirming your M3 Pro GPU is accessible for training.

Seeing False instead of True?

MPS requires macOS 14.0 or later. Check your macOS version in the Apple menu under About This Mac. If you're on an older version, update to macOS 14+ first.

If you're on macOS 14+ and still seeing False, make sure you installed PyTorch inside the virtual environment (your prompt should show (skill-extractor)).

Help me fix MPS support on my Mac.

Verify the full stack works together

The final check loads a real model tokenizer from Hugging Face. This confirms that PyTorch, Transformers, and your network connection to Hugging Face Hub all work correctly.

What do you think this command will output? It's loading DistilBERT's tokenizer and asking it to process the phrase "hello world".

• Run this smoke test to load a DistilBERT tokenizer:

python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased'); print(t('hello world'))"

You should see a dictionary printed with keys like input_ids and attention_mask. Those numbers are how the model sees text internally.

What just happened?

You downloaded DistilBERT's vocabulary and tokenizer configuration from Hugging Face Hub. The tokenizer split "hello world" into token IDs that the model can process.

This is the same tokenizer you'll use to prepare job posting data for training.

Seeing a connection error?

The tokenizer downloads model files from the internet. Make sure you have an active internet connection. If you're behind a corporate firewall or VPN, try disconnecting temporarily.

If you see an ImportError, confirm your virtual environment is still active (look for (skill-extractor) in your prompt).

Help me troubleshoot loading a Hugging Face tokenizer.

Your ML environment is locked and loaded. Next up, you'll load a real dataset of job postings with human-annotated skill labels and explore how the data is structured.

Load and Explore the SkillSpan Dataset

Your ML environment is ready. Now you need something to train on: real job posting sentences with human-annotated skill labels.

In this step, you will load the SkillSpan dataset from Hugging Face, explore its structure, and understand the BIO tagging scheme that tells the model which tokens are skills.

In this step, get ready to:

• Load 11,500+ annotated job posting sentences from the SkillSpan dataset.
• Understand how BIO tags mark skill spans at the token level.
• Build the label mappings your model needs for training.

Load the SkillSpan dataset from Hugging Face

The SkillSpan dataset contains 4,800 training sentences from real English job postings. Each sentence is split into tokens, and every token has a human-annotated label indicating whether it is part of a skill span.

• Create a file called train.py in your project folder by running:

touch train.py

• Confirm the file exists by running:

ls

You should see train.py listed in the output.

• Open train.py in your editor and add the following imports and dataset loading code:

import numpy as np
import evaluate
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    TrainingArguments,
    Trainer,
    DataCollatorForTokenClassification,
)

# Load dataset
dataset = load_dataset("jjzha/skillspan")

What does this code do?

The imports bring in everything you will need across the full training pipeline. You will not use all of them yet, but having them at the top keeps the file organized.

The load_dataset("jjzha/skillspan") call downloads the SkillSpan dataset from Hugging Face and splits it into train, validation, and test sets automatically.

• Save train.py and run it to download the dataset:

python train.py

You will see download progress bars as Hugging Face fetches the dataset files. This takes about 10-20 seconds depending on your connection.

Seeing a connection error?

Make sure your virtual environment is still active (you should see (skill-extractor) in your terminal prompt). If not, reactivate it with source skill-extractor/bin/activate.

If the download fails, check your internet connection and retry. The dataset is only 4.36 MB.

Why is my dataset download failing?

Inspect the data structure and label scheme

Now that the dataset is downloaded, you need to understand what is inside each example before you can preprocess it for the model.

• Explore the dataset structure by running this command:

python -c "from datasets import load_dataset; ds = load_dataset('jjzha/skillspan'); print(ds)"

You should see three splits printed: train (4800 rows), validation (3174 rows), and test (3569 rows).

• Now inspect a single example to see the fields by running:

python -c "
from datasets import load_dataset
ds = load_dataset('jjzha/skillspan')
print(ds['train'][53])
"

You should see an output like this, showing the tokens and their corresponding skill tags:

# Expected output (example at index 53):
# {'idx': 53,
#  'tokens': ['Drive', 'our', 'IT', 'compliance', 'agenda', 'and', 'develop', 'our', 'processes'],
#  'tags_skill': ['B', 'I', 'I', 'I', 'I', 'O', 'B', 'I', 'I'],
#  'tags_knowledge': ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],
#  'source': 'house'}

What is the BIO tagging scheme?

Each token in a sentence gets one of three labels:

• B (Beginning) marks the first token of a skill span.
• I (Inside) marks a continuation token within the same skill span.
• O (Outside) marks a token that is not part of any skill.

In the example above, "Drive our IT compliance agenda" is one skill (B-I-I-I-I). The word "and" is O (not a skill). Then "develop our processes" is a second skill (B-I-I). Multi-word skills are common in job postings.

Now you have enough understanding to build the label mappings. These tell the model how to convert between human-readable tag names and the integer IDs it trains on.

• Add the label mapping code below the dataset = load_dataset("jjzha/skillspan") line in train.py:

# Build label list from training data
label_list = ["O", "B", "I"]
id2label = {i: label for i, label in enumerate(label_list)}
label2id = {label: i for label, i in zip(label_list, range(len(label_list)))}

What does this code do?

The label_list defines the three possible tags in order: O=0, B=1, I=2. The model outputs integer predictions, so id2label converts model output (0, 1, 2) back to human-readable tags ("O", "B", "I").

The reverse mapping label2id converts tag strings to integers, which you will use when loading the model configuration in a later step.

• Save train.py and verify the mappings work by running:

python -c "
label_list = ['O', 'B', 'I']
id2label = {i: label for i, label in enumerate(label_list)}
label2id = {label: i for label, i in zip(label_list, range(len(label_list)))}
print('id2label:', id2label)
print('label2id:', label2id)
"

You should see id2label: {0: 'O', 1: 'B', 2: 'I'} and label2id: {'O': 0, 'B': 1, 'I': 2} printed.

Seeing a syntax error?

Make sure you have matching quotes and brackets in train.py. The label list must use straight quotes, not curly/smart quotes that some editors auto-insert.

Help me fix a syntax error in my label mappings code.

Visualize examples to build intuition

Seeing a few more annotated sentences helps you understand how varied skill spans can be. Some are single words ("Python"), others are long phrases ("IT compliance agenda").

• Before you run this, predict: will most tokens in a typical sentence be labeled as skills (B or I), or as non-skills (O)?
• Print three annotated examples by running:

python -c "
from datasets import load_dataset
ds = load_dataset('jjzha/skillspan')
for i in [53, 100, 200]:
    example = ds['train'][i]
    print(f\"\nExample {i}:\")
    for token, tag in zip(example['tokens'], example['tags_skill']):
        marker = f'[{tag}]' if tag != 'O' else '   '
        print(f'  {marker} {token}')
"

You will see that most tokens are labeled O (not a skill). Skill spans are relatively sparse, which is typical for NER tasks. The model needs to learn to pick out the signal from the noise.

Why are most tokens labeled O?

Job posting sentences contain many connecting words, articles, and general verbs that are not skills. Only the specific competencies ("IT compliance", "develop processes") get B/I tags. This class imbalance is normal in token classification tasks.

✔️ Awesome, I've got everything!

Great. Double check you have saved train.py with all the imports, the dataset loading line, and the label mappings.

ⓧ I'd like to double check the full code

Here is the complete train.py so far:

import numpy as np
import evaluate
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    TrainingArguments,
    Trainer,
    DataCollatorForTokenClassification,
)

# Load dataset
dataset = load_dataset("jjzha/skillspan")

# Build label list from training data
label_list = ["O", "B", "I"]
id2label = {i: label for i, label in enumerate(label_list)}
label2id = {label: i for label, i in zip(label_list, range(len(label_list)))}

You now have real labeled data loaded and understand exactly how skills are annotated at the token level. Next up, you will tokenize these sentences for DistilBERT and solve the tricky problem of aligning labels with subword tokens.

Tokenize and Align Labels

In the previous step, you explored the SkillSpan dataset and saw that each sentence has a list of tokens with matching BIO labels. Now you need to feed that data into DistilBERT.

There is a fundamental problem: DistilBERT's tokenizer splits words into subword pieces, but your labels exist at the word level. Without proper alignment, the model trains on garbage labels and learns nothing. In this step, you will solve this alignment problem.

In this step, get ready to:

• Load the DistilBERT tokenizer and discover the subword alignment problem.
• Write a tokenize_and_align_labels function that correctly maps labels to subtokens.
• Apply tokenization across all splits and create a data collator for batching.

Load the DistilBERT tokenizer and see the problem

Before you can tokenize the full dataset, you need to load the tokenizer and understand how it transforms your data.

• In train.py, find the line that says label2id = {label: i for label, i in zip(label_list, range(len(label_list)))}.
• Add the following code below it to load the tokenizer:

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")

What does this code do?

This loads DistilBERT's pretrained tokenizer, which knows how to split text into the subword pieces DistilBERT was trained on. The tokenizer uses WordPiece, meaning it breaks unfamiliar words into smaller known fragments.

Now let's see exactly what the tokenizer does to your data. Before you run this, think about the sample sentence from Step 2: ["Drive", "our", "IT", "compliance", "agenda", "and", "develop", "our", "processes"] has 9 words with 9 matching labels. What do you think happens when the tokenizer splits compliance into multiple subtokens?

• See the mismatch for yourself by running this command in your terminal:

python -c "
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')
tokens = ['Drive', 'our', 'IT', 'compliance', 'agenda', 'and', 'develop', 'our', 'processes']
result = t(tokens, is_split_into_words=True)
print(f'Original words: {len(tokens)}')
print(f'Tokens after tokenizer: {len(result.input_ids)}')
print(f'Actual tokens: {t.convert_ids_to_tokens(result.input_ids)}')
"

You will see output showing that 9 words became 13 tokens. The tokenizer added [CLS] and [SEP] special tokens, and split compliance into subword pieces like comp, ##lian, ##ce.

Your labels array is still 9 items long. If you naively pass 9 labels alongside 13 tokens, the model trains on misaligned garbage. This is the core NLP engineering challenge you need to solve.

Why does this matter?

PyTorch's CrossEntropyLoss expects exactly one label per token. If the lengths do not match, training crashes. Even if you padded to match lengths, the labels would be shifted and the model would learn the wrong patterns.

Write the label alignment function

The fix uses the tokenizer's word_ids() method, which maps every subtoken back to the original word it came from. Special tokens like [CLS] return None.

• In train.py, add the following function below the tokenizer line you just added:

def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(
        examples["tokens"], truncation=True, is_split_into_words=True
    )

    labels = []
    for i, label in enumerate(examples["tags_skill"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:
                label_ids.append(label_list.index(label[word_idx]))
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

What does this function do?

• tokenizer(examples["tokens"], truncation=True, is_split_into_words=True) tokenizes pre-split words into subword pieces, telling the tokenizer the input is already word-level.
• word_ids = tokenized_inputs.word_ids(batch_index=i) returns a list mapping each subtoken back to its original word index. Special tokens get None.
• The loop assigns the real label only to the first subtoken of each word. Subsequent subtokens and special tokens get -100, which PyTorch's loss function ignores automatically.
• label_list.index(label[word_idx]) converts the string BIO tag (like "B") into its integer ID for the model.

• Save train.py.
• Verify the alignment works by running this command:

python -c "
from transformers import AutoTokenizer
from datasets import load_dataset

tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')
dataset = load_dataset('jjzha/skillspan')
example = dataset['train'][53]

result = tokenizer(example['tokens'], is_split_into_words=True)
word_ids = result.word_ids()
tokens = tokenizer.convert_ids_to_tokens(result['input_ids'])

print('Token | Word ID | Label')
for tok, wid in zip(tokens, word_ids):
    if wid is None:
        label = '-100 (ignored)'
    else:
        label = example['tags_skill'][wid]
    print(f'{tok:15} | {str(wid):7} | {label}')
"

You should see a table where each subtoken maps to a word ID, and only the first subtoken of each word shows the real BIO label. Subtokens with None (special tokens) and repeated word IDs get -100 during training.

Seeing a KeyError or IndexError?

Make sure your function reads from examples["tags_skill"], not examples["ner_tags"]. The SkillSpan dataset uses tags_skill as the column name for skill labels.

Also confirm that label_list is defined before the function. The function references it to convert string tags to integers.

Why isn't my tokenize_and_align_labels function working correctly?

Apply tokenization to all splits and create the data collator

Now that the alignment function works on a single example, you need to apply it across all 11,543 sentences in the dataset.

• In train.py, add the following code below your tokenize_and_align_labels function:

# Tokenize all splits
tokenized_dataset = dataset.map(tokenize_and_align_labels, batched=True)

# Data collator
data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)

What does this code do?

• dataset.map(tokenize_and_align_labels, batched=True) applies your alignment function to every example in train, validation, and test splits at once. The batched=True flag processes examples in groups for speed.
• DataCollatorForTokenClassification handles dynamic padding at training time. Instead of padding every sentence to the maximum length in the whole dataset, it pads each batch to the longest sentence in that batch only. This saves memory and speeds up training.

• Save train.py.
• Before you run this, what columns do you expect to see after tokenization? The original dataset had tokens and tags_skill. What should the tokenized version add?
• Verify by running this command:

python -c "
from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorForTokenClassification

label_list = ['O', 'B', 'I']
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')
dataset = load_dataset('jjzha/skillspan')

def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples['tokens'], truncation=True, is_split_into_words=True)
    labels = []
    for i, label in enumerate(examples['tags_skill']):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:
                label_ids.append(label_list.index(label[word_idx]))
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)
    tokenized_inputs['labels'] = labels
    return tokenized_inputs

tokenized = dataset.map(tokenize_and_align_labels, batched=True)
print(tokenized['train'].column_names)
print(f'First example labels (first 15): {tokenized[\"train\"][0][\"labels\"][:15]}')
"

You should see that each split now has input_ids, attention_mask, and labels columns alongside the original data. The labels list will contain integers (0, 1, 2) for real labels and -100 for ignored positions.

Map is slow or seems stuck?

The first run downloads the tokenizer model and processes all 11,543 examples. This takes 15-30 seconds depending on your machine. Subsequent runs use the cached tokenizer and go faster.

If you see a PicklingError, make sure the tokenizer variable is defined at module level (not inside a function or if __name__ block).

Why is my dataset.map call failing?

✔️ Awesome, I've got everything!

Great. Double check you have saved train.py with all three additions: the tokenizer load, the alignment function, and the map plus data collator lines.

ⓧ I'd like to double check the full code

Your train.py should now contain these sections in order. Scroll up and confirm each one is present:

• Imports (numpy, evaluate, datasets, transformers).
• Dataset loading with load_dataset("jjzha/skillspan").
• Label list and mappings (label_list, id2label, label2id).
• Tokenizer loading with AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased").
• The tokenize_and_align_labels function (20 lines, using word_ids() and -100 for ignored tokens).
• tokenized_dataset = dataset.map(tokenize_and_align_labels, batched=True).
• data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer).

Your data is now fully preprocessed and ready for training. Every sentence has aligned token IDs, attention masks, and integer labels that the model can learn from. Next up, you will load DistilBERT with a classification head and train it to extract skills.

Fine-Tune DistilBERT on Skill Extraction

In the last step, you solved the tricky subword alignment problem so your labels line up perfectly with DistilBERT's tokens. Now the data is ready for the main event: teaching a pre-trained model to recognize skills.

DistilBERT already understands English grammar and word meaning from its original training on Wikipedia and books. In this step, you will add a classification head on top, define how to measure performance, and run the training loop that teaches it to extract skills from job postings.

In this step, get ready to:

• Define evaluation metrics so you can measure your model's skill-extraction accuracy.
• Load DistilBERT with a fresh token classification head wired for your three labels.
• Run the full training loop and watch F1 climb across five epochs on your MPS GPU.

Define the evaluation metrics

NER models need a specialized metric called seqeval that evaluates entity spans as a whole, not individual tokens. A model that tags half a skill correctly still gets penalized because partial spans are wrong.

• In train.py, find the line that says data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer).
• Below that line, add the evaluation metric and the compute_metrics function by pasting this code:

# Load evaluation metric
seqeval_metric = evaluate.load("seqeval")


def compute_metrics(p):
    predictions, labels = p
    predictions = np.argmax(predictions, axis=2)

    true_predictions = [
        [label_list[pred] for (pred, lab) in zip(prediction, label) if lab != -100]
        for prediction, label in zip(predictions, labels)
    ]
    true_labels = [
        [label_list[lab] for (pred, lab) in zip(prediction, label) if lab != -100]
        for prediction, label in zip(predictions, labels)
    ]

    results = seqeval_metric.compute(
        predictions=true_predictions, references=true_labels
    )
    return {
        "precision": results["overall_precision"],
        "recall": results["overall_recall"],
        "f1": results["overall_f1"],
        "accuracy": results["overall_accuracy"],
    }

What does this code do?

evaluate.load("seqeval") loads a span-level evaluation metric designed for NER tasks. It scores complete entity spans, not individual tokens.

The compute_metrics function converts raw model predictions (logits) into label names, strips out any token marked -100 (the ignored subtokens from alignment), and passes the cleaned predictions to seqeval. It returns precision, recall, F1, and accuracy after every evaluation epoch.

Load DistilBERT with a token classification head

DistilBERT's base model was trained to predict masked words, not classify tokens. You need to add a classification layer on top that maps each token's hidden state to one of your three labels (O, B, I).

• Below the compute_metrics function you just added, paste this model loading code:

# Load model
model = AutoModelForTokenClassification.from_pretrained(
    "distilbert/distilbert-base-uncased",
    num_labels=len(label_list),
    id2label=id2label,
    label2id=label2id,
)

What does this code do?

AutoModelForTokenClassification.from_pretrained downloads the pre-trained DistilBERT weights and attaches a fresh linear classification head on top. The num_labels=3 argument tells it to output three classes (O, B, I) for each token.

The id2label and label2id dictionaries attach human-readable names to each class index so the model's output is interpretable.

• Save train.py and run it to confirm the model loads successfully:

python train.py

The script will reload the dataset, tokenize it, then load the model. You will see a warning about newly initialized weights. This is expected.

Why the warning about initialized weights?

The classification head you just added is brand new. Its weights are random because it has never been trained. The warning is telling you exactly that: the body of DistilBERT is loaded from pre-training, but the classifier layer on top needs to be trained from scratch on your skill data.

Seeing an import error or download failure?

If you see ModuleNotFoundError, confirm your virtual environment is active. Run source skill-extractor/bin/activate and try again.

If the model download hangs or fails, check your internet connection. The download is roughly 260 MB for DistilBERT.

Help me debug my model loading error

Configure training and run the training loop

With the model loaded, you need to tell Hugging Face Transformers how to train it: learning rate, batch size, number of epochs, and where to save checkpoints.

• Below the model loading code, add the training configuration:

# Training arguments
training_args = TrainingArguments(
    output_dir="./skill-extractor-output",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

What do these training arguments control?

• learning_rate=2e-5 is the standard fine-tuning rate for transformer models. Too high and you destroy the pre-trained knowledge.
• num_train_epochs=5 means the model sees every training sentence five times.
• eval_strategy="epoch" runs your compute_metrics function after every full pass through the data, so you can track improvement.
• load_best_model_at_end=True automatically keeps the checkpoint with the highest eval score as your final model.

• Now add the Trainer, the training call, and the save logic below the training arguments:

# Create Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    processing_class=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

# Train
trainer.train()

# Save the final model
trainer.save_model("./skill-extractor-output")
tokenizer.save_pretrained("./skill-extractor-output")

print("Training complete! Model saved to ./skill-extractor-output")

What does the Trainer do?

The Trainer class wraps the entire PyTorch training loop: forward pass, loss computation, backpropagation, optimizer step, and evaluation. It automatically detects your MPS GPU and runs training there without any extra configuration.

After training completes, trainer.save_model() writes the fine-tuned weights to disk. You also save the tokenizer alongside it so inference works as a self-contained package.

• Save train.py.

Before you run the training, think about this: the model's classification head started with random weights. Do you expect the F1 score to be high or low in the first epoch? What about by epoch 5?

• Run the full training script:

python train.py

Training takes approximately 3-5 minutes on Apple Silicon. You will see a progress bar and a table of metrics printed after each epoch.

Watch the eval_f1 column. It starts low (around 0.3-0.5 after the first epoch) and climbs to roughly 0.6-0.7+ by epoch 5. That climb is your model learning what a skill span looks like.

What does improving F1 actually mean?

F1 combines precision ("of the spans I predicted as skills, how many were correct?") and recall ("of all the real skills in the text, how many did I find?"). A rising F1 means your model is both finding more real skills AND making fewer false claims.

Training seems stuck or slow?

If you see UserWarning: MPS warnings about unsupported operations, the Trainer is falling back to CPU for those ops. Set the environment variable before running:

PYTORCH_ENABLE_MPS_FALLBACK=1 python train.py

If training is significantly slower than 5 minutes, ensure no other heavy processes are consuming your GPU memory.

Help me debug slow training on my Apple Silicon Mac

✔️ Awesome, I've got everything!

Great. Confirm you see the final line: Training complete! Model saved to ./skill-extractor-output.

ⓧ I'd like to double check the full code

Your train.py should now contain these sections in order. Scroll up and confirm each is present:

• Imports (numpy, evaluate, load_dataset, transformers classes).
• Dataset loading with load_dataset("jjzha/skillspan").
• Label list, id2label, and label2id mappings.
• Tokenizer loaded from distilbert/distilbert-base-uncased.
• tokenize_and_align_labels function definition.
• tokenized_dataset created via .map().
• data_collator using DataCollatorForTokenClassification.
• seqeval_metric loaded via evaluate.load("seqeval").
• compute_metrics function returning precision, recall, f1, accuracy.
• Model loaded with AutoModelForTokenClassification.from_pretrained.
• training_args with TrainingArguments.
• Trainer created, trainer.train() called, model and tokenizer saved.
• Final print("Training complete! Model saved to ./skill-extractor-output") statement.

Your model just went from knowing nothing about skills to correctly extracting them from job text. Next up, you will point it at real job postings and see what it pulls out.

Run Inference on Real Job Postings

Training metrics like F1 and precision are useful for measuring progress, but they stay abstract until you see the model work on real text.

In this step, you'll load your fine-tuned model into a ready-to-use inference pipeline, feed it real job posting sentences, and compare its output to a generic NER model that has no concept of skills.

In this step, get ready to:

• Load your trained model into a Hugging Face inference pipeline.
• Extract skills from multiple job posting sentences.
• Compare your fine-tuned model to a generic NER model to see the value of fine-tuning.

Load your trained model into a pipeline

The Hugging Face Transformers library provides a pipeline function that wraps your model, tokenizer, and post-processing into a single callable object.

• Create a new file called inference.py in your project folder by running:

touch inference.py

• Open inference.py in your editor and add the following code to load your fine-tuned model as an inference pipeline:

from transformers import pipeline

# Load the fine-tuned model
skill_extractor = pipeline(
    "token-classification",
    model="./skill-extractor-output",
    aggregation_strategy="simple",
)

What does this code do?

The pipeline function loads your saved model and tokenizer from ./skill-extractor-output and wraps them into a callable that handles tokenization, inference, and decoding in one step.

The aggregation_strategy="simple" argument tells the pipeline to merge subword tokens back into full words. Without it, you would see individual subword pieces like comp and ##liance instead of the complete word compliance.

• Save inference.py. The next chunk adds test sentences that use this pipeline.

Test on real job posting text

Now you'll feed your model four sentences that could appear in any job posting and see which words it identifies as skills.

• Add the following test sentences and inference loop below the pipeline code in inference.py:

# Test sentences
test_sentences = [
    "We need someone with strong Python and SQL skills",
    "Drive our IT compliance agenda and develop our processes",
    "Experience with project management and stakeholder communication required",
    "Must have 3+ years of experience in machine learning and data analysis",
]

for sentence in test_sentences:
    print(f"\nInput: {sentence}")
    results = skill_extractor(sentence)
    skills = [r["word"] for r in results if r["score"] > 0.5]
    print(f"Extracted skills: {skills}")

What does this code do?

Each sentence is passed through your skill_extractor pipeline. The pipeline returns a list of detected entities with their text (word), confidence score, and entity label.

The list comprehension filters results to only keep predictions with a confidence score above 0.5. This removes low-confidence noise and shows only the skills the model is reasonably sure about.

• Save inference.py.

Before you run this, what do you think your model will extract from the sentence "We need someone with strong Python and SQL skills"? Will it tag "Python" and "SQL" individually, or the entire phrase?

• Run your inference script:

python inference.py

You should see each sentence printed with a list of extracted skills. The model identifies multi-word spans like "IT compliance agenda" and "project management" as single skills, matching how they were annotated in the training data.

Seeing an empty list for every sentence?

• Make sure ./skill-extractor-output exists and contains your trained model files. Run ls ./skill-extractor-output to confirm you see model.safetensors and config.json.
• If you restarted your terminal, make sure your virtual environment is active: source skill-extractor/bin/activate.
• If scores are all below 0.5, try lowering the threshold to 0.3 temporarily to see if the model is predicting anything at all.

Why are my skill extractions empty or wrong?

✔️ Awesome, I've got everything!

Great. Double check you've saved inference.py and move on to the comparison below.

ⓧ I'd like to double check the full code

from transformers import pipeline

# Load the fine-tuned model
skill_extractor = pipeline(
    "token-classification",
    model="./skill-extractor-output",
    aggregation_strategy="simple",
)

# Test sentences
test_sentences = [
    "We need someone with strong Python and SQL skills",
    "Drive our IT compliance agenda and develop our processes",
    "Experience with project management and stakeholder communication required",
    "Must have 3+ years of experience in machine learning and data analysis",
]

for sentence in test_sentences:
    print(f"\nInput: {sentence}")
    results = skill_extractor(sentence)
    skills = [r["word"] for r in results if r["score"] > 0.5]
    print(f"Extracted skills: {skills}")

Compare pre-trained vs fine-tuned behavior

Your model extracts skills because you taught it what skills look like. A generic NER model trained on news articles has no concept of "skills" as an entity category.

• Run this command to load a generic NER model and test the same sentence your skill extractor just processed:

python -c "
from transformers import pipeline
generic_ner = pipeline('token-classification', model='dslim/bert-base-NER', aggregation_strategy='simple')
results = generic_ner('We need someone with strong Python and SQL skills')
print('Generic NER results:')
for r in results:
    print(f'  {r[\"word\"]} -> {r[\"entity_group\"]} (score: {r[\"score\"]:.2f})')
"

What should I see?

The generic dslim/bert-base-NER model was trained on news articles to recognize people, organizations, locations, and miscellaneous entities. It has no label for "skill" at all.

You might see it tag "Python" as a miscellaneous entity or miss it entirely. It will never identify "IT compliance agenda" or "project management" as skills because that category does not exist in its training data. Your fine-tuned model does exactly this because you taught it with 4,800 labeled job posting sentences.

Model download taking a long time?

The dslim/bert-base-NER model is approximately 430 MB. On a slower connection this may take a minute or two. The download only happens once. Subsequent runs use the cached version.

Help me troubleshoot loading the generic NER model

This comparison is the core takeaway. A pre-trained model knows language structure but not your task. Fine-tuning adds the domain knowledge that makes it useful for skill extraction.

Your skill extractor is working. You've gone from abstract training metrics to a model you can point at any job description and get skill spans back. Next up, take on the Secret Mission and see if a bigger model can beat your DistilBERT's F1 score.

Secret mission

Swap in a Larger Model and Compare

Your DistilBERT model works, but DistilBERT is a compressed version of the full BERT model. What happens if you throw more parameters at the same task? Swap in BERT-base, retrain with identical settings, and find out whether the extra compute buys you better skill extraction.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your machine, so there are no ongoing cloud costs. Decide whether to keep your resources on disk, pause them to come back later, or delete them entirely.

Resources you used:

• skill-extractor virtual environment (contains PyTorch, Transformers, and all pip packages).
• train.py, inference.py, and train_bert.py Python scripts.
• skill-extractor-output directory (fine-tuned DistilBERT model, ~260MB).
• skill-extractor-bert-output directory (fine-tuned BERT-base model, ~440MB).
• Hugging Face cached models and datasets stored in ~/.cache/huggingface/.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your fine-tuned models or want to try new inference examples.

• Your virtual environment stays on disk ready to reactivate at any time.
• Your trained models remain in their output directories for immediate inference.
• The Hugging Face cache keeps downloaded base models available for future projects without re-downloading.

Pause - I'll come back to this later

Deactivate your virtual environment to free up terminal state. All files remain on disk for when you return.

• Deactivate the virtual environment by running:

deactivate

• To resume later, navigate back to your project folder and run:

source skill-extractor/bin/activate

Delete - I don't want to use this again

Remove all project files, model outputs, and the virtual environment to reclaim disk space.

• Deactivate the virtual environment first by running:

deactivate

• Delete the virtual environment, model output directories, and script files by running:

rm -rf skill-extractor skill-extractor-output skill-extractor-bert-output train.py inference.py train_bert.py

• Clear the Hugging Face cache to remove downloaded base models and datasets by running:

rm -rf ~/.cache/huggingface/hub

What does this remove?

The ~/.cache/huggingface/hub directory stores all base models and tokenizers downloaded by Hugging Face. Removing it frees several gigabytes but means any future Hugging Face project will need to re-download its models.

• Confirm everything is removed by running:

ls

You should no longer see any of the project files or directories in the output.

Nice Work!

Nice Work!

You just built a production-grade skill extraction model from scratch using PyTorch and Hugging Face Transformers.

You've learned how to:

• Built a complete NER fine-tuning pipeline at the framework level, from loading a pre-trained model to saving a trained checkpoint.
• Fine-tuned DistilBERT on real job postings with human-annotated skill labels and evaluated performance with seqeval metrics (precision, recall, F1).
• Solved subword tokenization alignment, mapping word-level BIO labels to subword tokens so the model trains on correct labels.
• Secret Mission: Compared DistilBERT vs BERT-base to measure the accuracy vs. efficiency trade-off firsthand.

Ready to quiz yourself?