Build a GPT Language Model From Scratch

Build a GPT-style language model that generates Shakespeare-style text.

Introduction

30 Second Summary

Every time you ask ChatGPT a question, a language model is predicting the next word in the response, one token at a time. The architecture behind that prediction is surprisingly learnable.

In this project, you will build a GPT-style language model from scratch using Python and PyTorch, train it on Shakespeare's collected works, and generate new text that mimics his style.

What You'll Build

You will feed your trained model a prompt like "ROMEO:" and watch it generate new Shakespeare-style dialogue, complete with character names, dramatic structure, and pentameter-ish phrasing.

By the end of this project, you'll have:

  • A working GPT language model (~10M parameters) that generates Shakespeare-style text after a few minutes of training on your laptop.
  • A concrete understanding of self-attention, transformer blocks, and autoregressive generation through code you wrote yourself.
  • A single Python file you can modify to experiment with model size, training data, and generation settings.
  • Secret Mission: Add temperature and top-k sampling to control how creative your model's output is.

Are there any prerequisites?

You should be comfortable with Python (classes, functions, loops) and have basic math intuition (matrix multiplication, probability). No prior deep learning or PyTorch experience required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. Understanding how a language model works from the inside out is the difference between being a user of AI and being a builder of AI.

Set Up Your Environment

To build your GPT from scratch, you need a deep learning framework that handles the math of neural networks. That framework is PyTorch.

In this step, you'll install PyTorch, download Shakespeare's collected works as training data, and write the tokenization code that converts raw text into numbers your model can process.

In this step, get ready to:
  • Install PyTorch, the deep learning framework.
  • Download the Tiny Shakespeare dataset as training data.
  • Write a character-level tokenizer that converts text to numbers and back.
Install PyTorch

PyTorch provides the building blocks for neural networks: tensors, layers, optimizers, and automatic differentiation. Everything you build in this project depends on it.

  • Install PyTorch by running this command in your terminal:
pip3 install torch

What does this command do?

pip3 install torch downloads PyTorch from the Python Package Index and installs it into your Python environment. On macOS, PyTorch runs on CPU by default, which is all you need for this project's 10M-parameter model.

  • Verify the installation by running:
python3 -c "import torch; print(torch.__version__)"

You should see a version number like 2.x.x printed in your terminal.

Seeing an error instead of a version number?

  • If you see command not found: pip3, your Python installation may not include pip. Try running python3 -m pip install torch instead.
  • If you see a permission error, add --user to the end of the install command: pip3 install torch --user.
  • If the import fails with ModuleNotFoundError, make sure you're using the same Python that pip installed to. Run which python3 and which pip3 to confirm they point to the same location.

Still stuck? Help me troubleshoot my PyTorch installation on macOS.

Create your project and download the dataset

Your model needs training data to learn from. You'll use the Tiny Shakespeare dataset, a single text file containing about 40,000 lines of Shakespeare's plays.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir gpt-from-scratch && cd gpt-from-scratch
  • Download the Tiny Shakespeare dataset by running:
curl -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
  • Confirm the download by running:
ls

You should see input.txt listed in the output.

Don't see input.txt?

  • If curl printed an error about the URL, check for typos in the command. Copy it exactly as shown above.
  • If you see a different folder's contents, make sure you ran cd ~/Desktop first, then mkdir gpt-from-scratch && cd gpt-from-scratch.

Need a hand? Help me download the Shakespeare dataset into my project folder.

Write the data loading and tokenization code

Before your model can learn from Shakespeare, it needs the text converted into numbers. A character-level tokenizer maps each unique character (letters, punctuation, spaces) to an integer, and back again.

  • Create a new file called gpt.py in your gpt-from-scratch folder and add the following code:
import torch
import torch.nn as nn
from torch.nn import functional as F

torch.manual_seed(1337)

# Load data
with open('input.txt', 'r', encoding='utf-8') as f:
    text = f.read()

# Character-level tokenizer
chars = sorted(list(set(text)))
vocab_size = len(chars)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])

What does this code do?

  • torch.manual_seed(1337) locks the random number generator so your results are reproducible every time you run the script.
  • The file read loads the entire Shakespeare text into a single string variable called text.
  • chars extracts every unique character from the text and sorts them alphabetically. This is your vocabulary.
  • stoi (string-to-integer) maps each character to a number. itos (integer-to-string) maps numbers back to characters.
  • encode converts a string into a list of integers. decode converts a list of integers back into a string.
  • Now add the training/validation split and print statements below the tokenizer code:
# Train/val split
data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]

print(f'vocab size: {vocab_size}')
print(f'sample encoding: {encode(text[:20])}')

What does this code do?

  • torch.tensor(encode(text), dtype=torch.long) encodes the entire Shakespeare text as a PyTorch tensor of integers. This is the format neural networks can process.
  • The first 90% of the data becomes train_data (what the model learns from). The remaining 10% becomes val_data (what you use to check if the model is overfitting).
  • The print statements confirm the tokenizer works by showing the vocabulary size and how the first 20 characters look as integers.
  • Save gpt.py.

Before you run this, what do you think the vocabulary size will be? Shakespeare uses letters, punctuation, spaces, and newlines. Take a guess at how many unique characters that adds up to.

  • Run your script to verify everything works:
python3 gpt.py

You should see two lines printed: the vocabulary size (the total number of unique characters in the dataset) and a list of integers showing how the first 20 characters of the text are encoded.

Seeing a FileNotFoundError?

  • If you see FileNotFoundError: [Errno 2] No such file or directory: 'input.txt', your terminal is not in the gpt-from-scratch folder. Run ls to see what files are in your current directory and cd ~/Desktop/gpt-from-scratch to get back to the right place.
  • If you see a SyntaxError, double-check that your code matches the snippet above exactly. Pay close attention to colons, indentation, and quotes.

Need help? Help me debug my gpt.py data loading and tokenization code.

✔️ Awesome, I've got everything!

Great. Double check you've saved gpt.py and that your terminal shows the vocab size and sample encoding output.

ⓧ I'd like to double check the full code

Here is the complete gpt.py file as it should look at the end of this step:

import torch
import torch.nn as nn
from torch.nn import functional as F

torch.manual_seed(1337)

# Load data
with open('input.txt', 'r', encoding='utf-8') as f:
    text = f.read()

# Character-level tokenizer
chars = sorted(list(set(text)))
vocab_size = len(chars)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])

# Train/val split
data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]

print(f'vocab size: {vocab_size}')
print(f'sample encoding: {encode(text[:20])}')

Your environment is ready and your tokenizer is working. Next up, you'll build your first language model: a simple bigram that predicts the next character using only the current one.

Build a Bigram Model

Your data pipeline is ready. You have Shakespeare encoded as numbers with a clean train/val split.

Now you will build the simplest possible language model: a bigram model. This model predicts the next character using ONLY the current character. It has no memory of anything that came before. The question is: how well can a model generate text when it has zero context?

In this step, get ready to:
  • Build a batch sampling function that feeds training data to the model.
  • Create a BigramLanguageModel class with forward and generate methods.
  • Train the model for 1000 steps and generate text from it.
Add hyperparameters and a batch sampling function

Before building the model, you need a way to feed training data to it in small random batches. You also need a few hyperparameters that control how training works.

  • In gpt.py, find the line torch.manual_seed(1337) at the top of your file.
  • Add the following hyperparameters ABOVE that line (right after your imports):
batch_size = 32  # how many sequences to process in parallel
block_size = 8   # maximum context length for predictions
learning_rate = 1e-3
device = 'cpu'

What do these hyperparameters control?

  • batch_size sets how many independent text chunks the model trains on simultaneously. Larger batches give more stable learning.
  • block_size is the maximum number of characters the model can look at when predicting the next one. For the bigram model this barely matters (it only looks at one character anyway), but you will increase it later.
  • device tells PyTorch where to run computations. CPU is fine for this model size.
  • Also remove the two print statements at the bottom of your file (the print(f'vocab size: {vocab_size}') and print(f'sample encoding: ...') lines). You no longer need those checks.
  • Now add the batch sampling function at the very bottom of your file, below the val_data line. Create this function by pasting the following code:
def get_batch(split):
    # Select training or validation data
    d = train_data if split == 'train' else val_data
    # Pick random starting positions for each batch item
    ix = torch.randint(len(d) - block_size, (batch_size,))
    # Stack input sequences and target sequences
    x = torch.stack([d[i:i+block_size] for i in ix])
    y = torch.stack([d[i+1:i+block_size+1] for i in ix])
    x, y = x.to(device), y.to(device)
    return x, y

What does get_batch do?

This function grabs random chunks of text from your dataset. Each chunk is block_size characters long. The input x is a sequence of characters, and the target y is the same sequence shifted by one position.

For example, if x is [H, e, l, l, o, ,, w] then y is [e, l, l, o, ,, w, o]. The model learns to predict each next character.

  • Save gpt.py.
  • Verify the batch function works by running this in your terminal:
python3 -c "exec(open('gpt.py').read()); xb, yb = get_batch('train'); print(xb.shape, yb.shape)"

You should see torch.Size([32, 8]) torch.Size([32, 8]) printed. This confirms you are getting 32 sequences of 8 characters each.

Seeing a NameError or FileNotFoundError?

  • Make sure you removed the old print statements. If they reference variables that no longer exist at that point, the file will error.
  • Confirm input.txt is in the same folder as gpt.py.
  • Check that the hyperparameters (batch_size, block_size, device) are defined before get_batch uses them.

Help me debug my get_batch function

Build the BigramLanguageModel class

A bigram model is the simplest neural language model. It uses a single embedding table where each character has a row of scores predicting what character comes next. No context from previous characters is used.

  • Add the following class below your get_batch function in gpt.py:
class BigramLanguageModel(nn.Module):
    def __init__(self):
        super().__init__()
        # Each token directly predicts the next token via a lookup table
        self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)

    def forward(self, idx, targets=None):
        # idx shape: (B, T) where B=batch, T=time/sequence
        logits = self.token_embedding_table(idx)
        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)
        return logits, loss

    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            logits, loss = self(idx)
            # Focus on the last time step only
            logits = logits[:, -1, :]
            # Convert to probabilities
            probs = F.softmax(logits, dim=-1)
            # Sample one token from the distribution
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

What does this class do?

  • token_embedding_table is a 65x65 matrix (one row per character in the vocabulary). Row i contains the raw scores for what character is likely to follow character i.
  • forward looks up scores for each input character, then computes cross-entropy loss against the actual next characters. This loss measures how wrong the predictions are.
  • generate produces new text one character at a time. It gets the scores, converts them to probabilities with softmax, randomly samples one character, appends it, and repeats.

The key limitation: the embedding table only sees the CURRENT character. Character R always produces the same probability distribution regardless of whether the previous characters were ROMEO: or RATHER.

  • Save gpt.py.
Train the model and generate text

Now wire everything together. You will instantiate the model, train it for 1000 steps with the AdamW optimizer, and then ask it to generate new text.

  • Add the following code at the very bottom of gpt.py, below the BigramLanguageModel class:
model = BigramLanguageModel()
model = model.to(device)

# Use AdamW optimizer for training
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

for iter in range(1000):
    # Print loss every 100 steps to track progress
    if iter % 100 == 0:
        xb, yb = get_batch('train')
        logits, loss = model(xb, yb)
        print(f'step {iter}: loss {loss.item():.4f}')
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

# Generate text starting from a newline character (index 0)
context = torch.zeros((1, 1), dtype=torch.long, device=device)
print(decode(model.generate(context, max_new_tokens=300)[0].tolist()))

What does the training loop do?

  • Each iteration grabs a random batch, computes how wrong the model's predictions are (loss), then adjusts the embedding table to make better predictions next time.
  • optimizer.zero_grad() clears old gradients. loss.backward() computes new gradients. optimizer.step() updates the weights.
  • After training, generation starts from a newline character (encoded as 0) and samples 300 characters one at a time.
  • Save gpt.py.

Before you run this, what do you think the generated text will look like? The model can only look at one character at a time. Will it produce real sentences?

  • Run the script:
python3 gpt.py

You should see loss values printed every 100 steps, starting around 4.1 and dropping to roughly 2.5. After training, 300 characters of generated text appear.

The generated text contains some real short words (like the, of, I) mixed with random-looking fragments. It looks like someone mashed keys on a keyboard that happens to be biased toward English letter patterns.

Why is the output so incoherent?

This is the fundamental limitation of the bigram approach. When the model sees the character t, it always produces the same probability distribution for the next character regardless of what came before. It cannot learn that th is usually followed by e in the context of the but by o in the context of thou.

To generate coherent text, the model needs to look at ALL previous characters and decide which ones matter for the current prediction. That mechanism is called self-attention.

Script running but loss not decreasing?

  • Confirm learning_rate = 1e-3 is set correctly (that is 0.001, not 0.01 or 0.0001).
  • Make sure optimizer.zero_grad(set_to_none=True) appears BEFORE loss.backward() in the loop body.
  • Check that you are calling optimizer.step() after loss.backward(). Without it, no weights update.

Help me debug my training loop

✔️ Awesome, I've got everything!

Great. Double check you have saved gpt.py and your training ran successfully with loss dropping to around 2.5.

ⓧ I'd like to double check the full code

Your gpt.py should now contain these sections in order. Scroll through your file and confirm each one is present:

  • Imports (torch, torch.nn, torch.nn.functional).
  • Hyperparameters (batch_size = 32, block_size = 8, learning_rate = 1e-3, device = 'cpu').
  • torch.manual_seed(1337) followed by data loading and the character-level tokenizer.
  • Train/val split (train_data and val_data).
  • get_batch function.
  • BigramLanguageModel class with __init__, forward, and generate methods.
  • Model instantiation, optimizer, and the 1000-step training loop.
  • Generation from a newline character, printing 300 decoded tokens.

Make sure you do NOT have the old print(f'vocab size: ...') and print(f'sample encoding: ...') lines anywhere in the file.

You have built the simplest possible language model and seen its ceiling. The loss reached about 2.5 but the text is still a mess. Next up, you will implement self-attention so every character can look at the full history and make much better predictions.

Implement Self-Attention

The bigram model proved a point: with no context, character predictions are largely random. Each character is generated using only the single character immediately before it.

Self-attention solves this by letting every token in the sequence look at all previous tokens and decide which ones are relevant for predicting what comes next. This is the core mechanism behind every modern LLM, from GPT-4 to Claude.

In this step, get ready to:
  • Add positional embeddings so the model knows where each character sits in the sequence.
  • Implement a self-attention Head that computes queries, keys, and values.
  • Wire attention into a new model and train for 5000 steps to compare with the bigram output.
Expand the context window and embedding dimension

The bigram model used block_size=8 and embedded each token directly into a vocab-sized vector. For attention to work, the model needs a richer representation of each token and a longer window of context to attend over.

  • In gpt.py, find the hyperparameters section at the top of the file.
  • Find this line:
block_size = 8
  • Change 8 to 64, so the line now looks like this:
block_size = 64
  • Now find this line:
device = 'cpu'
  • Add the following line directly below it:
n_embd = 32

What do these changes do?

block_size = 64 means the model can now look at up to 64 previous characters when predicting the next one (up from 8).

n_embd = 32 sets the embedding dimension. Instead of mapping each character directly to a vocab-sized vector, the model maps it to a smaller 32-dimensional space where it can learn meaningful relationships.

  • Now delete everything from class BigramLanguageModel(nn.Module): all the way to the end of the file.
  • You are removing the old bigram model, its training loop, and the generation call.
Implement the Head class

Self-attention has three ingredients: queries ("what am I looking for?"), keys ("what do I contain?"), and values ("what information do I give if matched?"). Each token broadcasts a key and a query, and attention scores tell each token how much to focus on every other token.

  • Add the following Head class where the old BigramLanguageModel used to be:
class Head(nn.Module):
    """One head of self-attention."""

    def __init__(self, head_size):
        super().__init__()
        # Project input into queries, keys, and values
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        # Causal mask: tokens can only attend to previous positions
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)
        q = self.query(x)
        # Compute attention scores, scaled by sqrt(head_size)
        wei = q @ k.transpose(-2, -1) * C**-0.5
        # Apply causal mask so tokens cannot look into the future
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = F.softmax(wei, dim=-1)
        # Weighted aggregation of values
        v = self.value(x)
        out = wei @ v
        return out

What does this code do?

  • self.key, self.query, and self.value are linear projections that transform each token's embedding into Q, K, and V vectors.
  • q @ k.transpose(-2, -1) * C**-0.5 computes how much each token should attend to every other token. The scaling by sqrt(head_size) prevents the dot products from becoming too large.
  • The causal mask (tril) fills future positions with -inf before softmax, so tokens can only attend to previous positions. This is what makes the model autoregressive.
  • The final wei @ v multiplies the attention weights by the value vectors, producing a weighted combination of information from all attended positions.

The model class will use this Head in the next chunk.

Wire attention into the model and train

Now you need a model that uses the Head. The new model adds a positional embedding (so the attention mechanism knows the order of tokens) and passes the combined embeddings through the attention head before projecting to vocabulary size.

  • Add the AttentionLanguageModel class directly below the Head class:
class AttentionLanguageModel(nn.Module):

    def __init__(self):
        super().__init__()
        # Token embedding: character -> n_embd dimensional vector
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        # Position embedding: position index -> n_embd dimensional vector
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        # Single self-attention head
        self.sa_head = Head(n_embd)
        # Final projection back to vocabulary size
        self.lm_head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding_table(idx)
        pos_emb = self.position_embedding_table(torch.arange(T, device=device))
        # Sum token and position info so attention knows both what and where
        x = tok_emb + pos_emb
        x = self.sa_head(x)
        logits = self.lm_head(x)
        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)
        return logits, loss

    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            # Crop context to block_size so position embeddings stay in range
            idx_cond = idx[:, -block_size:]
            logits, loss = self(idx_cond)
            logits = logits[:, -1, :]
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

What does this model do differently?

  • token_embedding_table now maps characters to a 32-dimensional space instead of directly to vocab_size. This gives the model room to learn relationships between characters.
  • position_embedding_table adds a unique vector for each position in the sequence. Without this, the attention mechanism would not know the ORDER of tokens.
  • idx_cond = idx[:, -block_size:] crops the context during generation so position embeddings never go out of range.
  • Add the training loop and generation call directly below the AttentionLanguageModel class:
model = AttentionLanguageModel()
model = model.to(device)

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

for iter in range(5000):
    if iter % 500 == 0:
        xb, yb = get_batch('train')
        logits, loss = model(xb, yb)
        print(f'step {iter}: loss {loss.item():.4f}')
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

context = torch.zeros((1, 1), dtype=torch.long, device=device)
print(decode(model.generate(context, max_new_tokens=300)[0].tolist()))

What changed in the training loop?

  • Training now runs for 5000 iterations instead of 1000. The attention model has more parameters and needs more training time to converge.
  • Loss is printed every 500 steps instead of every 100.
  • After training, the model generates 300 characters starting from a newline character, just like the bigram model did.
  • Save gpt.py.

Before you run this, think back to the bigram model's output. Do you think attention will produce noticeably better text, or just slightly better loss numbers?

  • Run the script by executing this command:
python3 gpt.py

Training takes about 30-60 seconds with these settings. You should see the loss decrease from around 4.1 down below 2.5 over the 5000 steps.

The generated text at the end shows a clear improvement over the bigram model. You should see more word-like structures, occasional real English words, and short phrases that resemble Shakespeare.

Why is this better than the bigram model?

The bigram model could only look at one character. The attention model looks at up to 64 previous characters and decides which ones matter for the current prediction.

For example, if the model has seen "ROMEO" followed by a colon, every character in "ROMEO:" can influence what comes next. The bigram model only saw the colon and had to guess blindly.

Seeing a shape mismatch or runtime error?

  • Check that you deleted the entire BigramLanguageModel class and all code below it before pasting the new code. Leftover code from the old model causes name conflicts.
  • Confirm n_embd = 32 appears after device = 'cpu' and before torch.manual_seed(1337). The Head class references this global variable.
  • Make sure block_size = 64 (not 8). The position embedding table uses block_size as its size, and if it doesn't match the data batches, you get a shape error.

Help me debug a shape mismatch error in my self-attention model.

✔️ Awesome, I've got everything!

Great. Double check you saved gpt.py and your training ran to completion with loss below 2.5.

ⓧ I'd like to double check the full code

Here is the complete gpt.py at this point. Scroll through and confirm your file matches:

Your file has these sections in order:

  • Imports (torch, torch.nn, torch.nn.functional).
  • Hyperparameters: batch_size = 32, block_size = 64, learning_rate = 1e-3, device = 'cpu', n_embd = 32.
  • Random seed, data loading, tokenizer, train/val split, and get_batch function (unchanged from Step 2).
  • Head class with Q/K/V projections, causal mask, and scaled dot-product attention.
  • AttentionLanguageModel class with token embeddings, position embeddings, attention head, and linear projection.
  • Training loop (5000 iterations) and generation call (300 tokens).

Your model can now attend to previous characters when making predictions. Next up, you will stack multiple attention heads and add feed-forward layers to build the full GPT architecture.

Assemble the Full GPT

Your single attention head already produces noticeably better text than the bigram model. But one head can only capture one type of relationship between tokens.

Real GPTs run multiple attention heads in parallel, each learning a different pattern. Those heads get stacked into repeated transformer blocks with feed-forward layers, residual connections, and layer normalization. This is the architecture that scales from your 10M parameter model all the way to GPT-4.

In this step, get ready to:
  • Build MultiHeadAttention, FeedForward, and Block classes.
  • Assemble the full GPTLanguageModel with 6 stacked transformer blocks.
  • Train for 5000 iterations and generate Shakespeare-style text.
Rebuild gpt.py with the full architecture

The architecture changes so fundamentally in this step that you will replace the entire contents of gpt.py. You will rebuild it section by section with the full GPT components.

  • Open gpt.py in your editor.
  • Select all the code and delete it (Cmd+A on macOS, Ctrl+A on Windows, then delete).
  • Paste the updated imports and hyperparameters by copying this code into the empty file:
import torch
import torch.nn as nn
from torch.nn import functional as F

# Hyperparameters
batch_size = 64
block_size = 256
max_iters = 5000
eval_interval = 500
learning_rate = 3e-4
device = 'cpu'
eval_iters = 200
n_embd = 384
n_head = 6
n_layer = 6
dropout = 0.2

torch.manual_seed(1337)

What changed in the hyperparameters?

You scaled up everything. n_embd jumped from 32 to 384 (giving the model much richer representations). block_size grew from 64 to 256 (the model can now look at 256 characters of context).

The new parameters n_head: 6 attention heads, n_layer: 6 stacked blocks, and dropout: 0.2 for regularization.

  • Add the data loading, tokenizer, and batch function below the hyperparameters by pasting this code:
# Load data
with open('input.txt', 'r', encoding='utf-8') as f:
    text = f.read()

# Character-level tokenizer
chars = sorted(list(set(text)))
vocab_size = len(chars)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])

# Train/val split
data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]


def get_batch(split):
    d = train_data if split == 'train' else val_data
    ix = torch.randint(len(d) - block_size, (batch_size,))
    x = torch.stack([d[i:i+block_size] for i in ix])
    y = torch.stack([d[i+1:i+block_size+1] for i in ix])
    x, y = x.to(device), y.to(device)
    return x, y

Why is this section mostly unchanged?

The data pipeline and tokenizer are the same as before. The only difference is that get_batch now uses the larger batch_size (64) and block_size (256) from the updated hyperparameters.

  • Save gpt.py and verify it runs without errors by running:
python3 gpt.py

The script loads the data and exits cleanly with no output. If you see a SyntaxError or NameError, double-check your pasted code matches the blocks above.

Seeing a FileNotFoundError?

Make sure you are running the command from inside your gpt-from-scratch folder where input.txt lives.

help me fix a FileNotFoundError when running gpt.py

Now you will add the loss estimation function and the updated attention head.

  • Add the estimate_loss function below get_batch by pasting this code:
@torch.no_grad()
def estimate_loss():
    out = {}
    model.eval()
    for split in ['train', 'val']:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            logits, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean()
    model.train()
    return out

What does estimate_loss do?

Instead of printing the loss from a single batch (which is noisy), this function averages loss over 200 batches. That gives a stable measurement of how well the model is learning.

The @torch.no_grad() decorator tells PyTorch not to track gradients during evaluation. This saves memory since you only need gradients during training.

  • Add the updated Head class below estimate_loss by pasting this code:
class Head(nn.Module):
    """One head of self-attention."""

    def __init__(self, head_size):
        super().__init__()
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)
        q = self.query(x)
        wei = q @ k.transpose(-2, -1) * C**-0.5
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = F.softmax(wei, dim=-1)
        wei = self.dropout(wei)
        v = self.value(x)
        out = wei @ v
        return out

What changed in the Head class?

One line added: self.dropout = nn.Dropout(dropout) in __init__, and wei = self.dropout(wei) in forward.

Dropout randomly zeroes out some attention weights during training. This prevents the model from over-relying on any single token relationship and improves generalization.

  • Save gpt.py and run it again to verify no syntax errors:
python3 gpt.py

The script should exit cleanly again with no output. Any error here means a typo in the code you just pasted.

Now for the three brand-new classes that turn a single head into a full transformer block.

  • Add the MultiHeadAttention and FeedForward classes below the Head class by pasting this code:
class MultiHeadAttention(nn.Module):
    """Multiple heads of self-attention in parallel."""

    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
        self.proj = nn.Linear(n_embd, n_embd)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.dropout(self.proj(out))
        return out


class FeedForward(nn.Module):
    """A simple linear layer followed by a non-linearity."""

    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),
            nn.Dropout(dropout),
        )

    def forward(self, x):
        return self.net(x)

What do these two classes do?

  • MultiHeadAttention runs 6 separate Head instances in parallel, each learning different patterns (one might focus on nearby characters, another on sentence structure). It concatenates their outputs and projects back to the embedding dimension.
  • FeedForward is a two-layer MLP that processes each position independently. It expands to 4x the embedding size (giving the model room to think), then compresses back down.
  • Add the Block class below FeedForward by pasting this code:
class Block(nn.Module):
    """Transformer block: communication followed by computation."""

    def __init__(self, n_embd, n_head):
        super().__init__()
        head_size = n_embd // n_head
        self.sa = MultiHeadAttention(n_head, head_size)
        self.ffwd = FeedForward(n_embd)
        self.ln1 = nn.LayerNorm(n_embd)
        self.ln2 = nn.LayerNorm(n_embd)

    def forward(self, x):
        x = x + self.sa(self.ln1(x))
        x = x + self.ffwd(self.ln2(x))
        return x

What are residual connections and layer norm?

The x = x + self.sa(self.ln1(x)) pattern is a residual connection. Instead of replacing x with the attention output, you ADD the attention output to x. This creates a "shortcut" that lets gradients flow easily through deep networks.

LayerNorm normalizes the values at each position before they enter attention or the feed-forward layer. This stabilizes training and helps the model converge faster.

  • Save gpt.py and run it again to check for syntax errors:
python3 gpt.py

Still exits cleanly. All the building blocks are now defined and ready to be assembled.

Seeing an IndentationError?

Python is strict about indentation. Make sure each class method is indented with exactly 4 spaces inside the class, and the method body with 8 spaces.

help me fix an IndentationError in my PyTorch model classes

Assemble the GPTLanguageModel class

Each Block combines multi-head attention with a feed-forward layer. Now you will stack 6 of these blocks into a complete GPT model.

  • Add the GPTLanguageModel class below the Block class by pasting this code:
class GPTLanguageModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])
        self.ln_f = nn.LayerNorm(n_embd)
        self.lm_head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding_table(idx)
        pos_emb = self.position_embedding_table(torch.arange(T, device=device))
        x = tok_emb + pos_emb
        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.lm_head(x)

        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)

        return logits, loss

What does this model do differently?

  • self.blocks stacks 6 transformer blocks in sequence. Data flows through all 6 blocks, with each block refining the representation.
  • self.ln_f applies a final layer normalization before the output projection.
  • The forward method sums token + position embeddings, passes through all blocks, normalizes, then projects to vocabulary size to predict the next character.
  • Add the generate method and model instantiation below the forward method (still inside the class) by pasting this code:
    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -block_size:]
            logits, loss = self(idx_cond)
            logits = logits[:, -1, :]
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx


model = GPTLanguageModel()
model = model.to(device)
print(f"{sum(p.numel() for p in model.parameters()) / 1e6:.2f}M parameters")

What does the generate method do?

It works the same as before: crop context to block_size, get predictions, sample the next token, append it, repeat. The difference is that now the model has 6 layers of attention processing the context instead of a single head.

Before you run this, what do you think the parameter count will be? The model has 384-dimensional embeddings, 6 layers, 6 heads per layer, and a 65-character vocabulary.

  • Save gpt.py and run it to see the parameter count:
python3 gpt.py

You should see something like 10.65M parameters printed. That is roughly 10 million learnable values across all the layers and projections.

Seeing a NameError or AttributeError?

Make sure the generate method is indented inside the GPTLanguageModel class (8 spaces for the method body). The model = GPTLanguageModel() line should be at the top level (0 indentation).

help me debug a NameError in my GPTLanguageModel

Train the full model and generate text

Your 10M parameter model is assembled. Now you will train it on the Shakespeare corpus and see the dramatic improvement in text quality.

  • Add the training loop and generation code at the bottom of gpt.py (below the parameter count print) by pasting this code:
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

for iter in range(max_iters):
    if iter % eval_interval == 0:
        losses = estimate_loss()
        print(f"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}")

    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

losses = estimate_loss()
print(f"step {max_iters}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}")

context = torch.zeros((1, 1), dtype=torch.long, device=device)
print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))

What does this training loop do?

  • Every 500 steps, it calls estimate_loss() to print a stable average of train and validation loss.
  • Each iteration samples a batch, computes loss, and updates weights with AdamW (learning rate 3e-4).
  • After 5000 steps, it generates 500 characters starting from an empty context so you can read the output quality.

Before you run training, what loss do you expect to end up with? In Step 3 with a single head, you saw loss around 2.5. With 6 layers and 6 heads, the model has far more capacity to learn patterns.

  • Save gpt.py and run the full training by running:
python3 gpt.py

How long will training take?

On Apple Silicon (M1/M2/M3), expect 3 to 8 minutes total. You will see loss printed every 500 steps so you can watch progress. Do not close the terminal while it runs.

You should see the loss drop steadily across 5000 steps. The final train loss should be around 1.5 or below, and the validation loss close to that.

After training finishes, the script prints 500 characters of generated text. You should see mostly real English words arranged in Shakespeare-like patterns: character names like ROMEO or KING followed by dialogue, line breaks in plausible places, and dramatic-sounding phrasing.

Loss not decreasing or script running too slowly?

If loss stays above 4.0 after step 500, check that your hyperparameters match exactly: learning_rate = 3e-4, batch_size = 64, block_size = 256.

If training is extremely slow (more than 15 minutes), ensure no other heavy processes are running and that your Python is not using Rosetta emulation on Apple Silicon.

help me debug slow training or high loss in my GPT model

✔️ Awesome, I've got everything!

Great. Double check you have saved gpt.py and that your training completed with a final loss around 1.5.

ⓧ I'd like to double check the full code

Here is the complete gpt.py file. Scroll through and confirm each section matches yours.

The file is long, so check it in sections:

  • Imports and hyperparameters (lines 1-18): 3 imports, 12 hyperparameter assignments, and the manual seed.
  • Data loading and get_batch (lines 20-46): file read, tokenizer setup, train/val split, and the get_batch function.
  • estimate_loss (lines 49-60): the @torch.no_grad() decorated function.
  • Head class (lines 63-84): includes self.dropout in __init__ and wei = self.dropout(wei) in forward.
  • MultiHeadAttention (lines 87-98): ModuleList of Heads, projection layer, dropout.
  • FeedForward (lines 101-112): Sequential with Linear, ReLU, Linear, Dropout.
  • Block (lines 115-127): LayerNorm, attention, residual, LayerNorm, feed-forward, residual.
  • GPTLanguageModel (lines 130-165): embeddings, 6 blocks, final LayerNorm, lm_head, forward, generate.
  • Training loop (lines 168-186): AdamW optimizer, 5000 iterations, estimate_loss every 500 steps, final generation.

You now have a working GPT that generates coherent Shakespeare-style text. The exact same architecture (stacked transformer blocks with multi-head attention) powers every modern LLM, just with bigger numbers.

Secret mission

Add Temperature and Top-K Sampling

Your model generates text the same way every time. Real LLMs like GPT-4 and Claude let you control creativity with a temperature dial and top-k filtering. Can you add these controls to your GPT?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • The gpt-from-scratch folder on your Desktop (contains gpt.py and input.txt).
  • PyTorch installed via pip3.

Keep everything running

No action needed. Choose this if you want to keep experimenting with your GPT model. You can re-run gpt.py anytime, tweak hyperparameters, or swap in different training data.

  • Your gpt-from-scratch folder stays on your Desktop.
  • PyTorch remains installed and available for other Python projects.

Pause - I'll come back to this later

Nothing is running in the background, so there is nothing to pause. Your files sit on disk until you return.

  • Your gpt-from-scratch folder and PyTorch installation will be ready whenever you come back.
  • No processes to stop and no services to shut down.

Delete - I don't want to use this again

Remove all project files and uninstall PyTorch to start fresh.

  • Delete the project folder by running this in your terminal:
  • macOS/Linux:
  • rm -rf ~/Desktop/gpt-from-scratch.
  • Windows:
  • rmdir /s /q %USERPROFILE%\Desktop\gpt-from-scratch.
  • Uninstall PyTorch by running:
  • pip3 uninstall torch -y.

Nice Work!

Nice Work!

Nice work! You just built a working GPT language model from scratch in a single Python file. Your model learned to generate Shakespeare-style text by predicting one character at a time, using the exact same architecture that powers GPT-4 and Claude.

You've learned how to:

  • Build self-attention from scratch (queries, keys, values, causal masking, scaled dot-product) and see firsthand why context matters for language generation by comparing a bigram baseline to the full attention mechanism.
  • Assemble a complete GPT architecture with multi-head attention, feed-forward networks, residual connections, layer normalization, and stacked transformer blocks into a ~10M parameter model.
  • Train a character-level language model on Shakespeare's collected works and generate coherent text from a prompt using autoregressive next-token prediction.
  • Secret Mission: Added temperature and top-k sampling to control the creativity of your model's output.

Ready to quiz yourself?