Build a GPT Model from Scratch

Build a GPT-2 style transformer in PyTorch and train it on Shakespeare.

Introduction

30 Second Summary

Every time you ask ChatGPT a question or get a code suggestion from an AI assistant, a transformer language model is doing the work behind the scenes. Most people treat these models as black boxes.

In this project, you will build a miniature GPT-style language model entirely from scratch using PyTorch. You'll train it on Shakespeare's collected works and watch it learn to generate Elizabethan-style text from any prompt you type.

What You'll Build

A miniature GPT-2 architecture implemented from scratch using basic PyTorch primitives, scaled down to ~800K parameters so it trains on your CPU in minutes.

By the end of this project, you'll have:

  • A model.py file containing every transformer component built from scratch: self-attention, feed-forward networks, layer normalization, and residual connections.
  • A train.py file that downloads Shakespeare's collected works, builds a character-level tokenizer, and trains the model from random weights.
  • A generate.py file that loads the trained model and generates Shakespeare-style text from any prompt you type.
  • Secret Mission: Swap in GPT-2's BPE tokenizer and discover how tokenization strategy changes what the model learns.

Are there any prerequisites?

This is an advanced project. You should be comfortable with Python programming, have a basic understanding of neural networks and gradient descent, and be familiar with PyTorch tensors.

Before We Start

Before We Start...

Set Up Your Project and Download the Dataset

Building a GPT model from scratch means writing every component yourself, from embeddings to the attention mechanism to the training loop.

You'll organize your code across three files: model.py for the transformer architecture, train.py for data loading and training, and generate.py for producing text from the trained model.

In this step, get ready to:
  • Create the gpt-from-scratch project folder with three Python files.
  • Verify your PyTorch installation.
  • Download the TinyShakespeare dataset.
Create the project folder and files
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (the backtick key, usually above Tab).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir gpt-from-scratch
cd gpt-from-scratch
  • Create the three project files by running this command:

macOS

touch model.py train.py generate.py

Windows

New-Item model.py -ItemType File
New-Item train.py -ItemType File
New-Item generate.py -ItemType File
  • Confirm the files exist by running:
ls

You should see model.py, train.py, and generate.py in the output.

  • Now open this folder in VS Code. Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the gpt-from-scratch folder, and click Open.

You should see model.py, train.py, and generate.py listed in the Explorer sidebar on the left.

Don't see your files in the Explorer?

Make sure you selected the gpt-from-scratch folder itself, not your Desktop. The folder name should appear at the top of the Explorer sidebar.

If the files are missing, open a new terminal in VS Code (Ctrl+`) and re-run the touch or New-Item command from the platform tab above.

Help me set up my gpt-from-scratch project folder in VS Code.

Verify your PyTorch installation

PyTorch is the deep learning framework you'll use to build every layer of your GPT model. Let's confirm it's ready.

  • Check your PyTorch version by running this command in the VS Code terminal:
python -c "import torch; print(torch.__version__)"

✔️ I see a version number

You're all set. Any version starting with 2. works for this project.

ⓧ ModuleNotFoundError

PyTorch isn't installed yet.

  • Install PyTorch by running this command:
pip install torch
  • After the install completes, run the version check command again. You should see a version number starting with 2. printed.
Download the TinyShakespeare dataset

Your model needs training data to learn from. TinyShakespeare is a ~1MB text file containing all of Shakespeare's collected works.

It's small enough to train on your CPU in minutes, but rich enough for the model to pick up patterns in English text.

  • Download the dataset into your project folder by running this command:
curl -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

What does this command do?

curl downloads files from a URL. The -o input.txt flag saves the downloaded content as input.txt in your current folder.

  • Confirm the file downloaded by running:
ls

You should see input.txt alongside your three Python files.

  • Click on input.txt in the Explorer sidebar to open it.

You should see Shakespeare's text starting with "First Citizen:" at the very top. This is the raw text your model will learn to imitate.

Curl not recognized?

If your terminal doesn't recognize curl, you can download the file using Python instead. Run this command:

python -c "import urllib.request; urllib.request.urlretrieve('https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt', 'input.txt'); print('Downloaded!')"

You should see Downloaded! printed. Then run ls to confirm input.txt is in your folder.

Help me download the TinyShakespeare dataset into my project folder.

Why TinyShakespeare?

This dataset has 65 unique characters (letters, punctuation, and whitespace). That tiny vocabulary means your model can use character-level tokenization, where each character maps directly to a token.

This keeps the architecture simple so you can focus on how the transformer works, not on complex tokenization. The Secret Mission later upgrades to a real subword tokenizer so you can compare the two approaches.

Your workspace is ready and your training data is waiting. Next, you'll dive into the core of the transformer architecture: the self-attention mechanism.

Build the Self-Attention Mechanism

Your project is set up and Shakespeare's complete works are sitting in your project folder. Now it's time to build the core mechanism that makes a transformer work.

Self-attention is the mechanism that lets every token in a sequence look at every other token to figure out what's relevant. Your model needs this to understand that "King" relates to "crown" even when they're far apart in a sentence. In this step, you'll implement scaled dot-product attention with causal masking from scratch.

In this step, get ready to:
  • Define the model's hyperparameters in a configuration dataclass.
  • Implement the CausalSelfAttention module with multi-head processing and causal masking.
  • Test the attention module to verify it produces the correct output shape.
Add imports and create the GPTConfig dataclass

Every model needs a central place that defines its architecture. A configuration dataclass holds all the hyperparameters that control the model's size, context window, and training behavior.

  • Open model.py in VS Code and add the following imports and configuration:
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from dataclasses import dataclass


# Hyperparameters for the GPT model
@dataclass
class GPTConfig:
    vocab_size: int = 65       # Number of unique characters in TinyShakespeare
    block_size: int = 256      # Maximum context length (tokens the model sees at once)
    n_layer: int = 4           # Number of transformer blocks stacked together
    n_head: int = 4            # Number of attention heads per block
    n_embd: int = 128          # Embedding dimension (size of each token's vector)
    dropout: float = 0.1       # Dropout rate for regularization

What does this code do?

The imports bring in PyTorch's neural network modules, functional utilities, and Python's dataclass decorator.

  • vocab_size=65 matches the 65 unique characters in the TinyShakespeare dataset.
  • block_size=256 sets the context window. The model can look at up to 256 tokens at a time when predicting the next one.
  • n_embd=128 is the size of each token's vector representation. Every token gets encoded as a list of 128 numbers.
  • n_head=4 means the model runs four parallel attention operations, each focusing on different types of relationships between tokens.

You won't see a visible change yet. The next chunk builds the attention module that uses this config.

Build the CausalSelfAttention class

This is the heart of the transformer. The attention mechanism lets each token compute a relevance score against every other token in the sequence, then use those scores to build a weighted summary of the most useful context.

"Causal" means the model can only look backward. Token 5 can attend to tokens 1 through 5, but not token 6. This is what makes the model autoregressive. It predicts the future without peeking at it.

  • In model.py, below the GPTConfig class, add the CausalSelfAttention class with its constructor:
class CausalSelfAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        # Embedding dim must be evenly divisible by number of heads
        assert config.n_embd % config.n_head == 0
        self.n_head = config.n_head
        self.head_dim = config.n_embd // config.n_head

        # Linear projections for Query, Key, and Value
        self.query = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.key = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.value = nn.Linear(config.n_embd, config.n_embd, bias=False)
        # Output projection to combine all heads back together
        self.proj = nn.Linear(config.n_embd, config.n_embd)

        self.attn_dropout = nn.Dropout(config.dropout)
        self.proj_dropout = nn.Dropout(config.dropout)

        # Causal mask: lower-triangular matrix prevents attending to future tokens
        self.register_buffer(
            "mask",
            torch.tril(torch.ones(config.block_size, config.block_size))
            .view(1, 1, config.block_size, config.block_size)
        )

What does this constructor set up?

  • Query, Key, and Value are three separate linear projections of the same input. Think of Query as "what am I looking for?", Key as "what do I contain?", and Value as "what information do I carry?".
  • head_dim splits the embedding across multiple heads. With n_embd=128 and n_head=4, each head works with 32-dimensional vectors.
  • register_buffer stores the causal mask as part of the module without treating it as a trainable parameter. torch.tril creates a lower-triangular matrix of ones. Positions with a zero will be blocked during attention.

The constructor defines all the layers. Next, you'll write the forward method that wires them together into the actual attention computation.

  • Still inside the CausalSelfAttention class, add the forward method below the __init__ method:
    def forward(self, x):
        B, T, C = x.shape  # Batch size, sequence length, embedding dimension

        # Project input into Q, K, V and reshape for multi-head processing
        q = self.query(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k = self.key(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = self.value(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)

        # Scaled dot-product attention: (Q @ K^T) / sqrt(d_k)
        att = (q @ k.transpose(-2, -1)) * (self.head_dim ** -0.5)
        # Apply causal mask so each token only attends to itself and previous tokens
        att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float('-inf'))
        att = F.softmax(att, dim=-1)
        att = self.attn_dropout(att)

        # Multiply attention weights by Values, then concatenate heads
        out = att @ v
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        # Final projection back to model dimension
        out = self.proj_dropout(self.proj(out))
        return out

What happens in the forward pass?

This is the attention formula from the original "Attention Is All You Need" paper, implemented step by step.

  • The input gets projected into Q, K, V, then reshaped from (batch, sequence, embedding) to (batch, heads, sequence, head_dim). This lets each head run its own attention computation independently.
  • q @ k.transpose(-2, -1) computes a score between every pair of tokens. Dividing by sqrt(d_k) prevents the scores from getting too large, which would push softmax into regions with tiny gradients.
  • masked_fill sets future positions to negative infinity. After softmax, those positions become zero, so the model cannot attend to tokens that come later in the sequence.
  • Finally, the attention-weighted Values from all heads are concatenated back into a single vector and projected to the output dimension.
  • Save model.py.

Seeing a red underline or import error?

  • Make sure the forward method is indented one level inside the CausalSelfAttention class. It should be at the same indentation level as __init__.
  • Check that import torch.nn.functional as F is at the top of the file. The F.softmax call in forward depends on it.

Why is my CausalSelfAttention class showing errors in VS Code?

Test your attention module

Before building more on top, you should verify that the attention module produces the right output shape. A quick test catches shape mismatches early, before they cascade into cryptic errors downstream.

  • At the bottom of model.py, below the CausalSelfAttention class, add this temporary test code:
# Temporary test - remove after verifying
config = GPTConfig()
attn = CausalSelfAttention(config)
x = torch.randn(1, 10, 128)  # batch=1, sequence=10, embedding=128
out = attn(x)
print(f"Output shape: {out.shape}")  # Should print torch.Size([1, 10, 128])

What does this test do?

It creates a random tensor shaped like a single batch of 10 tokens, each with 128-dimensional embeddings. If the attention module is wired correctly, the output should have the exact same shape. The data goes in, gets transformed through Q/K/V projections, attention, and the output projection, and comes back at the same dimensions.

  • Save model.py.
  • Run the test by executing this command in your terminal:
python model.py

You should see this output:

Output shape: torch.Size([1, 10, 128])

The output shape matches the input shape. Your attention module is working correctly.

Seeing a different output or an error?

  • If you see ModuleNotFoundError: No module named 'torch', PyTorch is not installed. Run pip install torch and try again.
  • If you see a RuntimeError about tensor shapes, double check that self.head_dim = config.n_embd // config.n_head is set in your __init__ method.
  • If the output shape doesn't match torch.Size([1, 10, 128]), check that your view and transpose calls in the forward method match the code above exactly.

Help me debug my CausalSelfAttention test output.

  • Now that the test passes, delete the temporary test code from the bottom of model.py. Remove everything below the CausalSelfAttention class definition.
  • Save model.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved model.py and removed the temporary test code at the bottom.

ⓧ I'd like to double check the full code

Here is what model.py should look like at this point. Make sure your file matches this exactly (the temporary test code should already be removed).

import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from dataclasses import dataclass


@dataclass
class GPTConfig:
    vocab_size: int = 65
    block_size: int = 256
    n_layer: int = 4
    n_head: int = 4
    n_embd: int = 128
    dropout: float = 0.1


class CausalSelfAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        assert config.n_embd % config.n_head == 0
        self.n_head = config.n_head
        self.head_dim = config.n_embd // config.n_head

        self.query = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.key = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.value = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.proj = nn.Linear(config.n_embd, config.n_embd)

        self.attn_dropout = nn.Dropout(config.dropout)
        self.proj_dropout = nn.Dropout(config.dropout)

        self.register_buffer(
            "mask",
            torch.tril(torch.ones(config.block_size, config.block_size))
            .view(1, 1, config.block_size, config.block_size)
        )

    def forward(self, x):
        B, T, C = x.shape

        q = self.query(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k = self.key(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = self.value(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)

        att = (q @ k.transpose(-2, -1)) * (self.head_dim ** -0.5)
        att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float('-inf'))
        att = F.softmax(att, dim=-1)
        att = self.attn_dropout(att)

        out = att @ v
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        out = self.proj_dropout(self.proj(out))
        return out

Why build attention from scratch instead of using PyTorch's built-in?

PyTorch provides nn.MultiheadAttention as a ready-made module. It works perfectly fine for production code.

Building it yourself forces you to understand every tensor reshape, every matrix multiplication, and exactly how the causal mask blocks future tokens. When you encounter attention in real codebases later, you'll know what's happening inside the abstraction.

Your self-attention mechanism is complete. In the next step, you'll wrap it inside a full transformer block with feed-forward networks, layer normalization, and residual connections, then stack those blocks into a complete GPT model.

Assemble the Transformer Block and Full GPT Model

In the previous step, you built the self-attention mechanism from scratch. Each token can now look at every previous token and decide how much to attend to it.

Self-attention is just one piece of the puzzle. A full transformer layer also needs a feed-forward network to process each token's information after attention. In this step, you will build the remaining components and stack them into a complete GPT model.

In this step, get ready to:
  • Build the FeedForward and TransformerBlock modules.
  • Assemble the full GPT class with embeddings, weight tying, and proper initialization.
  • Implement the forward pass and autoregressive text generation.
Build the FeedForward and TransformerBlock modules

After self-attention decides which tokens are relevant, each token needs its own processing step to transform that information into something useful. That is the job of the feed-forward network.

  • In model.py, add the FeedForward class below the CausalSelfAttention class by copying this code:
class FeedForward(nn.Module):
    def __init__(self, config):
        super().__init__()
        # Two-layer MLP: expand dimension by 4x, then project back down
        self.net = nn.Sequential(
            nn.Linear(config.n_embd, 4 * config.n_embd),
            nn.GELU(),
            nn.Linear(4 * config.n_embd, config.n_embd),
            nn.Dropout(config.dropout),
        )

    def forward(self, x):
        return self.net(x)

What does this code do?

The feed-forward network is a simple two-layer MLP that every token passes through independently. The first linear layer expands the dimension by 4x (from 128 to 512), giving the model more room to learn complex patterns.

GELU is the activation function that introduces non-linearity. The second linear layer projects back down to the original dimension. Dropout at the end randomly zeros out some values during training to prevent overfitting.

You won't see any output from this class on its own. The next chunk combines it with attention into a full transformer block.

A single transformer layer pairs self-attention with a feed-forward network, and wraps each one in layer normalization and a residual connection. This is the building block you will stack to create a deep model.

  • Add the TransformerBlock class directly below FeedForward by copying this code:
class TransformerBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        # Pre-norm: apply LayerNorm before each sub-layer
        self.ln1 = nn.LayerNorm(config.n_embd)
        self.attn = CausalSelfAttention(config)
        self.ln2 = nn.LayerNorm(config.n_embd)
        self.ffn = FeedForward(config)

    def forward(self, x):
        # Residual connections: add the input back after each sub-layer
        x = x + self.attn(self.ln1(x))
        x = x + self.ffn(self.ln2(x))
        return x

What does this code do?

Each line in the forward method does three things at once: normalize the input with LayerNorm, pass it through a sub-layer (attention or feed-forward), then add the original input back. That final addition is the residual connection.

Residual connections solve a critical problem in deep networks. Without them, gradients vanish as they flow backwards through many layers, and the model stops learning. Adding the original input back gives gradients a "shortcut" path to flow through.

Why pre-norm instead of post-norm?

The original "Attention Is All You Need" paper applied LayerNorm after each sub-layer (post-norm). GPT-2 moved LayerNorm to before each sub-layer (pre-norm), which makes training much more stable.

All modern LLMs use pre-norm. Your implementation follows this same pattern.

You won't see output yet. The next substep stacks these blocks into a full GPT model you can test.

Build the GPT class

The GPT class is the top-level model that ties everything together. It converts raw token IDs into embeddings, passes them through a stack of transformer blocks, and projects the result back to vocabulary-sized logits for predicting the next token.

  • Add the GPT class below TransformerBlock by copying this code:
class GPT(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        # Convert token IDs to dense vectors
        self.token_embedding = nn.Embedding(config.vocab_size, config.n_embd)
        # Encode each position in the sequence
        self.position_embedding = nn.Embedding(config.block_size, config.n_embd)
        self.drop = nn.Dropout(config.dropout)
        # Stack n_layer transformer blocks
        self.blocks = nn.Sequential(
            *[TransformerBlock(config) for _ in range(config.n_layer)]
        )
        # Final layer norm before the output projection
        self.ln_f = nn.LayerNorm(config.n_embd)
        # Project from embedding dimension back to vocabulary size
        self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

        # Weight tying: token embedding and output head share the same matrix
        self.token_embedding.weight = self.head.weight

        # Initialize all weights with small random values
        self.apply(self._init_weights)

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                torch.nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)

What does this code do?

  • token_embedding converts each integer token ID into a 128-dimensional vector the model can work with.
  • position_embedding gives the model a sense of order. Without it, the model has no way to tell if a token is first or last in the sequence.
  • blocks is a stack of 4 TransformerBlock layers. This is where the deep learning happens.
  • head projects the final hidden state back to 65 values (one per character in the vocabulary), producing the model's prediction for the next character.
  • _init_weights initializes all parameters with small random values (standard deviation of 0.02). This is the same initialization GPT-2 uses.

What is weight tying?

The line self.token_embedding.weight = self.head.weight makes the input embedding matrix and the output projection head share the exact same weight matrix. This means the model uses the same representation for "what does this token mean" and "which token should come next."

Weight tying reduces the total parameter count and improves training. GPT-2 uses this same trick.

  • Save model.py.
  • Verify the model structure by running this command in your terminal:
python -c "from model import GPT, GPTConfig; m = GPT(GPTConfig()); print(f'Parameters: {sum(p.numel() for p in m.parameters()):,}')"

You should see Parameters: 818,561 printed. That is roughly 800K parameters, a tiny fraction of GPT-2's 124 million, but the architecture is identical.

Seeing an ImportError or NameError?

  • Make sure you saved model.py before running the command.
  • Check that your terminal is in the gpt-from-scratch folder. Run ls and confirm you see model.py in the output.
  • If you see NameError: name 'CausalSelfAttention' is not defined, make sure the CausalSelfAttention class from the previous step is still in your file above the new classes.

Why can't Python find my model classes?

Implement the forward pass and generate method

The model structure is defined, but it does not know how to process input or produce output yet. You need two methods: forward for training (given a sequence, predict the next token at every position) and generate for inference (produce new tokens one at a time).

  • In model.py, add the forward method inside the GPT class, directly below the _init_weights method. Make sure it is indented at the same level as _init_weights:
    def forward(self, idx, targets=None):
        B, T = idx.shape

        # Create token embeddings and add positional information
        tok_emb = self.token_embedding(idx)
        pos_emb = self.position_embedding(torch.arange(T, device=idx.device))
        x = self.drop(tok_emb + pos_emb)

        # Pass through every transformer block, then final layer norm
        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.head(x)

        # Compute cross-entropy loss if training targets are provided
        loss = None
        if targets is not None:
            loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)), targets.view(-1)
            )

        return logits, loss

What does this code do?

  • idx is a batch of token ID sequences. B is batch size and T is the sequence length.
  • Token embeddings and positional embeddings are added together so each token knows both what it is and where it sits in the sequence.
  • The data flows through all transformer blocks, a final LayerNorm, and the output head to produce logits (raw prediction scores for each character in the vocabulary).
  • Cross-entropy loss measures how far the model's predictions are from the actual next characters. This is only computed during training, when targets are provided.

You won't see a change yet. The next chunk adds the generate method, which lets you test the full model.

The generate method is an autoregressive loop. It predicts one token at a time, appends it to the sequence, and feeds the extended sequence back in to predict the next one.

  • Add the generate method directly below forward, at the same indentation level:
    def generate(self, idx, max_new_tokens, temperature=1.0):
        for _ in range(max_new_tokens):
            # Crop to the last block_size tokens so we don't exceed the context window
            idx_cond = idx[:, -self.config.block_size:]
            logits, _ = self(idx_cond)
            # Take logits for the last token and scale by temperature
            logits = logits[:, -1, :] / temperature
            probs = F.softmax(logits, dim=-1)
            # Sample the next token from the probability distribution
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

What does this code do?

  • idx_cond crops the sequence to the last block_size (256) tokens. The model's positional embeddings only cover 256 positions, so longer sequences would cause an error.
  • temperature controls randomness. Lower values (like 0.5) make the model more conservative. Higher values (like 1.5) make it more creative and unpredictable.
  • torch.multinomial samples a token from the probability distribution instead of always picking the most likely one. This is what gives generated text its variety.
  • Save model.py.
  • Test the complete model by running this command in your terminal:
python -c "from model import GPT, GPTConfig; import torch; m = GPT(GPTConfig()); idx = torch.zeros((1,1), dtype=torch.long); print(m.generate(idx, max_new_tokens=10).shape)"

You should see torch.Size([1, 11]) printed. The model started with 1 token and generated 10 more, giving you 11 total. The output is random gibberish right now because the model has not been trained yet.

Seeing an error instead of a tensor shape?

  • If you see AttributeError: 'GPT' object has no attribute 'generate', check that the generate method is indented inside the GPT class, not at the module level.
  • If you see IndentationError, make sure forward and generate are indented with 4 spaces (one level inside the class), matching __init__ and _init_weights.
  • If you see a RuntimeError about tensor shapes, double check that forward uses torch.arange(T, device=idx.device) for positional embeddings, not a hardcoded value.

Help me debug my GPT model

✔️ Awesome, I've got everything!

Great. Double check you have saved model.py and confirmed the generation test prints torch.Size([1, 11]).

ⓧ I'd like to double check the full code

Here is the complete model.py file. Compare it with yours line by line.

import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from dataclasses import dataclass


@dataclass
class GPTConfig:
    vocab_size: int = 65
    block_size: int = 256
    n_layer: int = 4
    n_head: int = 4
    n_embd: int = 128
    dropout: float = 0.1


class CausalSelfAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        assert config.n_embd % config.n_head == 0
        self.n_head = config.n_head
        self.head_dim = config.n_embd // config.n_head

        self.query = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.key = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.value = nn.Linear(config.n_embd, config.n_embd, bias=False)
        self.proj = nn.Linear(config.n_embd, config.n_embd)

        self.attn_dropout = nn.Dropout(config.dropout)
        self.proj_dropout = nn.Dropout(config.dropout)

        self.register_buffer(
            "mask",
            torch.tril(torch.ones(config.block_size, config.block_size))
            .view(1, 1, config.block_size, config.block_size)
        )

    def forward(self, x):
        B, T, C = x.shape

        q = self.query(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k = self.key(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = self.value(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)

        att = (q @ k.transpose(-2, -1)) * (self.head_dim ** -0.5)
        att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float('-inf'))
        att = F.softmax(att, dim=-1)
        att = self.attn_dropout(att)

        out = att @ v
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        out = self.proj_dropout(self.proj(out))
        return out


class FeedForward(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(config.n_embd, 4 * config.n_embd),
            nn.GELU(),
            nn.Linear(4 * config.n_embd, config.n_embd),
            nn.Dropout(config.dropout),
        )

    def forward(self, x):
        return self.net(x)


class TransformerBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.ln1 = nn.LayerNorm(config.n_embd)
        self.attn = CausalSelfAttention(config)
        self.ln2 = nn.LayerNorm(config.n_embd)
        self.ffn = FeedForward(config)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        x = x + self.ffn(self.ln2(x))
        return x


class GPT(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.token_embedding = nn.Embedding(config.vocab_size, config.n_embd)
        self.position_embedding = nn.Embedding(config.block_size, config.n_embd)
        self.drop = nn.Dropout(config.dropout)
        self.blocks = nn.Sequential(
            *[TransformerBlock(config) for _ in range(config.n_layer)]
        )
        self.ln_f = nn.LayerNorm(config.n_embd)
        self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

        self.token_embedding.weight = self.head.weight

        self.apply(self._init_weights)

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                torch.nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        B, T = idx.shape

        tok_emb = self.token_embedding(idx)
        pos_emb = self.position_embedding(torch.arange(T, device=idx.device))
        x = self.drop(tok_emb + pos_emb)

        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.head(x)

        loss = None
        if targets is not None:
            loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)), targets.view(-1)
            )

        return logits, loss

    def generate(self, idx, max_new_tokens, temperature=1.0):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -self.config.block_size:]
            logits, _ = self(idx_cond)
            logits = logits[:, -1, :] / temperature
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

Your model architecture is complete. Every component inside a modern LLM is now sitting in your model.py file. Next up, you will prepare the training data and teach this model to write like Shakespeare.

Prepare the Data and Train Your Model

Your GPT model is fully assembled. Every component from embeddings to attention to the output head is wired up and waiting.

A model with random weights is just a pile of numbers. It needs data to learn from and a training loop that iterates over that data thousands of times, nudging the weights a tiny bit each step. In this step, you will build a character-level tokenizer, prepare batches of Shakespeare text, and train the model until it starts producing coherent output.

In this step, get ready to:
  • Build a character-level tokenizer that converts text to numbers and back.
  • Write data batching and loss estimation utilities.
  • Train the model for 3000 steps and save a checkpoint.
Load the dataset and read it into memory

Before the model can learn anything, it needs text to study. The TinyShakespeare dataset is a ~1MB file containing all of Shakespeare's collected works. It is small enough to train on your CPU in minutes.

  • Open train.py in VS Code.
  • Add the imports, dataset URL constant, and file-loading logic by copying the following code:
import torch
import os
import urllib.request
from model import GPT, GPTConfig

# URL for the TinyShakespeare dataset (~1MB of Shakespeare's works)
DATA_URL = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
DATA_FILE = "input.txt"

# Download the dataset if it isn't already in the project folder
if not os.path.exists(DATA_FILE):
    print("Downloading TinyShakespeare dataset...")
    urllib.request.urlretrieve(DATA_URL, DATA_FILE)

# Read the entire text file into a single string
with open(DATA_FILE, "r", encoding="utf-8") as f:
    text = f.read()

What does this code do?

The imports bring in PyTorch for tensors and training, plus GPT and GPTConfig from the model.py you built in the previous steps.

The download block checks whether input.txt already exists. If it does (you downloaded it in Step 1), it skips the download. If not, it fetches the file from GitHub. Either way, the full text ends up in the text variable as one long string.

You won't see any output yet. The next chunk builds the tokenizer and prints diagnostics so you can verify everything works.

Language models don't read text directly. They operate on numbers. A tokenizer converts each character into an integer and back. For this project, you will use character-level tokenization where every unique character in Shakespeare (letters, punctuation, spaces) gets its own integer ID.

  • Add the tokenizer and diagnostic prints below the file-loading code by copying this:
# Build a character-level tokenizer
# Extract every unique character and sort them into a stable order
chars = sorted(list(set(text)))
vocab_size = len(chars)

# Create mappings: character -> integer (stoi) and integer -> character (itos)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: "".join([itos[i] for i in l])

# Print diagnostics to verify the tokenizer works
print(f"Dataset size: {len(text):,} characters")
print(f"Vocabulary size: {vocab_size} unique characters")
print(f"Sample: '{text[:20]}' -> {encode(text[:20])}")

What does this code do?

  • chars is a sorted list of every unique character in the dataset (65 total, including letters, digits, punctuation, and whitespace).
  • stoi (string-to-integer) maps each character to its integer ID. itos (integer-to-string) reverses it.
  • encode converts a string into a list of integers. decode converts a list of integers back into a string. These two functions are the entire tokenizer.
  • Save train.py.
  • Run the file to verify the tokenizer works by running this command:
python train.py

You should see three lines printed: the dataset size (around 1,115,394 characters), the vocabulary size (65 unique characters), and a sample showing the first 20 characters converted to integer IDs.

Seeing a ModuleNotFoundError?

If you see ModuleNotFoundError: No module named 'model', make sure your terminal is in the gpt-from-scratch folder where both model.py and train.py live.

If you see a KeyError, check that input.txt is in the same folder. The tokenizer can only map characters that exist in the dataset.

Help me debug my train.py tokenizer output

Why character-level tokenization?

Real LLMs like GPT-2 use subword tokenization (called Byte Pair Encoding) where tokens represent common character sequences like "the" or "ing". Character-level tokenization is simpler: each token is exactly one character, giving a tiny vocabulary of 65.

The trade-off is that the model has to learn spelling, word boundaries, and grammar entirely from scratch. The Secret Mission at the end of this project upgrades to BPE so you can compare the two approaches.

Split the data and write batch utilities

Training a model on the same data it gets evaluated on would be like grading a test with the answer key in front of you. You need separate training and validation sets so you can measure whether the model is actually learning patterns or just memorizing the text.

  • In train.py, add the data encoding, train/val split, and device detection below the tokenizer prints by copying this code:
# Encode the entire text as a sequence of integers
data = torch.tensor(encode(text), dtype=torch.long)

# Split into 90% training and 10% validation
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]

# Use GPU if available, otherwise CPU
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Create the model config using the dataset's vocabulary size
config = GPTConfig(vocab_size=vocab_size)
batch_size = 32

What does this code do?

  • data is the entire Shakespeare text converted into a single PyTorch tensor of integer token IDs.
  • The first 90% becomes train_data. The last 10% becomes val_data. The model trains on the first and gets tested on the second.
  • GPTConfig(vocab_size=vocab_size) overrides the default vocabulary size with the actual count from the dataset (65). All other config values stay at their defaults.

The model learns from short chunks of text, not the entire dataset at once. A batch function randomly samples these chunks and stacks them together for efficient parallel training.

  • Add the get_batch and estimate_loss utility functions below the config by copying this code:

def get_batch(split):
    """Randomly sample a batch of training or validation sequences."""
    d = train_data if split == "train" else val_data
    # Pick random starting positions for each sequence in the batch
    ix = torch.randint(len(d) - config.block_size, (batch_size,))
    # x is the input sequence, y is the target (shifted by one position)
    x = torch.stack([d[i:i + config.block_size] for i in ix])
    y = torch.stack([d[i + 1:i + config.block_size + 1] for i in ix])
    return x.to(device), y.to(device)


@torch.no_grad()
def estimate_loss(model, eval_iters=50):
    """Evaluate average loss on train and val splits without updating weights."""
    model.eval()
    out = {}
    for split in ["train", "val"]:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            _, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean()
    model.train()
    return out

What do these functions do?

get_batch picks random starting points in the text, then grabs chunks of block_size (256) consecutive characters. The input x and target y are offset by one position. This is how the model learns to predict the next character: given characters 0 through 255, predict characters 1 through 256.

estimate_loss runs the model on 50 random batches from each split and averages the loss. The @torch.no_grad() decorator tells PyTorch not to track gradients during evaluation, which saves memory and speeds things up.

  • Save train.py.

You won't see new output yet. These utility functions are called by the training loop you will add next.

Run the training loop and save a checkpoint

Everything is in place: the model, the tokenizer, the data splits, and the batch utilities. Now you will create the model instance, set up the optimizer, and run a training loop that iterates 3000 times over random batches of Shakespeare.

  • Add the model instantiation and optimizer setup below the estimate_loss function by copying this code:

# Create the model and move it to the compute device
model = GPT(config).to(device)
param_count = sum(p.numel() for p in model.parameters())
print(f"Model parameters: {param_count:,}")

# AdamW optimizer with a learning rate of 3e-4
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

# Training hyperparameters
max_iters = 3000
eval_interval = 500

What does this code do?

  • GPT(config).to(device) creates your model and moves it to the GPU if one is available. On CPU, this is a no-op.
  • AdamW is the optimizer that adjusts the model's weights based on the computed gradients. The learning rate of 3e-4 is a standard starting point for transformer training.
  • The model will train for 3000 steps, printing train/val loss every 500 steps so you can watch it learn.

The training loop is the final piece. Each iteration samples a random batch, computes the loss (how wrong the predictions are), calculates gradients, and nudges the weights to reduce that loss.

  • Add the training loop, final evaluation, and checkpoint save below the hyperparameters by copying this code:
print("\nStarting training...")
for step in range(max_iters):
    # Evaluate and print loss every eval_interval steps
    if step % eval_interval == 0:
        losses = estimate_loss(model)
        print(
            f"Step {step:>5d} | "
            f"train loss: {losses['train']:.4f} | "
            f"val loss: {losses['val']:.4f}"
        )

    # Sample a batch of training data
    xb, yb = get_batch("train")
    # Forward pass: compute predictions and loss
    logits, loss = model(xb, yb)
    # Backward pass: compute gradients
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    # Update weights
    optimizer.step()

# Print final loss after all training steps
losses = estimate_loss(model)
print(
    f"Step {max_iters:>5d} | "
    f"train loss: {losses['train']:.4f} | "
    f"val loss: {losses['val']:.4f}"
)

# Save the trained model, config, and character list to a checkpoint file
torch.save({
    "model_state_dict": model.state_dict(),
    "config": config,
    "chars": chars,
}, "gpt_shakespeare.pt")
print("\nModel saved to gpt_shakespeare.pt")

What does this code do?

The training loop repeats the same four operations 3000 times: forward pass (predict the next character for every position in the batch), compute cross-entropy loss (how far off the predictions are), backward pass (calculate how to adjust each weight), and optimizer step (apply those adjustments).

Every 500 steps, the loop pauses to estimate loss on both the training and validation sets. You should see both numbers drop steadily. If val loss starts rising while train loss keeps falling, the model is memorizing instead of learning.

The checkpoint saves three things: the trained weights (model_state_dict), the config (so you can reconstruct the model architecture), and the chars list (so you can rebuild the tokenizer). All three are needed to generate text later.

  • Save train.py.
  • Run the training script by running this command:
python train.py

This will take a while

Training takes roughly 5-15 minutes on a modern CPU. You will see loss printed every 500 steps. The initial loss should be around 4.1-4.3 (near random guessing for 65 characters). By step 3000, expect train loss around 1.4-1.6 and val loss around 1.6-1.8.

When training finishes, you should see Model saved to gpt_shakespeare.pt in the terminal output.

  • Confirm the checkpoint file exists by running:
ls

You should see gpt_shakespeare.pt listed alongside your .py files and input.txt.

Training seems stuck or loss isn't decreasing?

If loss stays above 4.0 after 1000 steps, double check that your model.py matches the complete code from the previous step. A bug in the attention mask or residual connections can silently prevent learning.

If you see an OutOfMemoryError, reduce batch_size from 32 to 16 in train.py and run again.

Help me debug my GPT training loop

✔️ Awesome, I've got everything!

Double check that you have saved train.py and that gpt_shakespeare.pt exists in your project folder.

ⓧ I'd like to double check the full code

import torch
import os
import urllib.request
from model import GPT, GPTConfig

DATA_URL = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
DATA_FILE = "input.txt"

if not os.path.exists(DATA_FILE):
    print("Downloading TinyShakespeare dataset...")
    urllib.request.urlretrieve(DATA_URL, DATA_FILE)

with open(DATA_FILE, "r", encoding="utf-8") as f:
    text = f.read()

chars = sorted(list(set(text)))
vocab_size = len(chars)

stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: "".join([itos[i] for i in l])

print(f"Dataset size: {len(text):,} characters")
print(f"Vocabulary size: {vocab_size} unique characters")
print(f"Sample: '{text[:20]}' -> {encode(text[:20])}")

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

config = GPTConfig(vocab_size=vocab_size)
batch_size = 32


def get_batch(split):
    d = train_data if split == "train" else val_data
    ix = torch.randint(len(d) - config.block_size, (batch_size,))
    x = torch.stack([d[i:i + config.block_size] for i in ix])
    y = torch.stack([d[i + 1:i + config.block_size + 1] for i in ix])
    return x.to(device), y.to(device)


@torch.no_grad()
def estimate_loss(model, eval_iters=50):
    model.eval()
    out = {}
    for split in ["train", "val"]:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            _, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean()
    model.train()
    return out


model = GPT(config).to(device)
param_count = sum(p.numel() for p in model.parameters())
print(f"Model parameters: {param_count:,}")

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

max_iters = 3000
eval_interval = 500

print("\nStarting training...")
for step in range(max_iters):
    if step % eval_interval == 0:
        losses = estimate_loss(model)
        print(
            f"Step {step:>5d} | "
            f"train loss: {losses['train']:.4f} | "
            f"val loss: {losses['val']:.4f}"
        )

    xb, yb = get_batch("train")
    logits, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

losses = estimate_loss(model)
print(
    f"Step {max_iters:>5d} | "
    f"train loss: {losses['train']:.4f} | "
    f"val loss: {losses['val']:.4f}"
)

torch.save({
    "model_state_dict": model.state_dict(),
    "config": config,
    "chars": chars,
}, "gpt_shakespeare.pt")
print("\nModel saved to gpt_shakespeare.pt")

Your model just went from random noise to something that has learned the statistical patterns of Elizabethan English. Next up, you will load the checkpoint and watch it generate Shakespeare-style text from any prompt you give it.

Generate Shakespeare-Style Text

Your model finished training and saved its learned weights to a checkpoint file. Now it is time for the payoff: watching your model write Shakespeare.

In this step, you will load the trained model, feed it a prompt, and let it generate text one character at a time. You will also explore how a single parameter called temperature controls whether your model writes conservatively or creatively.

In this step, get ready to:
  • Load the trained checkpoint and rebuild the model and tokenizer.
  • Generate 500 characters of Shakespeare-style text from a prompt.
  • Experiment with temperature to control coherence vs. creativity.
Load the checkpoint and rebuild the model

During training, you saved everything the model needs into gpt_shakespeare.pt: the trained weights, the config, and the character list. Now you need to unpack all of that and reconstruct a working model.

  • Open generate.py in VS Code. The file is currently empty.
  • Add the following code to load the checkpoint, rebuild the tokenizer, and reconstruct the model:
import torch
from model import GPT, GPTConfig

# Load the saved checkpoint from training
checkpoint = torch.load("gpt_shakespeare.pt", weights_only=False)
config = checkpoint["config"]
chars = checkpoint["chars"]

# Rebuild character-level tokenizer from the saved character list
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: "".join([itos[i] for i in l])

# Reconstruct the model architecture and load trained weights
device = "cuda" if torch.cuda.is_available() else "cpu"
model = GPT(config).to(device)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

What does this code do?

  • torch.load("gpt_shakespeare.pt", weights_only=False) reads the checkpoint file. The weights_only=False flag tells PyTorch to allow loading the full Python objects (like GPTConfig) stored inside.
  • The tokenizer is rebuilt from chars exactly the same way as in train.py. Saving the character list in the checkpoint guarantees encode and decode produce the same mappings the model was trained on.
  • model.eval() switches the model from training mode to inference mode. This disables dropout so the model uses everything it learned.
  • Save generate.py.

You will not see a change yet. The next chunk adds the generation code that actually produces output.

Run the generation pipeline

The model's generate method works autoregressively. It predicts one token, appends it to the sequence, and feeds the growing sequence back in to predict the next token. You just need to give it a starting prompt.

  • Add the following code to the bottom of generate.py, below the model.eval() line:
# Set the starting prompt and print a header
prompt = "To be or not to be"
print(f"Prompt: {prompt}")
print("-" * 50)

# Encode the prompt into token IDs, generate 500 new tokens, then decode back to text
context = torch.tensor([encode(prompt)], dtype=torch.long, device=device)
generated = model.generate(context, max_new_tokens=500, temperature=0.8)
print(decode(generated[0].tolist()))

What does this code do?

  • encode(prompt) converts the text string into a list of integer token IDs using the character-level tokenizer.
  • torch.tensor([encode(prompt)]) wraps the IDs into a tensor with shape (1, 18). The first dimension is the batch size (1 sequence).
  • model.generate() runs the autoregressive loop 500 times, predicting and appending one character each iteration.
  • temperature=0.8 controls how "surprising" each next-character choice is. Lower values make the model pick the most likely character. Higher values spread probability across more characters.
  • decode(generated[0].tolist()) converts the full sequence of token IDs (prompt + generated) back into readable text.
  • Save generate.py.
  • Run the script by running this command in your terminal:
python generate.py

You should see your prompt printed, a line of dashes, and then roughly 500 characters of Shakespeare-style text. The output will resemble Elizabethan English with words like "thou", "hath", and "wherefore" sprinkled throughout.

What should I expect from the output?

Your model has ~800K parameters and was trained on a single text file. The output will not be perfect Shakespeare. You should see recognizable English words, speaker names followed by colons, and text that looks like dialogue from a play.

Some words will be misspelled or nonsensical. That is expected for a character-level model this small. The impressive part is that it learned any coherent structure at all from raw characters.

Seeing a FileNotFoundError or KeyError?

  • If you see FileNotFoundError: gpt_shakespeare.pt, make sure you ran train.py to completion in the previous step. The checkpoint file is created at the very end of training.
  • If you see a KeyError, double-check the key names in your torch.load calls. They must match exactly: "config", "chars", and "model_state_dict".
  • Make sure your terminal is in the gpt-from-scratch folder, not a parent or child directory.

Why is my generate.py throwing an error?

✔️ Awesome, I've got everything!

Double check you have saved generate.py.

ⓧ I'd like to double check the full code

import torch
from model import GPT, GPTConfig

checkpoint = torch.load("gpt_shakespeare.pt", weights_only=False)
config = checkpoint["config"]
chars = checkpoint["chars"]

stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: "".join([itos[i] for i in l])

device = "cuda" if torch.cuda.is_available() else "cpu"
model = GPT(config).to(device)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

prompt = "To be or not to be"
print(f"Prompt: {prompt}")
print("-" * 50)

context = torch.tensor([encode(prompt)], dtype=torch.long, device=device)
generated = model.generate(context, max_new_tokens=500, temperature=0.8)
print(decode(generated[0].tolist()))
Experiment with temperature and prompts

Temperature is one of the most important parameters in text generation. It controls how the model samples from its predicted probability distribution over the next token.

What is temperature?

When the model predicts the next character, it produces a probability for every character in the vocabulary. Temperature scales these probabilities before sampling.

  • Low temperature (e.g. 0.5) sharpens the distribution, making the model strongly favor the most likely character. The output is more repetitive but more coherent.
  • High temperature (e.g. 1.5) flattens the distribution, giving less likely characters a bigger chance. The output is more creative but often drifts into nonsense.

Temperature 1.0 uses the raw probabilities with no scaling at all.

  • In generate.py, find this line near the bottom of the file:
generated = model.generate(context, max_new_tokens=500, temperature=0.8)
  • Change temperature=0.8 to temperature=0.5, so the line looks like this:
generated = model.generate(context, max_new_tokens=500, temperature=0.5)
  • Save generate.py and re-run it:
python generate.py

Notice how the output is more conservative. The model sticks closer to common Shakespeare patterns, and you may see repetitive phrases.

  • Now change temperature to 1.0, save, and re-run. Compare the output to what you saw at 0.5.
  • Change temperature to 1.5, save, and re-run. The text should look much more chaotic, with unusual character sequences and broken words.
  • Now try changing the prompt itself. Find this line in generate.py:
prompt = "To be or not to be"
  • Replace the prompt string with something different. Try a character name like ROMEO: or an opening line like O fair maiden.
  • Save and re-run generate.py each time to see how the prompt influences the model's continuation.

Try a character name with a colon

The training data is formatted as play scripts, with speaker names followed by colons. A prompt like HAMLET:\n often produces the most convincing output because it matches the structure the model saw during training.

  • When you are done experimenting, set the prompt back to "To be or not to be" and the temperature back to 0.8.
  • Save generate.py.

Secret mission

Upgrade to a BPE Tokenizer

Your character-level tokenizer treats every letter as its own token. Real LLMs like GPT-2 use something smarter called Byte Pair Encoding. Can you swap in GPT-2's actual tokenizer and see what difference it makes?

Before you go

Clean Up Your Resources

This project runs entirely on your local machine, so there are no ongoing cloud costs. Decide whether to keep your files around, or clean them up entirely.

Resources you used:

  • The gpt-from-scratch project folder on your Desktop.
  • Python files: model.py, train.py, and generate.py.
  • The TinyShakespeare dataset file: input.txt.
  • The saved model checkpoint: gpt_shakespeare.pt.

Keep everything running

No action needed. Choose this if you want to keep experimenting with prompts, temperatures, or model hyperparameters.

  • Your project folder, trained model, and dataset stay on your Desktop.
  • You can re-run generate.py any time to generate new text without retraining.
  • The total disk space used is under 10 MB.

Pause - I'll come back to this later

Since everything is local files, there is nothing to pause. Your files sit on disk and use no CPU or memory when you are not running a script.

  • Close any open terminal sessions running your scripts.
  • Close the project in VS Code if you want to free up the editor.
  • Your trained model checkpoint (gpt_shakespeare.pt) is saved. You can pick up right where you left off.

Delete - I don't want to use this again

Remove all project files from your machine. This deletes your trained model, dataset, and all code.

  • Close the project in VS Code.
  • Delete the entire gpt-from-scratch folder from your Desktop.
  • If you installed tiktoken for the Secret Mission and want to remove it, run pip uninstall tiktoken in your terminal.
  • PyTorch itself is useful for future deep learning projects, so consider keeping it installed.

Mission accomplished

That's a Wrap!

You just built a GPT language model from an empty file. Every component inside modern LLMs like GPT-4 and Claude now has a concrete, code-level meaning to you.

Here's what you accomplished:

  • Implemented scaled dot-product attention with causal masking from scratch, the core mechanism that lets transformers process sequences one token at a time.
  • Assembled a complete transformer architecture with multi-head attention, feed-forward networks, residual connections, pre-norm layer normalization, and weight tying.
  • Trained your model on Shakespeare and watched it learn to generate coherent text from completely random weights.
  • Upgraded from character-level to BPE tokenization and saw firsthand how tokenization strategy affects what a model can learn.

Ready to quiz yourself?