Build a Lexer for Mini-Python in Rust

Build a lexer in Rust that tokenizes Python-like source code into typed tokens.

Introduction

30 Second Summary

Every programming language you use, from Python to JavaScript, starts the same way. It reads your code as raw text and figures out what each piece means by breaking that text into meaningful chunks.

In this project, you will build a lexer in Rust that takes Python-like source code and converts it into a stream of classified tokens. You will learn Rust's core features by building a real compiler component from scratch.

What You'll Build

A command-line Rust program that reads a string of Python-like source code and outputs a list of tokens identifying each number, keyword, operator, and identifier.

By the end of this project, you'll have:

  • A Token enum in Rust representing every possible piece of Mini-Python syntax.
  • A Lexer struct that scans characters one by one and classifies them into tokens using pattern matching.
  • A working tokenizer that processes multi-line Python snippets and prints the full token stream.
  • Secret Mission: Add line and column tracking to every token for error reporting.

Are there any prerequisites?

You should be familiar with Python syntax (variables, if statements, print). No Rust experience is required. This project teaches you Rust from the ground up.

Set Up Your Rust Project

Before you can build a lexer, you need a Rust project set up and ready to compile code.

In this step, you will verify your Rust installation, create a new project with Cargo (Rust's build tool and package manager), and confirm everything compiles.

In this step, get ready to:
  • Verify your Rust and Cargo installation.
  • Create a new Cargo project for the lexer.
  • Confirm the project compiles and runs successfully.
Verify your Rust installation

Your lexer will be written in Rust, so the compiler and build tool need to be working on your machine.

  • Check your Rust compiler and Cargo versions by running these commands:
rustc --version
cargo --version

What should I see?

You should see version numbers printed for both commands, such as rustc 1.96.0 and a matching cargo version.

Any recent stable version will work for this project.

  • If you want to update to the latest stable version, run this command:
rustup update stable

Command not found?

If rustc or cargo is not recognized, your Rust installation may not be on your system PATH.

Try closing and reopening your terminal first. If the issue persists, reinstall Rust from rustup.rs and follow the prompts to add it to your PATH.

Why can't my terminal find rustc or cargo?

Create your Rust project

Cargo can scaffold an entire Rust project for you with a single command. This gives you a ready-to-compile project structure with zero manual setup.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your lexer project by running this command:
cargo new mini_python_lexer

You should see a message confirming the project was created.

  • Move into your new project folder by running:
cd mini_python_lexer
  • Confirm the project files exist by running:
ls

You should see Cargo.toml and a src folder in the output.

Now open this project in VS Code.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the mini_python_lexer folder, and click Open.

What did Cargo generate?

In the VS Code sidebar, you will see two key items.

  • Cargo.toml is the project configuration file. It defines your project name, version, and dependencies.
  • src/main.rs is where your Rust code lives. Cargo generated a default "Hello, world!" program here.
Run the default program

Every new Cargo project comes with a working "Hello, world!" program out of the box. Running it confirms your compiler and build system are working correctly.

  • Open a terminal in VS Code by pressing Ctrl+` (backtick key, below Escape).
  • Compile and run the default program by running this command:
cargo run

What does this command do?

The cargo run command does two things: it compiles your Rust source code into a binary, then immediately runs that binary.

The first time you run it, Cargo also downloads and compiles any dependencies (none in this project). This one-step compile-and-run workflow is what makes Cargo so convenient.

You should see Hello, world! printed in your terminal.

Not seeing Hello, world!?

Make sure your terminal is inside the mini_python_lexer folder. Run pwd to check your current location.

If you see a compilation error, check that src/main.rs has not been modified from the default. You can re-create the project if needed.

Help me debug my cargo run error.

✔️ Awesome, I've got everything!

Great. Double check you see Hello, world! in your terminal output, then move on.

ⓧ I'd like to double check the full code

Here is what your Cargo.toml should look like. The edition field may show a different year depending on your Rust version. That is fine.

[package]
name = "mini_python_lexer"
version = "0.1.0"
edition = "2021"

[dependencies]

Here is what src/main.rs should look like.

fn main() {
    println!("Hello, world!");
}

Your Rust project is ready to go. Next up, you will define the token types that your lexer will recognize in Python source code.

Define Your Token Types

Your Rust project compiles and runs. Now it's time to teach your lexer what kinds of tokens exist.

Before the lexer can scan source code, it needs a vocabulary. Every meaningful chunk (a number, an operator, a keyword) needs its own type. In Rust, the perfect tool for this is an enum: a single type that can be any one of several named variants.

In this step, get ready to:
  • Define a Token enum with all the token types your Mini-Python lexer will recognize.
  • Test the enum by creating and printing sample tokens.
Define the Token enum

A lexer classifies every chunk of source code into a category. The number 42 is a numeric literal. The symbol + is an operator. The word print is a keyword. In Rust, you model these categories as variants of an enum.

  • In VS Code, open src/main.rs.
  • Select all the existing code and replace it with this Token enum definition:
#[derive(Debug, PartialEq)]
enum Token {
    // Literals
    Number(i64),
    StringLit(String),

    // Identifiers and keywords
    Ident(String),
    Print,
    If,
    Else,

    // Operators
    Plus,
    Minus,
    Star,
    Slash,
    Equals,
    EqualEqual,
    BangEqual,
    Less,
    Greater,

    // Delimiters
    LeftParen,
    RightParen,
    Colon,

    // Special
    Eof,
    Illegal(char),
}

What does this code do?

#[derive(Debug, PartialEq)] automatically gives your enum two abilities: Debug lets you print tokens with {:?} for inspection, and PartialEq lets you compare tokens with ==.

Each variant represents a different kind of token. Some carry data (Number(i64) holds the numeric value, Ident(String) holds the identifier name). Others are simple labels like Plus or Print that need no extra data.

Illegal(char) captures any character the lexer doesn't recognize, and Eof signals the end of the input.

  • Save src/main.rs.
  • You won't see any output yet. The next substep adds a main function to test these tokens.
Create sample tokens to test the enum

An enum definition alone does nothing visible. You need a main function that creates a few token values and prints them to confirm everything works.

  • Below the closing } of the Token enum, add this main function:
fn main() {
    let t1 = Token::Number(42);
    let t2 = Token::Plus;
    let t3 = Token::Ident(String::from("hello"));
    println!("{:?}", t1);
    println!("{:?}", t2);
    println!("{:?}", t3);
}

What does this code do?

Token::Number(42) creates a Number variant holding the value 42. Token::Plus creates a simple label variant with no data. Token::Ident(String::from("hello")) creates an identifier token holding an owned String.

println!("{:?}", t1) uses the Debug trait (from #[derive(Debug)]) to print each token's variant name and any data it carries.

  • Save src/main.rs.
  • Compile and run your program by running:
cargo run

You should see three lines printed in your terminal:

Number(42)
Plus
Ident("hello")

Seeing a compiler error?

  • If you see cannot find type `Token`, make sure the enum Token { ... } block appears above the fn main() function in the same file.
  • If you see a syntax error, check that every variant inside the enum ends with a comma and that the closing brace } is present.
  • If you accidentally left the old Hello World code in the file, select all and replace with just the enum and main function.

Help me fix a compiler error in my Token enum Rust code

✔️ Awesome, I've got everything!

Great. Double check you've saved src/main.rs and move on to the next step.

ⓧ I'd like to double check the full code

Here's the full src/main.rs file at this point:

#[derive(Debug, PartialEq)]
enum Token {
    // Literals
    Number(i64),
    StringLit(String),

    // Identifiers and keywords
    Ident(String),
    Print,
    If,
    Else,

    // Operators
    Plus,
    Minus,
    Star,
    Slash,
    Equals,
    EqualEqual,
    BangEqual,
    Less,
    Greater,

    // Delimiters
    LeftParen,
    RightParen,
    Colon,

    // Special
    Eof,
    Illegal(char),
}

fn main() {
    let t1 = Token::Number(42);
    let t2 = Token::Plus;
    let t3 = Token::Ident(String::from("hello"));
    println!("{:?}", t1);
    println!("{:?}", t2);
    println!("{:?}", t3);
}

Your token vocabulary is defined. Next up, you'll build the lexer engine that reads source code character by character and matches each chunk to one of these token types.

Build the Lexer Core

Your Token enum defines what tokens exist, but it cannot read source code on its own. You need a separate piece of machinery that walks through characters and decides which token each one belongs to.

This machinery is the Lexer. A lexer reads input one character at a time, classifies each character, and collects groups of characters into complete tokens. In this step, you will build the full Lexer struct with all the methods it needs to handle every token type in your enum.

In this step, get ready to:
  • Define a Lexer struct that holds the input and tracks your position in it.
  • Implement scanning methods that collect multi-character tokens like numbers, identifiers, and strings.
  • Build a token dispatcher that classifies each character into the correct token type.
Define the Lexer struct and helper methods

The lexer needs to track two things: the source code it is reading, and where it currently is in that source code. You will store the source as a Vec<char> (a vector of individual characters) and the current position as a usize index.

  • In src/main.rs, find the closing brace } of your Token enum.
  • Add the following code below it (above the fn main() function):
struct Lexer {
    input: Vec<char>,
    position: usize,
}

impl Lexer {
    fn new(input: &str) -> Self {
        Lexer {
            input: input.chars().collect(),
            position: 0,
        }
    }

    fn peek(&self) -> Option<char> {
        self.input.get(self.position).copied()
    }

    fn advance(&mut self) -> Option<char> {
        let ch = self.input.get(self.position).copied();
        self.position += 1;
        ch
    }
}

What does this code do?

  • struct Lexer defines the lexer's state: input holds all characters and position tracks where you are in the input.
  • new takes a string slice (&str) and converts it into a Vec<char> so you can index into it by position.
  • peek looks at the current character without moving forward. It returns Option<char> because the input might be exhausted.
  • advance consumes the current character and moves position forward by one. This is how the lexer "eats" through the source code.

The .copied() call converts Option<&char> (a reference) into Option<char> (an owned value). This works because char implements the Copy trait.

  • Save src/main.rs.
  • Run cargo run in your terminal to confirm the project still compiles. You should still see the sample token output from your main function.

Seeing a compile error?

  • Check that the struct and impl block are placed between the Token enum's closing brace and fn main(). Rust requires items to be defined before they are used.
  • Make sure you have matching curly braces for both the struct and the impl block.
  • help me fix a compile error in my Lexer struct
Implement the scanning methods

Before you build the main dispatch logic, you need helper methods that know how to collect multi-character tokens. Each scanner follows the same pattern: loop through characters while a condition holds, collect them into a value, then return the appropriate token.

  • Inside your impl Lexer block, add the following four methods after the advance method (before the closing } of the impl block):
    fn skip_whitespace(&mut self) {
        while let Some(ch) = self.peek() {
            if ch.is_ascii_whitespace() {
                self.advance();
            } else {
                break;
            }
        }
    }

    fn scan_number(&mut self) -> Token {
        let mut number_str = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_ascii_digit() {
                number_str.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        let value = number_str.parse::<i64>().unwrap();
        Token::Number(value)
    }

    fn scan_identifier(&mut self) -> Token {
        let mut ident = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_alphanumeric() || ch == '_' {
                ident.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        match ident.as_str() {
            "print" => Token::Print,
            "if" => Token::If,
            "else" => Token::Else,
            _ => Token::Ident(ident),
        }
    }

    fn scan_string(&mut self) -> Token {
        self.advance(); // consume opening quote
        let mut string_content = String::new();
        while let Some(ch) = self.peek() {
            if ch == '"' {
                self.advance(); // consume closing quote
                return Token::StringLit(string_content);
            }
            string_content.push(ch);
            self.advance();
        }
        Token::Illegal('"') // unterminated string
    }

What do these methods do?

  • skip_whitespace advances past spaces, tabs, and newlines until it hits a non-whitespace character.
  • scan_number collects consecutive digits into a string, parses it as an i64, and returns a Number token.
  • scan_identifier collects alphanumeric and underscore characters, then checks if the result matches a keyword (print, if, else). If not, it returns a generic Ident token.
  • scan_string consumes the opening quote, collects everything until a closing quote, and returns a StringLit token. If no closing quote is found, it returns Illegal.

All four methods use &mut self because they modify the lexer's position field as they advance through characters.

  • Save src/main.rs.
  • Run cargo run to confirm everything still compiles without errors. You should see the same sample token output as before.

Compile error mentioning Token?

  • The scan methods return Token values, so the Token enum must be defined above the impl Lexer block. Confirm your file order is: Token enum, then Lexer struct, then impl block.
  • Check that the match arms in scan_identifier use => (fat arrow) and end with commas.
  • help me fix a compile error in my scan methods
Build the token dispatcher

The next_token method is the heart of the lexer. It looks at the current character and decides which token to produce. Single characters map directly to tokens. Multi-character sequences (like ==) use a peek-ahead technique to check what comes next.

  • Inside your impl Lexer block, add the next_token method after scan_string (before the closing } of the impl block):
    fn next_token(&mut self) -> Token {
        self.skip_whitespace();

        let ch = match self.peek() {
            Some(ch) => ch,
            None => return Token::Eof,
        };

        match ch {
            '+' => { self.advance(); Token::Plus }
            '-' => { self.advance(); Token::Minus }
            '*' => { self.advance(); Token::Star }
            '/' => { self.advance(); Token::Slash }
            '(' => { self.advance(); Token::LeftParen }
            ')' => { self.advance(); Token::RightParen }
            ':' => { self.advance(); Token::Colon }
            '=' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::EqualEqual
                } else {
                    Token::Equals
                }
            }
            '!' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::BangEqual
                } else {
                    Token::Illegal('!')
                }
            }
            '<' => { self.advance(); Token::Less }
            '>' => { self.advance(); Token::Greater }
            '"' => self.scan_string(),
            c if c.is_ascii_digit() => self.scan_number(),
            c if c.is_alphabetic() || c == '_' => self.scan_identifier(),
            c => { self.advance(); Token::Illegal(c) }
        }
    }

How does the dispatch work?

  • First, skip_whitespace moves past any spaces or newlines so you start on a meaningful character.
  • If there is no character left (peek() returns None), the method returns Token::Eof to signal end of input.
  • Single-character operators like + advance once and return the token directly.
  • For =, the lexer advances past it, then peeks at the next character. If it sees another =, it returns EqualEqual. Otherwise it returns Equals. The same peek-ahead pattern handles !=.
  • Match guards like c if c.is_ascii_digit() delegate to the appropriate scanner method for multi-character tokens.

Now replace your existing main function with this version that tests the lexer:

fn main() {
    let mut lexer = Lexer::new("x = 42 + y");
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
}
  • Save src/main.rs.
  • Run cargo run in your terminal. You should see this output:
Ident("x")
Equals
Number(42)
Plus
Ident("y")
Eof

Each token in the output corresponds to a piece of the input x = 42 + y. The lexer correctly identified identifiers, the assignment operator, a number, the plus operator, and the end of input.

Output doesn't match?

  • If you see Illegal tokens in the output, check that your next_token match arms include the guards for digits (c if c.is_ascii_digit()) and letters (c if c.is_alphabetic() || c == '_'). These must appear before the final c => fallthrough arm.
  • If you see a "dead code" warning about the old main function, make sure you fully replaced it rather than adding a second main function.
  • help me debug my next_token output

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.rs and that cargo run shows the correct token output.

ⓧ I'd like to double check the full code

Here is the complete src/main.rs file at this point:

#[derive(Debug, PartialEq)]
enum Token {
    // Literals
    Number(i64),
    StringLit(String),

    // Identifiers and keywords
    Ident(String),
    Print,
    If,
    Else,

    // Operators
    Plus,
    Minus,
    Star,
    Slash,
    Equals,
    EqualEqual,
    BangEqual,
    Less,
    Greater,

    // Delimiters
    LeftParen,
    RightParen,
    Colon,

    // Special
    Eof,
    Illegal(char),
}

struct Lexer {
    input: Vec<char>,
    position: usize,
}

impl Lexer {
    fn new(input: &str) -> Self {
        Lexer {
            input: input.chars().collect(),
            position: 0,
        }
    }

    fn peek(&self) -> Option<char> {
        self.input.get(self.position).copied()
    }

    fn advance(&mut self) -> Option<char> {
        let ch = self.input.get(self.position).copied();
        self.position += 1;
        ch
    }

    fn skip_whitespace(&mut self) {
        while let Some(ch) = self.peek() {
            if ch.is_ascii_whitespace() {
                self.advance();
            } else {
                break;
            }
        }
    }

    fn scan_number(&mut self) -> Token {
        let mut number_str = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_ascii_digit() {
                number_str.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        let value = number_str.parse::<i64>().unwrap();
        Token::Number(value)
    }

    fn scan_identifier(&mut self) -> Token {
        let mut ident = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_alphanumeric() || ch == '_' {
                ident.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        match ident.as_str() {
            "print" => Token::Print,
            "if" => Token::If,
            "else" => Token::Else,
            _ => Token::Ident(ident),
        }
    }

    fn scan_string(&mut self) -> Token {
        self.advance(); // consume opening quote
        let mut string_content = String::new();
        while let Some(ch) = self.peek() {
            if ch == '"' {
                self.advance(); // consume closing quote
                return Token::StringLit(string_content);
            }
            string_content.push(ch);
            self.advance();
        }
        Token::Illegal('"') // unterminated string
    }

    fn next_token(&mut self) -> Token {
        self.skip_whitespace();

        let ch = match self.peek() {
            Some(ch) => ch,
            None => return Token::Eof,
        };

        match ch {
            '+' => { self.advance(); Token::Plus }
            '-' => { self.advance(); Token::Minus }
            '*' => { self.advance(); Token::Star }
            '/' => { self.advance(); Token::Slash }
            '(' => { self.advance(); Token::LeftParen }
            ')' => { self.advance(); Token::RightParen }
            ':' => { self.advance(); Token::Colon }
            '=' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::EqualEqual
                } else {
                    Token::Equals
                }
            }
            '!' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::BangEqual
                } else {
                    Token::Illegal('!')
                }
            }
            '<' => { self.advance(); Token::Less }
            '>' => { self.advance(); Token::Greater }
            '"' => self.scan_string(),
            c if c.is_ascii_digit() => self.scan_number(),
            c if c.is_alphabetic() || c == '_' => self.scan_identifier(),
            c => { self.advance(); Token::Illegal(c) }
        }
    }
}

fn main() {
    let mut lexer = Lexer::new("x = 42 + y");
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
}

Your lexer can now break any combination of operators, numbers, identifiers, and strings into tokens. Next up, you will feed it a full multi-line Python program and collect all the tokens into a single list.

Tokenize a Full Program

In the last step, you built all the individual scanning methods your lexer needs to recognize every token type in Mini-Python. Right now, your main function calls next_token manually one call at a time. That works for testing, but a real lexer needs to process an entire source file in one pass.

In this step, you'll add a tokenize method that loops through the entire input and collects every token into a list. Then you'll test it on a multi-line Python snippet to see your complete lexer in action.

In this step, get ready to:
  • Add a tokenize method that scans the full input into a Vec<Token>.
  • Replace main with a multi-line Python test program and print the full token stream.
  • Verify the output matches the expected token sequence.
Implement the tokenize method

Right now you call next_token once per token and print each one manually. A tokenize method automates this by looping until it sees Token::Eof, collecting every token into a Vec<Token>.

  • In src/main.rs, scroll to the bottom of your impl Lexer block (after the closing brace of next_token).
  • Add the following tokenize method before the closing } of the impl block:
    fn tokenize(&mut self) -> Vec<Token> {
        let mut tokens = Vec::new();
        loop {
            let token = self.next_token();
            // Check if we've reached the end before pushing
            let is_eof = token == Token::Eof;
            tokens.push(token);
            if is_eof {
                break;
            }
        }
        tokens
    }

What does this code do?

This method drives the entire lexing process from start to finish.

  • Vec::new() creates an empty vector to hold the collected tokens.
  • The loop calls next_token repeatedly. Each call advances the lexer's position through the input.
  • We check is_eof before pushing because once we push the token, we've moved ownership into the vector and can no longer compare it. This is Rust's ownership system in action.
  • The Eof token is still pushed into the vector so consumers of the token stream know exactly where the input ended.

Why does the equality check work here?

The token == Token::Eof comparison works because you added #[derive(PartialEq)] to the Token enum back in Step 2. Without that derive, Rust would refuse to compile the == operator on your custom type.

  • Save src/main.rs.

Seeing a compiler error about PartialEq?

  • Make sure your Token enum still has #[derive(Debug, PartialEq)] above it. The comparison in tokenize requires this.
  • Check that the tokenize method is inside the impl Lexer { ... } block, not outside it.

Help me fix a PartialEq compiler error in my tokenize method

Replace main with a multi-line test program

Your current main function tests a single expression. To prove your lexer handles a real program, you'll feed it a multi-line Python snippet with variables, strings, keywords, and operators all mixed together.

  • In src/main.rs, find your current main function:
fn main() {
    let mut lexer = Lexer::new("x = 42 + y");
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
    println!("{:?}", lexer.next_token());
}
  • Replace the entire main function with the following code:
fn main() {
    // A multi-line Python snippet to tokenize
    let source = r#"x = 42
name = "hello"
if x == 42:
    print(name)
else:
    print("goodbye")
"#;

    let mut lexer = Lexer::new(source);
    let tokens = lexer.tokenize();

    // Print each token on its own line
    for token in &tokens {
        println!("{:?}", token);
    }
}

What does this code do?

  • r#"..."# is a Rust raw string literal. It lets you include double quotes and newlines without escaping them. Perfect for embedding source code as test input.
  • The test snippet covers variables (x, name), numbers (42), strings ("hello"), keywords (if, else, print), and operators (=, ==).
  • for token in &tokens borrows the vector instead of consuming it. The & means "I want to look at these tokens, not take ownership of them."
  • Save src/main.rs.
  • Run your program by executing this command in the terminal:
cargo run

You should see the full token stream printed, one token per line. The output starts with Ident("x") and ends with Eof.

What should I see?

Your terminal should print something like this:

Ident("x")
Equals
Number(42)
Ident("name")
Equals
StringLit("hello")
If
Ident("x")
EqualEqual
Number(42)
Colon
Print
LeftParen
Ident("name")
RightParen
Else
Colon
Print
LeftParen
StringLit("goodbye")
RightParen
Eof

Each line corresponds to one token your lexer extracted from the source code. The lexer correctly skipped all whitespace (spaces, tabs, newlines) and classified every character sequence into the right token type.

Output doesn't match or you see Illegal tokens?

  • If you see Illegal('"'), check that your raw string literal uses r#" at the start and "# at the end. Regular strings would need escaped quotes.
  • If you see extra tokens or missing tokens, double-check that your test snippet exactly matches the code above. Even an extra space can produce a different token count.
  • If the program doesn't compile at all, make sure the tokenize method is inside the impl Lexer block and that you didn't accidentally delete the closing } of the impl.

Help me debug unexpected output from my Rust lexer

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete src/main.rs file as it should look at the end of this step:

#[derive(Debug, PartialEq)]
enum Token {
    // Literals
    Number(i64),
    StringLit(String),

    // Identifiers and keywords
    Ident(String),
    Print,
    If,
    Else,

    // Operators
    Plus,
    Minus,
    Star,
    Slash,
    Equals,
    EqualEqual,
    BangEqual,
    Less,
    Greater,

    // Delimiters
    LeftParen,
    RightParen,
    Colon,

    // Special
    Eof,
    Illegal(char),
}

struct Lexer {
    input: Vec<char>,
    position: usize,
}

impl Lexer {
    fn new(input: &str) -> Self {
        Lexer {
            input: input.chars().collect(),
            position: 0,
        }
    }

    fn peek(&self) -> Option<char> {
        self.input.get(self.position).copied()
    }

    fn advance(&mut self) -> Option<char> {
        let ch = self.input.get(self.position).copied();
        self.position += 1;
        ch
    }

    fn skip_whitespace(&mut self) {
        while let Some(ch) = self.peek() {
            if ch.is_ascii_whitespace() {
                self.advance();
            } else {
                break;
            }
        }
    }

    fn scan_number(&mut self) -> Token {
        let mut number_str = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_ascii_digit() {
                number_str.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        let value = number_str.parse::<i64>().unwrap();
        Token::Number(value)
    }

    fn scan_identifier(&mut self) -> Token {
        let mut ident = String::new();
        while let Some(ch) = self.peek() {
            if ch.is_alphanumeric() || ch == '_' {
                ident.push(ch);
                self.advance();
            } else {
                break;
            }
        }
        match ident.as_str() {
            "print" => Token::Print,
            "if" => Token::If,
            "else" => Token::Else,
            _ => Token::Ident(ident),
        }
    }

    fn scan_string(&mut self) -> Token {
        self.advance(); // consume opening quote
        let mut string_content = String::new();
        while let Some(ch) = self.peek() {
            if ch == '"' {
                self.advance(); // consume closing quote
                return Token::StringLit(string_content);
            }
            string_content.push(ch);
            self.advance();
        }
        Token::Illegal('"') // unterminated string
    }

    fn next_token(&mut self) -> Token {
        self.skip_whitespace();

        let ch = match self.peek() {
            Some(ch) => ch,
            None => return Token::Eof,
        };

        match ch {
            '+' => { self.advance(); Token::Plus }
            '-' => { self.advance(); Token::Minus }
            '*' => { self.advance(); Token::Star }
            '/' => { self.advance(); Token::Slash }
            '(' => { self.advance(); Token::LeftParen }
            ')' => { self.advance(); Token::RightParen }
            ':' => { self.advance(); Token::Colon }
            '=' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::EqualEqual
                } else {
                    Token::Equals
                }
            }
            '!' => {
                self.advance();
                if self.peek() == Some('=') {
                    self.advance();
                    Token::BangEqual
                } else {
                    Token::Illegal('!')
                }
            }
            '<' => { self.advance(); Token::Less }
            '>' => { self.advance(); Token::Greater }
            '"' => self.scan_string(),
            c if c.is_ascii_digit() => self.scan_number(),
            c if c.is_alphabetic() || c == '_' => self.scan_identifier(),
            c => { self.advance(); Token::Illegal(c) }
        }
    }

    fn tokenize(&mut self) -> Vec<Token> {
        let mut tokens = Vec::new();
        loop {
            let token = self.next_token();
            let is_eof = token == Token::Eof;
            tokens.push(token);
            if is_eof {
                break;
            }
        }
        tokens
    }
}

fn main() {
    let source = r#"x = 42
name = "hello"
if x == 42:
    print(name)
else:
    print("goodbye")
"#;

    let mut lexer = Lexer::new(source);
    let tokens = lexer.tokenize();

    for token in &tokens {
        println!("{:?}", token);
    }
}

Your lexer is complete. It takes raw Python source code and produces a clean, structured stream of tokens. This is exactly what the first stage of a real compiler does. Next up, you'll tackle the Secret Mission and add line and column tracking to every token for error reporting.

Secret mission

Add Location Tracking

Your lexer produces tokens, but if a user makes a syntax error, you can't tell them where the problem is. Real compilers report errors like 'Line 5, Col 12: unexpected character'. Can you upgrade your lexer to track the line and column of every token it produces?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project folder for future use or delete it entirely.

Resources you used:

  • The mini_python_lexer project folder on your Desktop (contains Cargo.toml and src/main.rs).

Keep everything running

No action needed. Choose this if you plan to build on this lexer in a follow-up project (like building a parser on top of it).

  • Your mini_python_lexer folder stays on your Desktop, ready to extend.
  • Your Rust toolchain remains installed and available for other projects.

Pause - I'll come back to this later

Since this is a local project with no running processes, there is nothing to pause. Your files will be exactly where you left them when you come back.

  • Close the VS Code window for this project.
  • Your mini_python_lexer folder remains on your Desktop.

Delete - I don't want to use this again

Remove the project folder to start fresh. This deletes all your lexer code permanently.

  • Close VS Code if it still has the project open.
  • Delete the project folder by running this command in your terminal:

macOS

rm -rf ~/Desktop/mini_python_lexer

Windows

rmdir /s /q %USERPROFILE%\Desktop\mini_python_lexer
  • Confirm the folder is gone by checking your Desktop. You should no longer see mini_python_lexer listed.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete lexer from scratch in Rust with zero external dependencies.

You've learned how to:

  • Model a token stream using Rust enums with data variants, structs, and pattern matching to classify characters into typed tokens.
  • Handle multi-character tokens (==, !=, numbers, identifiers, strings) using peek-ahead logic and loop-based scanning methods.
  • Work with Rust's ownership and borrowing system through &str, String, Vec<char>, and the difference between &self and &mut self.
  • Secret Mission: Added line and column tracking to every token for error reporting.

Ready to quiz yourself?