Train a DQN Agent for Vehicle Navigation
Build a DQN agent in PyTorch that learns to navigate a vehicle through obstacles.
Introduction
30 Second Summary
Self-driving cars learn to navigate by making thousands of decisions and improving from their mistakes. The same technique powers robots, game AI, and any system that needs to figure out the right action without being explicitly programmed.
In this project, you will build a Deep Q-Network agent that learns to drive a vehicle through obstacles to reach a goal. You'll create the environment from scratch, implement the algorithm in PyTorch, and watch your agent improve in real-time.
What You'll Build
A reinforcement learning system where a DQN agent learns autonomous navigation through a custom 2D environment built with Gymnasium and rendered with Pygame.
By the end of this project, you'll have:
- A custom Gymnasium-compatible navigation environment with obstacles, a goal, and reward shaping.
- A complete DQN agent with replay buffer, target network, and epsilon-greedy exploration built from scratch in PyTorch.
- Training visualizations showing reward curves and exploration decay saved as a portfolio-ready image.
- Secret Mission: Implement Double DQN to reduce Q-value overestimation and improve navigation stability.
Are there any prerequisites?
You need Python 3.10 or higher installed, VS Code set up, and an NVIDIA GPU with drivers installed. This project uses your GPU to accelerate training, so a CUDA-capable graphics card is required.
Before We Start
Training a Deep Q-Network agent requires a specific set of tools working together. You need a GPU-accelerated deep learning framework, a reinforcement learning environment API, and a real-time visualization library.
In this step, you will create a project folder, set up an isolated Python environment, and install PyTorch with GPU support alongside Gymnasium, Pygame, NumPy, and Matplotlib.
In this step, get ready to:
- Create a project folder with a Python virtual environment.
- Install PyTorch with CUDA and all supporting packages.
- Verify your GPU is accessible for training.
Create a project folder and virtual environment
A virtual environment keeps your project's packages isolated from the rest of your system. This prevents version conflicts between projects.
- Press the Windows key to open your search bar.
- Type PowerShell and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir dqn-navigation
cd dqn-navigation- Open this folder in VS Code by running:
code .What does this command do?
The code . command opens VS Code with your current folder as the workspace. The dot (.) means "this directory".
- In VS Code, open the integrated terminal by pressing Ctrl+` (backtick key, top-left of your keyboard).
- Create a Python virtual environment by running:
python -m venv venv- Activate the virtual environment by running:
venv\Scripts\activateYou should see (venv) appear at the beginning of your terminal prompt.
Seeing a script execution error?
PowerShell may block script execution by default. Run this command to allow activation scripts, then try the activate command again:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUserHelp me fix this execution policy error.
Install all dependencies
PyTorch needs to be installed with CUDA support to use your GPU for training. The other packages handle environment simulation, rendering, and plotting.
- Install PyTorch with CUDA 12.6 support by running this command:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126What does this command do?
The --index-url flag points pip at a special package index that contains GPU-enabled builds of PyTorch. Without it, you would get the CPU-only version which trains much slower.
The cu126 part specifies CUDA 12.6 compatibility, which works with your RTX 4050 GPU.
This download is large
PyTorch with CUDA is approximately 2-3 GB. The download may take a few minutes depending on your internet connection.
- Install the remaining packages by running:
pip install gymnasium==1.3.0 pygame==2.6.1 numpy matplotlibWhat are these packages for?
- gymnasium provides the standard API for building reinforcement learning environments.
- pygame handles real-time rendering so you can watch the agent navigate.
- numpy handles array math for observations and rewards.
- matplotlib creates training performance plots.
Installation failed?
Make sure you see (venv) in your terminal prompt. If not, the virtual environment is not active and packages will install globally.
If you see a network error, check your internet connection and try the command again.
Help me troubleshoot my package installation.
Verify CUDA is accessible
Before writing any training code, confirm that PyTorch can see your GPU. This catches driver issues early before they become confusing errors mid-training.
- Check that PyTorch detects your GPU by running:
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"You should see two lines printed: True on the first line and NVIDIA GeForce RTX 4050 on the second.
What is this checking?
torch.cuda.is_available() confirms PyTorch can communicate with your GPU through CUDA drivers.
torch.cuda.get_device_name(0) prints the name of your first GPU, confirming the exact hardware PyTorch will use for tensor operations during training.
Seeing False or an error?
If torch.cuda.is_available() returns False, your NVIDIA drivers may be outdated. Update them from the NVIDIA website, restart your computer, and try again.
If you see a ModuleNotFoundError, PyTorch did not install correctly. Make sure your virtual environment is active and re-run the PyTorch install command.
Help me get PyTorch to detect my GPU.
Your environment is ready. Next up, you will build a custom Gymnasium environment that simulates a vehicle navigating through obstacles.
Build the Custom Navigation Environment
Your tools are installed and your GPU is ready. Now you need a world for your agent to learn in.
In reinforcement learning, the environment defines everything the agent interacts with. It determines what the agent sees (observations), what it can do (actions), and how it gets scored (rewards). You will build this environment from scratch using the Gymnasium API so that your DQN agent can plug into it with a standard interface.
In this step, get ready to:
- Create a custom Gymnasium environment with a vehicle, goal, and obstacles.
- Define the reward structure that teaches the agent good navigation behavior.
- Add Pygame rendering for live visualization during evaluation.
Create the environment file and define the class
The VehicleNavigationEnv class subclasses gymnasium.Env to follow the standard RL environment interface. The __init__ method sets up the arena dimensions, object sizes, and tells Gymnasium what the observation and action spaces look like.
- In VS Code, create a new file called environment.py inside your dqn-navigation folder.
- Add the following code to set up the class and its initialization:
import gymnasium as gym
from gymnasium import spaces
import numpy as np
import pygame
class VehicleNavigationEnv(gym.Env):
metadata = {"render_modes": ["human", "rgb_array"], "render_fps": 30}
def __init__(self, render_mode=None):
super().__init__()
self.render_mode = render_mode
self.width = 800
self.height = 600
self.vehicle_size = 20
self.obstacle_size = 30
self.goal_size = 20
self.step_size = 20
self.n_obstacles = 5
self.max_steps = 200
self.action_space = spaces.Discrete(4)
self.observation_space = spaces.Box(
low=0.0, high=1.0, shape=(14,), dtype=np.float32
)
self.window = None
self.clock = NoneWhat does this code do?
The arena is 800x600 pixels. The vehicle, goal, and obstacles are all squares of different sizes.
- spaces.Discrete(4) means the agent has 4 possible actions: up, down, left, and right.
- spaces.Box(low=0.0, high=1.0, shape=(14,)) means the agent observes 14 float values between 0 and 1. These will encode the positions of the vehicle, goal, and all 5 obstacles.
- max_steps = 200 caps each episode so the agent cannot wander forever.
- Save environment.py.
Why 14 dimensions?
The observation vector encodes: 2 values for the vehicle position (x, y), 2 for the goal position, and 2 for each of the 5 obstacles. That gives 2 + 2 + (5 * 2) = 14 total values. All are normalized to the 0-1 range by dividing by the arena dimensions.
Implement the reset and step methods
Every Gymnasium environment needs two core methods. The reset() method initializes a fresh episode with randomized positions. The step() method takes an action, moves the vehicle, checks for collisions, and returns the new state with a reward signal.
- In environment.py, add the reset() method below the __init__ method:
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.steps = 0
self.vehicle_x = self.np_random.integers(50, self.width // 3)
self.vehicle_y = self.np_random.integers(50, self.height - 50)
self.goal_x = self.np_random.integers(2 * self.width // 3, self.width - 50)
self.goal_y = self.np_random.integers(50, self.height - 50)
self.obstacles = []
for _ in range(self.n_obstacles):
ox = self.np_random.integers(self.width // 4, 3 * self.width // 4)
oy = self.np_random.integers(50, self.height - 50)
self.obstacles.append((ox, oy))
observation = self._get_observation()
return observation, {}What does this code do?
Each episode starts fresh with randomized positions so the agent cannot memorize a single layout.
- The vehicle always spawns in the left third of the arena.
- The goal always spawns in the right third.
- Obstacles are scattered in the middle region, forcing the agent to navigate around them.
- The method returns a tuple of (observation, info) as required by the Gymnasium API.
- Save environment.py.
- Now add the step() method directly below reset():
def step(self, action):
self.steps += 1
if action == 0:
self.vehicle_y -= self.step_size
elif action == 1:
self.vehicle_y += self.step_size
elif action == 2:
self.vehicle_x -= self.step_size
elif action == 3:
self.vehicle_x += self.step_size
self.vehicle_x = int(np.clip(self.vehicle_x, 0, self.width - self.vehicle_size))
self.vehicle_y = int(np.clip(self.vehicle_y, 0, self.height - self.vehicle_size))
terminated = False
truncated = False
reward = -0.1
dist_to_goal = np.sqrt(
(self.vehicle_x - self.goal_x) ** 2 + (self.vehicle_y - self.goal_y) ** 2
)
if dist_to_goal < (self.vehicle_size + self.goal_size) / 2:
reward = 100.0
terminated = True
if not terminated:
for ox, oy in self.obstacles:
dist = np.sqrt(
(self.vehicle_x - ox) ** 2 + (self.vehicle_y - oy) ** 2
)
if dist < (self.vehicle_size + self.obstacle_size) / 2:
reward = -50.0
terminated = True
break
if self.steps >= self.max_steps:
truncated = True
if not terminated and not truncated:
max_dist = np.sqrt(self.width**2 + self.height**2)
reward += (1.0 - dist_to_goal / max_dist) * 0.1
observation = self._get_observation()
return observation, reward, terminated, truncated, {}How does the reward structure work?
The reward structure teaches the agent what "good" navigation looks like through four signals:
- +100 for reaching the goal. This is the primary objective.
- -50 for hitting an obstacle. This teaches avoidance behavior.
- -0.1 per step. This small penalty encourages efficiency. Without it, the agent would wander forever.
- A small proximity bonus when the agent gets closer to the goal. This guides exploration in the right direction during early training.
What is reward shaping?
The proximity bonus is an example of reward shaping. Raw sparse rewards (+100 or -50) only fire when the episode ends, which makes early learning nearly impossible. The proximity bonus gives the agent a continuous signal to follow, like breadcrumbs leading toward the goal.
- Save environment.py.
- Now add the _get_observation() helper method directly below step():
def _get_observation(self):
obs = np.array(
[
self.vehicle_x / self.width,
self.vehicle_y / self.height,
self.goal_x / self.width,
self.goal_y / self.height,
]
+ [
coord / dim
for ox, oy in self.obstacles
for coord, dim in [(ox, self.width), (oy, self.height)]
],
dtype=np.float32,
)
return obsWhat does this code do?
This method builds the 14-dimensional observation vector that the agent "sees" at each step.
- Indices 0-1 hold the vehicle's normalized x and y position.
- Indices 2-3 hold the goal's normalized position.
- Indices 4-13 hold the normalized positions of all 5 obstacles (2 values each).
- Every value is divided by the arena width or height, producing floats between 0 and 1. This normalization helps the neural network learn faster.
- Save environment.py.
Seeing an indentation error?
All three methods (reset, step, _get_observation) must be indented one level inside the VehicleNavigationEnv class. Make sure each def line starts with exactly 4 spaces.
Help me fix indentation errors in my environment.py file.
Add Pygame rendering and test the environment
The render() method draws the arena using Pygame so you can watch your trained agent navigate in real-time later. This is what makes your project visually impressive for a portfolio.
- Add the render() and close() methods below _get_observation():
def render(self):
if self.render_mode is None:
return
if self.window is None:
pygame.init()
if self.render_mode == "human":
self.window = pygame.display.set_mode((self.width, self.height))
pygame.display.set_caption("Autonomous Vehicle Navigation")
self.clock = pygame.time.Clock()
surface = pygame.Surface((self.width, self.height))
surface.fill((30, 30, 30))
for ox, oy in self.obstacles:
pygame.draw.rect(
surface,
(220, 50, 50),
(
ox - self.obstacle_size // 2,
oy - self.obstacle_size // 2,
self.obstacle_size,
self.obstacle_size,
),
)
pygame.draw.rect(
surface,
(255, 215, 0),
(
self.goal_x - self.goal_size // 2,
self.goal_y - self.goal_size // 2,
self.goal_size,
self.goal_size,
),
)
pygame.draw.rect(
surface,
(50, 220, 50),
(
self.vehicle_x - self.vehicle_size // 2,
self.vehicle_y - self.vehicle_size // 2,
self.vehicle_size,
self.vehicle_size,
),
)
if self.render_mode == "human":
self.window.blit(surface, (0, 0))
pygame.display.flip()
self.clock.tick(self.metadata["render_fps"])
def close(self):
if self.window is not None:
pygame.quit()
self.window = NoneWhat does this code do?
The render() method draws the arena at 30 FPS when render_mode="human" is set.
- Red squares are obstacles the agent must avoid.
- The gold square is the goal the agent tries to reach.
- The green square is the vehicle (the agent).
- The close() method shuts down Pygame cleanly when the environment is done.
- Save environment.py.
- Now add a test block at the very bottom of the file (outside the class, with no indentation) to verify everything works:
if __name__ == "__main__":
env = VehicleNavigationEnv()
obs, info = env.reset()
print(f"Observation shape: {obs.shape}")
print(f"Sample observation: {obs}")
next_obs, reward, terminated, truncated, info = env.step(3)
print(f"After step - reward: {reward:.3f}, terminated: {terminated}")
env.close()What does this code do?
This test block only runs when you execute environment.py directly. It creates the environment, resets it, takes one action (move right), and prints the observation shape and reward. This confirms your environment class is wired up correctly.
- Save environment.py.
- Run the environment test by running this command in your terminal:
python environment.py- You should see output like this (exact numbers will vary due to randomization):
Observation shape: (14,)
Sample observation: [0.14 0.55 0.87 0.42 ...]
After step - reward: -0.089, terminated: FalseThe key confirmation is that the observation shape is (14,). The exact numbers and reward value will differ each time because positions are randomized.
Seeing a ModuleNotFoundError?
Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt. If not, activate it again with venv\Scripts\activate.
Help me debug a ModuleNotFoundError when running my environment.py file.
✔️ Awesome, I've got everything!
Great. Double check you have saved environment.py and that your terminal output shows Observation shape: (14,).
ⓧ I'd like to double check the full code
Here is the complete environment.py file. Compare yours against it:
import gymnasium as gym
from gymnasium import spaces
import numpy as np
import pygame
class VehicleNavigationEnv(gym.Env):
metadata = {"render_modes": ["human", "rgb_array"], "render_fps": 30}
def __init__(self, render_mode=None):
super().__init__()
self.render_mode = render_mode
self.width = 800
self.height = 600
self.vehicle_size = 20
self.obstacle_size = 30
self.goal_size = 20
self.step_size = 20
self.n_obstacles = 5
self.max_steps = 200
self.action_space = spaces.Discrete(4)
self.observation_space = spaces.Box(
low=0.0, high=1.0, shape=(14,), dtype=np.float32
)
self.window = None
self.clock = None
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.steps = 0
self.vehicle_x = self.np_random.integers(50, self.width // 3)
self.vehicle_y = self.np_random.integers(50, self.height - 50)
self.goal_x = self.np_random.integers(2 * self.width // 3, self.width - 50)
self.goal_y = self.np_random.integers(50, self.height - 50)
self.obstacles = []
for _ in range(self.n_obstacles):
ox = self.np_random.integers(self.width // 4, 3 * self.width // 4)
oy = self.np_random.integers(50, self.height - 50)
self.obstacles.append((ox, oy))
observation = self._get_observation()
return observation, {}
def step(self, action):
self.steps += 1
if action == 0:
self.vehicle_y -= self.step_size
elif action == 1:
self.vehicle_y += self.step_size
elif action == 2:
self.vehicle_x -= self.step_size
elif action == 3:
self.vehicle_x += self.step_size
self.vehicle_x = int(np.clip(self.vehicle_x, 0, self.width - self.vehicle_size))
self.vehicle_y = int(np.clip(self.vehicle_y, 0, self.height - self.vehicle_size))
terminated = False
truncated = False
reward = -0.1
dist_to_goal = np.sqrt(
(self.vehicle_x - self.goal_x) ** 2 + (self.vehicle_y - self.goal_y) ** 2
)
if dist_to_goal < (self.vehicle_size + self.goal_size) / 2:
reward = 100.0
terminated = True
if not terminated:
for ox, oy in self.obstacles:
dist = np.sqrt(
(self.vehicle_x - ox) ** 2 + (self.vehicle_y - oy) ** 2
)
if dist < (self.vehicle_size + self.obstacle_size) / 2:
reward = -50.0
terminated = True
break
if self.steps >= self.max_steps:
truncated = True
if not terminated and not truncated:
max_dist = np.sqrt(self.width**2 + self.height**2)
reward += (1.0 - dist_to_goal / max_dist) * 0.1
observation = self._get_observation()
return observation, reward, terminated, truncated, {}
def _get_observation(self):
obs = np.array(
[
self.vehicle_x / self.width,
self.vehicle_y / self.height,
self.goal_x / self.width,
self.goal_y / self.height,
]
+ [
coord / dim
for ox, oy in self.obstacles
for coord, dim in [(ox, self.width), (oy, self.height)]
],
dtype=np.float32,
)
return obs
def render(self):
if self.render_mode is None:
return
if self.window is None:
pygame.init()
if self.render_mode == "human":
self.window = pygame.display.set_mode((self.width, self.height))
pygame.display.set_caption("Autonomous Vehicle Navigation")
self.clock = pygame.time.Clock()
surface = pygame.Surface((self.width, self.height))
surface.fill((30, 30, 30))
for ox, oy in self.obstacles:
pygame.draw.rect(
surface,
(220, 50, 50),
(
ox - self.obstacle_size // 2,
oy - self.obstacle_size // 2,
self.obstacle_size,
self.obstacle_size,
),
)
pygame.draw.rect(
surface,
(255, 215, 0),
(
self.goal_x - self.goal_size // 2,
self.goal_y - self.goal_size // 2,
self.goal_size,
self.goal_size,
),
)
pygame.draw.rect(
surface,
(50, 220, 50),
(
self.vehicle_x - self.vehicle_size // 2,
self.vehicle_y - self.vehicle_size // 2,
self.vehicle_size,
self.vehicle_size,
),
)
if self.render_mode == "human":
self.window.blit(surface, (0, 0))
pygame.display.flip()
self.clock.tick(self.metadata["render_fps"])
def close(self):
if self.window is not None:
pygame.quit()
self.window = None
if __name__ == "__main__":
env = VehicleNavigationEnv()
obs, info = env.reset()
print(f"Observation shape: {obs.shape}")
print(f"Sample observation: {obs}")
next_obs, reward, terminated, truncated, info = env.step(3)
print(f"After step - reward: {reward:.3f}, terminated: {terminated}")
env.close()Why build a custom environment instead of using a pre-built one?
Pre-built Gymnasium environments (like CartPole or LunarLander) are great for learning algorithms, but they do not demonstrate your ability to design the problem itself. Building the environment from scratch shows you understand the RL interface at a deeper level.
In real autonomous systems work, the environment is rarely off-the-shelf. You will always need to define what the agent observes, how it acts, and what constitutes success. This is that skill in practice.
Your environment is ready. The agent now has a world to explore, a goal to chase, and obstacles to avoid. Next up, you will build the DQN agent that learns to navigate this environment using a neural network and experience replay.
Build the DQN Agent
Your environment can simulate a vehicle navigating through obstacles. Now you need a brain for that vehicle.
A Deep Q-Network (DQN) learns which action to take in any given state by approximating a Q-value function with a neural network. In this step, you will build the full DQN agent from scratch: the neural network that predicts action values, a replay buffer that stores past experiences for efficient learning, and the agent logic that ties everything together.
In this step, get ready to:
- Build a 3-layer neural network that maps states to Q-values.
- Implement a replay buffer for storing and sampling experiences.
- Create the DQN agent with epsilon-greedy exploration and target network updates.
Create the neural network architecture
The Q-network takes a 14-dimensional state observation and outputs a Q-value for each of the 4 possible actions. The action with the highest Q-value is what the agent considers the best move.
- In VS Code, create a new file called agent.py inside your dqn-navigation folder.
- Add the imports and the DQN class by pasting this code:
import numpy as np
import random
from collections import deque
import torch
import torch.nn as nn
import torch.optim as optim
class DQN(nn.Module):
def __init__(self, state_size, action_size):
super(DQN, self).__init__()
# Three fully connected layers: 14 -> 128 -> 64 -> 4
self.fc1 = nn.Linear(state_size, 128)
self.fc2 = nn.Linear(128, 64)
self.fc3 = nn.Linear(64, action_size)
def forward(self, x):
# ReLU activations between layers, raw output for Q-values
x = torch.relu(self.fc1(x))
x = torch.relu(self.fc2(x))
return self.fc3(x)What does this code do?
The DQN class is a PyTorch neural network with three linear layers. It takes a 14-dimensional state vector and produces 4 output values, one per action.
ReLU activations between layers introduce non-linearity so the network can learn complex patterns. The final layer has no activation because Q-values can be any real number (positive or negative).
You won't see a change yet. The next chunk builds the replay buffer that feeds training data to this network.
Implement the replay buffer
Training a neural network on sequential experience is unstable because consecutive samples are highly correlated. A replay buffer stores past transitions and lets you sample random mini-batches, breaking the correlation and stabilizing learning.
- In agent.py, add the ReplayBuffer class below the DQN class:
class ReplayBuffer:
def __init__(self, capacity):
# Fixed-size buffer using a deque that auto-discards old entries
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
# Store a single transition
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
# Randomly sample a batch for training
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (
np.array(states, dtype=np.float32),
np.array(actions, dtype=np.int64),
np.array(rewards, dtype=np.float32),
np.array(next_states, dtype=np.float32),
np.array(dones, dtype=np.float32),
)
def __len__(self):
return len(self.buffer)What does this code do?
The ReplayBuffer stores up to 10,000 transitions as (state, action, reward, next_state, done) tuples. Once full, the oldest entries are automatically discarded.
The sample() method pulls a random batch and converts the data into NumPy arrays ready for tensor conversion. This is what makes experience replay possible.
You won't see a change yet. The next chunk uses this buffer inside the full agent class.
Implement the DQN agent
The agent ties everything together. It manages two networks (a policy network for decision-making and a target network for stable training), selects actions using epsilon-greedy exploration, and updates its weights by learning from replay buffer samples.
- In agent.py, add the DQNAgent class below the ReplayBuffer class:
class DQNAgent:
def __init__(self, state_size, action_size, device):
self.state_size = state_size
self.action_size = action_size
self.device = device
# Policy network makes decisions, target network provides stable targets
self.policy_net = DQN(state_size, action_size).to(device)
self.target_net = DQN(state_size, action_size).to(device)
self.target_net.load_state_dict(self.policy_net.state_dict())
self.target_net.eval()
self.optimizer = optim.Adam(self.policy_net.parameters(), lr=0.001)
self.memory = ReplayBuffer(10000)
# Hyperparameters
self.gamma = 0.99
self.epsilon = 1.0
self.epsilon_min = 0.01
self.epsilon_decay = 0.995
self.batch_size = 64
self.target_update_freq = 10
def select_action(self, state):
# Epsilon-greedy: random action or best Q-value action
if random.random() < self.epsilon:
return random.randrange(self.action_size)
with torch.no_grad():
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
q_values = self.policy_net(state_tensor)
return q_values.argmax().item()
def store_transition(self, state, action, reward, next_state, done):
self.memory.push(state, action, reward, next_state, done)
def learn(self):
# Only learn when we have enough samples
if len(self.memory) < self.batch_size:
return
states, actions, rewards, next_states, dones = self.memory.sample(
self.batch_size
)
# Convert to tensors on the GPU
states = torch.FloatTensor(states).to(self.device)
actions = torch.LongTensor(actions).to(self.device)
rewards = torch.FloatTensor(rewards).to(self.device)
next_states = torch.FloatTensor(next_states).to(self.device)
dones = torch.FloatTensor(dones).to(self.device)
# Current Q-values for the actions we actually took
current_q = self.policy_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
# Target Q-values from the stable target network
with torch.no_grad():
next_q = self.target_net(next_states).max(1)[0]
target_q = rewards + (1 - dones) * self.gamma * next_q
# Update policy network via MSE loss
loss = nn.MSELoss()(current_q, target_q)
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
def update_epsilon(self):
self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
def update_target_network(self):
self.target_net.load_state_dict(self.policy_net.state_dict())What does this code do?
- The __init__ creates two identical networks. The policy network is trained every step. The target network is a frozen copy that provides stable Q-value targets during learning.
- The select_action() method uses epsilon-greedy exploration. Early in training, epsilon is high (1.0) so the agent explores randomly. Over time, epsilon decays to 0.01 and the agent mostly exploits what it has learned.
- The learn() method samples a batch from the replay buffer, computes the difference between predicted Q-values and target Q-values, then updates the policy network to reduce that error.
- The update_target_network() method copies the policy network weights into the target network every 10 episodes, keeping the training target stable between updates.
Why two networks instead of one?
If you use the same network to both predict Q-values and generate targets, every weight update shifts the target. The network ends up chasing a moving goal and training becomes unstable.
The target network provides a fixed reference point. By updating it only every 10 episodes, the agent has a stable target to learn towards between updates.
- Add the test code at the bottom of agent.py to verify the agent works:
if __name__ == "__main__":
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
agent = DQNAgent(state_size=14, action_size=4, device=device)
dummy_state = np.random.rand(14).astype(np.float32)
action = agent.select_action(dummy_state)
print(f"Selected action: {action} (device: {device})")What does this code do?
This test block creates a DQNAgent with the same dimensions as your environment (14 inputs, 4 actions) and runs a single action selection on a random state. It confirms the network, replay buffer, and CUDA device are all wired up correctly.
- Save agent.py.
- Run the file to verify everything works by running this command:
python agent.py- You should see output like Selected action: 2 (device: cuda). The action number will vary, but device: cuda confirms your GPU is being used.
Seeing an error or device: cpu?
- If you see device: cpu instead of device: cuda, your PyTorch installation may not have CUDA support. Re-run the PyTorch install command from Step 1 with the --index-url flag.
- If you see an ImportError, make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
- If you see a SyntaxError, check for mismatched parentheses in the sample() method. The return statement has multiple levels of nesting.
Help me debug my agent.py file
✔️ Awesome, I've got everything!
Great. Double check you have saved agent.py and confirmed the output shows your CUDA device.
ⓧ I'd like to double check the full code
Here is the complete agent.py file:
import numpy as np
import random
from collections import deque
import torch
import torch.nn as nn
import torch.optim as optim
class DQN(nn.Module):
def __init__(self, state_size, action_size):
super(DQN, self).__init__()
self.fc1 = nn.Linear(state_size, 128)
self.fc2 = nn.Linear(128, 64)
self.fc3 = nn.Linear(64, action_size)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = torch.relu(self.fc2(x))
return self.fc3(x)
class ReplayBuffer:
def __init__(self, capacity):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (
np.array(states, dtype=np.float32),
np.array(actions, dtype=np.int64),
np.array(rewards, dtype=np.float32),
np.array(next_states, dtype=np.float32),
np.array(dones, dtype=np.float32),
)
def __len__(self):
return len(self.buffer)
class DQNAgent:
def __init__(self, state_size, action_size, device):
self.state_size = state_size
self.action_size = action_size
self.device = device
self.policy_net = DQN(state_size, action_size).to(device)
self.target_net = DQN(state_size, action_size).to(device)
self.target_net.load_state_dict(self.policy_net.state_dict())
self.target_net.eval()
self.optimizer = optim.Adam(self.policy_net.parameters(), lr=0.001)
self.memory = ReplayBuffer(10000)
self.gamma = 0.99
self.epsilon = 1.0
self.epsilon_min = 0.01
self.epsilon_decay = 0.995
self.batch_size = 64
self.target_update_freq = 10
def select_action(self, state):
if random.random() < self.epsilon:
return random.randrange(self.action_size)
with torch.no_grad():
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
q_values = self.policy_net(state_tensor)
return q_values.argmax().item()
def store_transition(self, state, action, reward, next_state, done):
self.memory.push(state, action, reward, next_state, done)
def learn(self):
if len(self.memory) < self.batch_size:
return
states, actions, rewards, next_states, dones = self.memory.sample(
self.batch_size
)
states = torch.FloatTensor(states).to(self.device)
actions = torch.LongTensor(actions).to(self.device)
rewards = torch.FloatTensor(rewards).to(self.device)
next_states = torch.FloatTensor(next_states).to(self.device)
dones = torch.FloatTensor(dones).to(self.device)
current_q = self.policy_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
with torch.no_grad():
next_q = self.target_net(next_states).max(1)[0]
target_q = rewards + (1 - dones) * self.gamma * next_q
loss = nn.MSELoss()(current_q, target_q)
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
def update_epsilon(self):
self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
def update_target_network(self):
self.target_net.load_state_dict(self.policy_net.state_dict())
if __name__ == "__main__":
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
agent = DQNAgent(state_size=14, action_size=4, device=device)
dummy_state = np.random.rand(14).astype(np.float32)
action = agent.select_action(dummy_state)
print(f"Selected action: {action} (device: {device})")Your DQN agent is ready to learn. Next up, you will write the training loop that runs hundreds of episodes, letting the agent explore, collect experiences, and gradually improve its navigation policy.
Train the Agent and Visualize Results
You have a custom environment and a DQN agent ready to go. The missing piece is the training loop that connects them together.
In this step, you will write the script that runs 500 training episodes on your GPU, plots the learning curves, and then lets you watch the trained agent navigate in real-time through a Pygame window.
In this step, get ready to:
- Write a training loop that runs 500 episodes and tracks scores.
- Generate a training performance chart with a moving average and epsilon decay plot.
- Evaluate the trained agent with live Pygame visualization.
Create the training loop
The training loop is where your agent actually learns. Each episode, the agent interacts with the environment, stores experiences, and updates its neural network weights based on what worked and what did not.
- Create a new file called main.py in your dqn-navigation folder.
- Add the imports and the train() function by copying this code into main.py:
import numpy as np
import matplotlib.pyplot as plt
import torch
import pygame
from environment import VehicleNavigationEnv
from agent import DQNAgent
def train(num_episodes=500):
# Detect GPU and create environment
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Training on: {device}")
env = VehicleNavigationEnv()
state_size = env.observation_space.shape[0]
action_size = env.action_space.n
agent = DQNAgent(state_size, action_size, device)
# Track training metrics
scores = []
epsilon_history = []
successes = 0
for episode in range(num_episodes):
state, _ = env.reset()
total_reward = 0
done = False
# Run one episode: act, store, learn
while not done:
action = agent.select_action(state)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
agent.store_transition(state, action, reward, next_state, terminated)
agent.learn()
state = next_state
total_reward += reward
# Decay exploration rate after each episode
agent.update_epsilon()
# Sync target network periodically
if episode % agent.target_update_freq == 0:
agent.update_target_network()
# Track successes (score above 50 means goal was likely reached)
if total_reward > 50:
successes += 1
scores.append(total_reward)
epsilon_history.append(agent.epsilon)
# Print progress every 50 episodes
if (episode + 1) % 50 == 0:
avg_score = np.mean(scores[-50:])
success_rate = successes / (episode + 1) * 100
print(
f"Episode {episode + 1}/{num_episodes} | "
f"Avg Score: {avg_score:.1f} | "
f"Epsilon: {agent.epsilon:.3f} | "
f"Success Rate: {success_rate:.1f}%"
)
env.close()
return agent, scores, epsilon_historyWhat does this code do?
The train() function orchestrates the entire learning process across 500 episodes.
- Each episode resets the environment, then loops: the agent picks an action, the environment responds with a reward, and the agent stores that experience and learns from a batch of past experiences.
- After each episode, epsilon decays (less random exploration over time) and the target network syncs every 10 episodes to keep learning stable.
- Progress is printed every 50 episodes so you can watch the agent improve.
- Save main.py.
Seeing an import error?
Make sure environment.py and agent.py are in the same folder as main.py. Python looks for imports relative to the file being run.
Help me debug my import error in main.py
Implement the results plotting function
Raw episode scores are noisy. A moving average smooths out the randomness so you can actually see whether the agent is improving over time.
- In main.py, add the plot_results() function below the train() function:
def plot_results(scores, epsilon_history):
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
# Top plot: episode scores with a smoothed moving average
ax1.plot(scores, alpha=0.3, color="blue", label="Score")
window = 50
if len(scores) >= window:
moving_avg = np.convolve(scores, np.ones(window) / window, mode="valid")
ax1.plot(
range(window - 1, len(scores)),
moving_avg,
color="red",
label=f"{window}-Episode Moving Average",
)
ax1.set_xlabel("Episode")
ax1.set_ylabel("Score")
ax1.set_title("Training Scores")
ax1.legend()
ax1.grid(True, alpha=0.3)
# Bottom plot: how exploration decays over training
ax2.plot(epsilon_history, color="green")
ax2.set_xlabel("Episode")
ax2.set_ylabel("Epsilon")
ax2.set_title("Exploration Rate Decay")
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("training_results.png", dpi=150)
plt.show()
print("Results saved to training_results.png")What does this code do?
- The top subplot shows raw scores (semi-transparent blue) overlaid with a 50-episode moving average (red line). The moving average reveals the true learning trend underneath the noise.
- The bottom subplot tracks epsilon over time. You should see it decay from 1.0 (pure random) down toward 0.01 (almost pure exploitation).
- The chart is saved as training_results.png at 150 dpi for portfolio-quality resolution.
- Save main.py.
Implement the evaluation function and run everything
Training happens without any visualization to maximize speed. The evaluation function creates a separate environment with Pygame rendering so you can watch the trained agent in action.
- In main.py, add the evaluate() function below plot_results():
def evaluate(agent, num_episodes=5):
# Create environment with Pygame rendering enabled
env = VehicleNavigationEnv(render_mode="human")
for episode in range(num_episodes):
state, _ = env.reset()
done = False
total_reward = 0
while not done:
action = agent.select_action(state)
state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
total_reward += reward
env.render()
# Allow closing the Pygame window mid-evaluation
for event in pygame.event.get():
if event.type == pygame.QUIT:
env.close()
return
print(f"Evaluation Episode {episode + 1}: Score = {total_reward:.1f}")
env.close()What does this code do?
- The evaluate() function creates the environment with render_mode="human", which opens a Pygame window showing the vehicle, obstacles, and goal.
- The agent runs with epsilon set to 0 (pure exploitation), so it uses only what it learned. No randomness.
- The Pygame event loop lets you close the window at any time without crashing the script.
- Now add the __main__ block at the bottom of main.py to tie everything together:
if __name__ == "__main__":
print("=" * 50)
print("DQN Autonomous Vehicle Navigation")
print("=" * 50)
# Train the agent for 500 episodes
agent, scores, epsilon_history = train(num_episodes=500)
# Generate and save the training performance chart
plot_results(scores, epsilon_history)
# Evaluate with live visualization (epsilon = 0, no exploration)
print("\nEvaluating trained agent...")
agent.epsilon = 0.0
evaluate(agent, num_episodes=5)What does this code do?
- The __main__ block runs the full pipeline in order: train, plot, evaluate.
- Setting agent.epsilon = 0.0 before evaluation forces the agent to use its learned policy with zero random actions.
- Save main.py.
- Run the full training and evaluation pipeline by running this command in your terminal:
python main.pyWhat should I see?
Training takes under 2 minutes on your GPU. You will see progress updates every 50 episodes in the terminal, like this:
Episode 50/500 | Avg Score: -18.2 | Epsilon: 0.779 | Success Rate: 2.0%
The average score should trend upward and success rate should climb as epsilon decreases. After training, a Matplotlib chart window pops up, then the Pygame window opens showing 5 evaluation episodes where the green vehicle navigates toward the gold goal while avoiding red obstacles.
Close the chart to continue
The Matplotlib chart window blocks the script until you close it. Close the chart window and the Pygame evaluation will start immediately after.
- Confirm that training_results.png was saved in your project folder by running:
ls- You should see training_results.png alongside main.py, agent.py, and environment.py in the output.
Training seems stuck or scores aren't improving?
- Check that the terminal prints Training on: cuda at the start. If it says cpu, training will be slower and your venv may not have the CUDA version of PyTorch installed.
- Reinforcement learning is stochastic. Some runs take longer to converge. If scores are slowly increasing (even slightly) by episode 200, the agent is learning. Let it finish all 500 episodes.
- If scores stay flat after 300 episodes, double-check that your environment.py reward structure matches the code from the previous step (especially the +100 goal reward and the proximity bonus).
Help me debug why my DQN agent isn't learning
✔️ Awesome, I've got everything!
Great. Double check you have saved main.py and confirmed training_results.png exists in your project folder.
ⓧ I'd like to double check the full code
import numpy as np
import matplotlib.pyplot as plt
import torch
import pygame
from environment import VehicleNavigationEnv
from agent import DQNAgent
def train(num_episodes=500):
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Training on: {device}")
env = VehicleNavigationEnv()
state_size = env.observation_space.shape[0]
action_size = env.action_space.n
agent = DQNAgent(state_size, action_size, device)
scores = []
epsilon_history = []
successes = 0
for episode in range(num_episodes):
state, _ = env.reset()
total_reward = 0
done = False
while not done:
action = agent.select_action(state)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
agent.store_transition(state, action, reward, next_state, terminated)
agent.learn()
state = next_state
total_reward += reward
agent.update_epsilon()
if episode % agent.target_update_freq == 0:
agent.update_target_network()
if total_reward > 50:
successes += 1
scores.append(total_reward)
epsilon_history.append(agent.epsilon)
if (episode + 1) % 50 == 0:
avg_score = np.mean(scores[-50:])
success_rate = successes / (episode + 1) * 100
print(
f"Episode {episode + 1}/{num_episodes} | "
f"Avg Score: {avg_score:.1f} | "
f"Epsilon: {agent.epsilon:.3f} | "
f"Success Rate: {success_rate:.1f}%"
)
env.close()
return agent, scores, epsilon_history
def plot_results(scores, epsilon_history):
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
ax1.plot(scores, alpha=0.3, color="blue", label="Score")
window = 50
if len(scores) >= window:
moving_avg = np.convolve(scores, np.ones(window) / window, mode="valid")
ax1.plot(
range(window - 1, len(scores)),
moving_avg,
color="red",
label=f"{window}-Episode Moving Average",
)
ax1.set_xlabel("Episode")
ax1.set_ylabel("Score")
ax1.set_title("Training Scores")
ax1.legend()
ax1.grid(True, alpha=0.3)
ax2.plot(epsilon_history, color="green")
ax2.set_xlabel("Episode")
ax2.set_ylabel("Epsilon")
ax2.set_title("Exploration Rate Decay")
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("training_results.png", dpi=150)
plt.show()
print("Results saved to training_results.png")
def evaluate(agent, num_episodes=5):
env = VehicleNavigationEnv(render_mode="human")
for episode in range(num_episodes):
state, _ = env.reset()
done = False
total_reward = 0
while not done:
action = agent.select_action(state)
state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
total_reward += reward
env.render()
for event in pygame.event.get():
if event.type == pygame.QUIT:
env.close()
return
print(f"Evaluation Episode {episode + 1}: Score = {total_reward:.1f}")
env.close()
if __name__ == "__main__":
print("=" * 50)
print("DQN Autonomous Vehicle Navigation")
print("=" * 50)
agent, scores, epsilon_history = train(num_episodes=500)
plot_results(scores, epsilon_history)
print("\nEvaluating trained agent...")
agent.epsilon = 0.0
evaluate(agent, num_episodes=5)You have a trained DQN agent that learned to navigate autonomously, complete with performance charts and live visualization. Next up, a Secret Mission that will make your agent even smarter.
Secret mission
Implement Double DQN
Your standard DQN agent sometimes overestimates Q-values, leading to unstable training in later episodes. Double DQN decouples action selection from value estimation to produce more reliable learning. Can you modify a single method to fix this fundamental bias?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project files for future experimentation, or delete them to free up disk space.
Resources you used:
- The dqn-navigation project folder (includes source files, virtual environment, and generated images).
Keep everything running
No action needed. Choose this if you plan to keep experimenting with hyperparameters, reward shaping, or extending the agent further.
- Your project folder stays on your Desktop with all files intact.
- The virtual environment stays ready to reactivate whenever you return.
Pause - I'll come back to this later
Deactivate the virtual environment to free up your terminal session. Your files remain on disk for later.
- Deactivate the virtual environment by running:
deactivate- When you come back, reactivate it with venv\Scripts\activate from inside the dqn-navigation folder.
Delete - I don't want to use this again
Remove the entire project folder to reclaim disk space. This deletes all source code, the virtual environment, and generated images.
- Deactivate the virtual environment first by running:
deactivate- Navigate to the Desktop and delete the project folder.
macOS
cd ~/Desktop
rm -rf dqn-navigationWindows
cd ~\Desktop
rmdir /s /q dqn-navigationNice Work!
Nice Work!
Nice work! You just built a complete reinforcement learning system from scratch, training a DQN agent to autonomously navigate a vehicle through obstacles using your GPU.
You've learned how to:
- Build a custom Gymnasium environment from scratch, modeling autonomous vehicle navigation with obstacles, a goal, and a shaped reward signal.
- Implement a complete DQN agent in PyTorch with a neural network, replay buffer, epsilon-greedy exploration, and target network.
- Train the agent using GPU acceleration over 500 episodes and produce portfolio-quality training visualizations showing the learning process.
- Secret Mission: Implemented Double DQN to improve training stability by decoupling action selection from value estimation.
Ready to quiz yourself?