Train a Self-Driving AI with RL

Train a neural network to drive a car from raw pixels using PPO.

Introduction

30 Second Summary

Self-driving cars learn to drive the same way you did. They practice over and over, getting better each time. Instead of a driving instructor, they have a reward signal that tells them when they're doing well and when they're about to crash.

In this project, you will train a neural network to drive a car around a race track using deep reinforcement learning. Your AI agent starts knowing nothing, and through thousands of practice laps, it learns to steer, accelerate, and brake from raw pixel observations.

What You'll Build

A complete reinforcement learning pipeline that trains a convolutional neural network to drive a car from 96x96 pixel screenshots of a top-down racing environment.

By the end of this project, you'll have:

  • A Python training script that configures a PPO agent and trains it to race from raw pixels.
  • A trained model file you can reload and evaluate at any time.
  • Recorded .mp4 videos comparing a random agent's chaotic driving against your trained agent's smooth laps.
  • Secret Mission: Switch to SAC (Soft Actor-Critic) and compare its learning curve against PPO.

Are there any prerequisites?

You need basic Python knowledge (variables, functions, loops). No machine learning or reinforcement learning experience is required.

Before We Start

Training an AI to drive requires a specific set of tools. You need a physics simulator for the car, a neural network library for the brain, and a reinforcement learning framework to connect them.

In this step, you will install everything on your Fedora machine. By the end, you will have a working environment that can launch a virtual race track and render frames for your future AI agent.

In this step, get ready to:
  • Install system dependencies and create a Python virtual environment.
  • Install PyTorch, Gymnasium, and Stable-Baselines3.
  • Verify the environment by launching CarRacing-v2.
Install system dependencies and create a virtual environment

The racing environment uses Box2D physics under the hood. Building the Box2D Python bindings requires a system tool called SWIG that generates the glue code between C++ and Python.

  • Press the Windows key to open your search bar, type Terminal, and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir self-driving-rl
cd self-driving-rl
  • Confirm the folder exists by running:
ls ~/Desktop

You should see self-driving-rl in the output.

  • Install SWIG on your system by running:
sudo dnf install swig

What does this command do?

SWIG (Simplified Wrapper and Interface Generator) creates Python bindings for C++ libraries. The Box2D physics engine that powers the racing environment needs these bindings to work from Python.

Without SWIG installed at the system level, the gymnasium[box2d] package will fail to build later.

  • When prompted, type y and press Enter to confirm the installation.
  • Create a Python virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms you are working inside the isolated environment.

Don't see (venv) in your prompt?

Make sure you typed source at the beginning of the command. On Fedora, source venv/bin/activate is the correct activation command.

If python3 -m venv failed earlier, confirm Python 3 is installed by running python3 --version.

Help me troubleshoot my virtual environment activation on Fedora.

Install PyTorch, Gymnasium, and Stable-Baselines3

With your virtual environment active, you can now install the three core Python packages. You will use the CPU-only version of PyTorch since this project does not require a GPU.

  • Install PyTorch CPU by running:
pip install torch --index-url https://download.pytorch.org/whl/cpu

Why CPU-only PyTorch?

PyTorch ships separate builds for CPU and GPU (CUDA). The CPU build is much smaller and faster to download.

For this project, 100,000 training timesteps completes in about 10-15 minutes on CPU. The --index-url flag tells pip to pull wheels from PyTorch's CPU-only repository instead of the default.

  • Install Gymnasium with Box2D support by running:
pip install gymnasium[box2d]
  • Install Stable-Baselines3 by running:
pip install stable-baselines3

Each install may take a minute or two. You should see Successfully installed at the end of each command's output.

Seeing a build error during gymnasium install?

If you see an error mentioning box2d-py or swig, the system SWIG package was not installed correctly. Go back and run sudo dnf install swig again, then retry the pip install gymnasium[box2d] command.

Help me fix a Box2D build failure on Fedora.

Verify the installation

The quickest way to confirm everything works is to write a short script that creates the CarRacing-v2 environment and prints its observation shape.

  • Create a file called verify_install.py in your self-driving-rl folder by running:
touch verify_install.py
  • Press the Windows key to open your search bar, type VS Code, and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to Desktop, select the self-driving-rl folder, then click Open.
  • Click on verify_install.py in the Explorer sidebar to open it.
  • Copy the following code into verify_install.py:
import gymnasium as gym

# Create the CarRacing-v2 environment in offscreen mode
env = gym.make("CarRacing-v2", render_mode="rgb_array")
# Reset the environment and capture the first observation
obs, info = env.reset()
# Print the shape and spaces to confirm everything works
print(f"Observation shape: {obs.shape}")
print(f"Action space: {env.action_space}")
print(f"Observation space: {env.observation_space}")
env.close()
print("Installation verified successfully!")

What does this code do?

This script creates the CarRacing-v2 environment in rgb_array mode (offscreen rendering, no window needed). It resets the environment to generate the starting frame, then prints the observation shape and action space.

You should see an observation shape of (96, 96, 3). That means each frame is a 96x96 pixel RGB image. This is exactly what your neural network will learn from.

  • Save the file with Ctrl+S.
  • Run the verification script in your terminal by running:
python verify_install.py

You should see output like this:

Observation shape: (96, 96, 3)
Action space: Box(-1.0, 1.0, (3,), float32)
Observation space: Box(0, 255, (96, 96, 3), uint8)
Installation verified successfully!

Seeing a ModuleNotFoundError?

Make sure your virtual environment is active. You should see (venv) in your terminal prompt. If not, run source venv/bin/activate again from inside the self-driving-rl folder.

Also check that you ran python (not python3) since the virtual environment's python command automatically points to the correct version.

Help me fix a ModuleNotFoundError when running my verify script.

✔️ Awesome, I've got everything!

Great. Double check you have saved verify_install.py and that the output shows (96, 96, 3) for the observation shape.

ⓧ I'd like to double check the full code

Here is the complete verify_install.py file:

import gymnasium as gym

env = gym.make("CarRacing-v2", render_mode="rgb_array")
obs, info = env.reset()
print(f"Observation shape: {obs.shape}")
print(f"Action space: {env.action_space}")
print(f"Observation space: {env.observation_space}")
env.close()
print("Installation verified successfully!")

Your environment is ready. Next up, you will run a random agent to see what untrained driving looks like and explore the reinforcement learning loop.

Understand the RL Loop with a Random Agent

Before you train an AI to drive, you need to understand what "bad driving" looks like. A random agent picks steering, gas, and brake values completely at random on every frame.

In this step, you will run a random agent to establish a baseline score. This gives you a concrete comparison for later when your trained agent learns to stay on the track.

In this step, get ready to:
  • Write a random agent script that drives with random actions.
  • Record a video of the random agent for comparison later.
  • Understand the environment's state, action, and reward structure.
Write the random agent script

Every reinforcement learning system follows the same loop: observe the world, take an action, receive a reward, repeat. Your random agent will execute this loop with no intelligence at all.

  • Create a new file called random_agent.py in your project folder by running:
touch random_agent.py
  • Confirm the file exists by running:
ls
  • You should see random_agent.py alongside verify_install.py in the output.
  • Open random_agent.py in VS Code.
  • Add the environment setup code at the top of the file:
import gymnasium as gym
from gymnasium.wrappers import RecordVideo

# Create the racing environment with RGB image rendering
env = gym.make("CarRacing-v2", render_mode="rgb_array")
# Wrap the environment to automatically save video of the first episode
env = RecordVideo(env, video_folder="videos/random", name_prefix="random-agent", episode_trigger=lambda x: x == 0)

# Reset the environment to get the initial observation
obs, info = env.reset()
total_reward = 0
steps = 0

What does this code do?

gym.make("CarRacing-v2", render_mode="rgb_array") creates a top-down racing game where the agent sees a 96x96 pixel image every frame.

RecordVideo wraps the environment so every frame is captured and saved as a .mp4 file automatically. The episode_trigger=lambda x: x == 0 means it only records the very first episode.

env.reset() places the car at the starting line and returns the first 96x96 image (the "state").

  • Add the RL loop below the setup code:
# Run 1000 steps with completely random actions
for step in range(1000):
    # Sample a random action: [steering, gas, brake]
    action = env.action_space.sample()
    # Take the action and receive new state + reward
    obs, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    steps += 1
    # Stop if the episode ends naturally
    if terminated or truncated:
        break

What is the RL loop?

This is the core reinforcement learning loop in action. Each iteration follows the same pattern: observe state, choose action, receive reward, check if done.

  • env.action_space.sample() picks random values for steering (-1 to +1), gas (0 to 1), and brake (0 to 1).
  • env.step(action) advances the simulation by one frame and returns five values: the new image, the reward, whether the car crashed (terminated), whether time ran out (truncated), and extra info.
  • total_reward accumulates the score. A trained agent will maximize this value.
  • Add the results summary at the bottom of the file:
env.close()

# Print a summary of what happened
print(f"Random Agent Results:")
print(f"  Steps taken: {steps}")
print(f"  Total reward: {total_reward:.1f}")
print(f"  Observation shape: {obs.shape}")
print(f"  Action space: steering [-1, 1], gas [0, 1], brake [0, 1]")
print(f"  Reward per frame: -0.1 (penalty) + tile bonus for visiting track tiles")
print(f"Video saved to videos/random/")

What does the reward structure mean?

The environment penalizes the agent -0.1 for every frame it is alive. This means doing nothing costs points.

The agent earns bonus points (+1000/N where N is total track tiles) for each new track tile it drives over. A good agent must visit tiles faster than the per-frame penalty drains its score.

  • Save random_agent.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before running it.

ⓧ I'd like to double check the full code

import gymnasium as gym
from gymnasium.wrappers import RecordVideo

env = gym.make("CarRacing-v2", render_mode="rgb_array")
env = RecordVideo(env, video_folder="videos/random", name_prefix="random-agent", episode_trigger=lambda x: x == 0)

obs, info = env.reset()
total_reward = 0
steps = 0

for step in range(1000):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    steps += 1
    if terminated or truncated:
        break

env.close()

print(f"Random Agent Results:")
print(f"  Steps taken: {steps}")
print(f"  Total reward: {total_reward:.1f}")
print(f"  Observation shape: {obs.shape}")
print(f"  Action space: steering [-1, 1], gas [0, 1], brake [0, 1]")
print(f"  Reward per frame: -0.1 (penalty) + tile bonus for visiting track tiles")
print(f"Video saved to videos/random/")
Run the script and watch the random agent

Now you will run the script and see how badly a random agent drives. This establishes your baseline for comparison once training is complete.

  • Run the random agent script by executing:
python random_agent.py
  • Wait for the script to finish. It takes a few seconds to run 1000 steps and encode the video.

What should I see?

Your terminal prints a summary showing steps taken (often less than 1000 if the episode ended early), a total reward somewhere between -50 and +50, and the observation shape (96, 96, 3).

The script also creates a videos/random/ directory with a .mp4 file inside it.

  • Confirm the video file was created by running:
ls videos/random/
  • You should see a file like random-agent-episode-0.mp4 in the output.
  • Open the video file with your system video player to watch the random agent. You will see the car swerving wildly, driving off the track, and generally performing terribly.

Seeing a moviepy error?

The RecordVideo wrapper uses moviepy to encode video. If you see an error mentioning moviepy is not installed, install it into your virtual environment:

pip install moviepy

Then run python random_agent.py again.

Help me debug my RecordVideo error

Understand the environment details

The printed output tells you everything an RL agent needs to know about its problem. Each field maps directly to a core RL concept.

What do these values mean for RL?

  • Observation shape (96, 96, 3) is the "state". The agent sees a 96x96 RGB image of the track from above. This is what the neural network will learn to interpret.
  • Action space: steering [-1, 1], gas [0, 1], brake [0, 1] are the three continuous controls the agent can output. Every frame, it picks values for all three simultaneously.
  • Reward per frame: -0.1 penalty + tile bonus defines the optimization target. The agent must learn to visit track tiles quickly to earn points faster than the per-frame penalty takes them away.
  • Total reward (~-50 to +50 for random) is the baseline. A trained agent should score 200+ by staying on the track. The gap between random and trained is what your neural network learns to bridge.

Why does a single frame not tell the agent enough?

A single 96x96 image shows where the car is, but not how fast it is going or which direction it is moving. In the next step, you will use frame stacking (4 consecutive frames) to give the network temporal information. This is the same technique DeepMind used for Atari games.

Your random agent just scored around 0 by driving aimlessly. Next up, you will train a PPO agent that learns from these same pixels and steadily improves its score to 200+.

Train Your PPO Agent

You saw how the random agent crashes and spins with a terrible score. The problem is clear: random actions produce random results.

Now you will train a PPO (Proximal Policy Optimization) agent that learns from its mistakes. PPO feeds 96x96 pixel frames through a convolutional neural network and gradually improves its steering, acceleration, and braking over 100,000 timesteps.

In this step, get ready to:
  • Write a training script that configures PPO with frame stacking and a CNN policy.
  • Launch training for 100,000 timesteps and monitor progress in the terminal.
  • Plot the reward curve to visualize how the agent improves over time.
Write your training script

The training script needs to do three things: prepare the environment with frame stacking, configure the PPO algorithm with good hyperparameters, and run the training loop.

  • Create a new file called train_ppo.py in your project folder.
  • Add the environment setup code by pasting the following into train_ppo.py:
import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecFrameStack
from stable_baselines3.common.env_util import make_vec_env

def make_env():
    env = gym.make("CarRacing-v2", continuous=True)
    return env

vec_env = make_vec_env(make_env, n_envs=1)
vec_env = VecFrameStack(vec_env, n_stack=4)

What does this code do?

The make_env function creates a single CarRacing-v2 environment in continuous action mode (steering, gas, brake as floats).

The make_vec_env wrapper makes it compatible with Stable-Baselines3's vectorized training loop.

The VecFrameStack stacks the last 4 frames together. A single frame is just a static image. Stacking 4 frames gives the neural network temporal information so it can infer speed and direction.

  • Below the environment setup, add the PPO model configuration:
model = PPO(
    "CnnPolicy",
    vec_env,
    learning_rate=1e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    clip_range=0.2,
    verbose=1,
    tensorboard_log="./tb_logs/",
    device="cpu",
)

What do these hyperparameters mean?

  • CnnPolicy tells SB3 to use a convolutional neural network for image-based observations.
  • learning_rate=1e-4 controls how big each optimization step is. Smaller means more stable but slower learning.
  • n_steps=2048 is how many frames the agent collects before each policy update.
  • gamma=0.99 is the discount factor. It controls how much the agent cares about future rewards versus immediate ones.
  • clip_range=0.2 is PPO's core trick. It limits how much the policy can change in one update, preventing catastrophic forgetting.
  • device="cpu" forces training to run on CPU since this project does not require a GPU.
  • Finally, add the training execution code at the bottom of the file:
print("Starting PPO training for 100,000 timesteps...")
print("This will take approximately 10-15 minutes on CPU.")
model.learn(total_timesteps=100_000, tb_log_name="ppo_car_racing")
model.save("ppo_car_racing")
print("Training complete! Model saved to ppo_car_racing.zip")

vec_env.close()

What does this code do?

The model.learn() call runs the full training loop for 100,000 timesteps. The agent collects experience, computes advantages, and updates its neural network weights repeatedly.

The tb_log_name="ppo_car_racing" parameter writes training metrics to TensorBoard log files. You will use these logs to plot reward curves later.

The model.save() call writes the trained neural network weights to ppo_car_racing.zip so you can reload and evaluate it without retraining.

  • Save train_ppo.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecFrameStack
from stable_baselines3.common.env_util import make_vec_env

def make_env():
    env = gym.make("CarRacing-v2", continuous=True)
    return env

vec_env = make_vec_env(make_env, n_envs=1)
vec_env = VecFrameStack(vec_env, n_stack=4)

model = PPO(
    "CnnPolicy",
    vec_env,
    learning_rate=1e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    clip_range=0.2,
    verbose=1,
    tensorboard_log="./tb_logs/",
    device="cpu",
)

print("Starting PPO training for 100,000 timesteps...")
print("This will take approximately 10-15 minutes on CPU.")
model.learn(total_timesteps=100_000, tb_log_name="ppo_car_racing")
model.save("ppo_car_racing")
print("Training complete! Model saved to ppo_car_racing.zip")

vec_env.close()
Launch training

Training 100,000 timesteps on CPU takes approximately 10-15 minutes. The agent will crash, spin, and fail repeatedly at first. Over time, it gradually learns to stay on the track.

  • Start training by running this command:
python train_ppo.py

What should I see during training?

Every 2048 steps, the terminal prints a table of training metrics. Look for the rollout/ep_rew_mean value. This is the mean episode reward. It starts negative and should climb toward positive values as training progresses.

You will also see time/total_timesteps counting up toward 100,000. The time/fps shows how many frames per second the CPU is processing.

  • Wait for training to complete. You should see the message Training complete! Model saved to ppo_car_racing.zip when it finishes.
  • Confirm the model file exists by running:
ls ppo_car_racing.zip
  • You should see ppo_car_racing.zip in the output.

Training seems stuck or crashed?

  • If you see a memory error, close other applications to free up RAM. PPO with CnnPolicy and frame stacking uses significant memory.
  • If the terminal hangs with no output, wait a minute. The first few iterations take longer as the CNN warms up.
  • If you see an import error, make sure your virtual environment is activated before running the script.

Help me debug my PPO training script that's failing to run.

Plot the reward curve

The training logs contain the agent's reward at each checkpoint. Plotting these rewards shows whether the agent actually improved over time.

  • Install matplotlib for plotting by running:
pip install matplotlib
  • Create a new file called plot_rewards.py and paste the following code:
import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3.common.results_plotter import load_results, ts2xy

try:
    from tensorboard.backend.event_processing.event_accumulator import EventAccumulator

    ea = EventAccumulator("./tb_logs/ppo_car_racing_1")
    ea.Reload()

    reward_events = ea.Scalars("rollout/ep_rew_mean")
    steps = [e.step for e in reward_events]
    rewards = [e.value for e in reward_events]

    plt.figure(figsize=(10, 6))
    plt.plot(steps, rewards, label="PPO", color="blue")
    plt.xlabel("Timesteps")
    plt.ylabel("Mean Episode Reward")
    plt.title("PPO Training Progress - CarRacing-v2")
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.savefig("reward_curve.png", dpi=150, bbox_inches="tight")
    plt.close()
    print("Reward curve saved to reward_curve.png")

except Exception as e:
    print(f"Could not plot from tensorboard logs: {e}")
    print("You can view live training curves by running: tensorboard --logdir=./tb_logs/")

What does this code do?

The script reads the TensorBoard event files that PPO wrote during training. It extracts the rollout/ep_rew_mean metric (mean episode reward over time) and plots it as a line graph.

If the TensorBoard files cannot be found, the script prints a fallback command so you can view live curves in a browser instead.

  • Save plot_rewards.py and generate the plot by running:
python plot_rewards.py
  • You should see Reward curve saved to reward_curve.png printed in the terminal.
  • Open reward_curve.png in your file manager. The graph should show rewards starting negative (the agent crashes early) and trending upward toward 200+ as training progresses.

Plot script says it could not find logs?

  • Check that the tb_logs/ppo_car_racing_1 directory exists by running ls tb_logs/. If the directory name has a different suffix (like _2), update the path in plot_rewards.py to match.
  • Make sure training completed fully before running the plot script. The log files are written during training.

Help me fix my reward plotting script that cannot find the TensorBoard logs.

Your agent went from random flailing to purposeful driving. Next up, you will load this trained model, watch it race in real time, and compare its performance against the random agent with hard numbers.

Evaluate and Record Your Trained Agent

Your PPO agent has finished training and the reward curve shows it learning over time. Now it is time to see if the agent can actually drive.

In this step, you will load the trained model, run it through 5 evaluation episodes with deterministic actions, and record a video of it racing. Then you will compare the result against the random agent from earlier to measure how much your agent improved.

In this step, get ready to:
  • Write an evaluation script that loads the saved model and runs 5 episodes.
  • Record a video of the trained agent racing and print mean reward.
  • Compare trained vs random agent performance numerically.
Write the evaluation script

During training, the agent explored different actions stochastically to learn what works. For evaluation, you want the agent to always pick the best action it knows, which is called deterministic inference.

  • Create a new file called evaluate_agent.py in your project folder.
  • Add the imports, model loading, and environment setup by pasting this code:
import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecFrameStack
from gymnasium.wrappers import RecordVideo

# Load the trained model from disk
model = PPO.load("ppo_car_racing")

# Create an evaluation environment with video recording
def make_eval_env():
    env = gym.make("CarRacing-v2", render_mode="rgb_array", continuous=True)
    env = RecordVideo(env, video_folder="videos/trained", name_prefix="trained-agent", episode_trigger=lambda x: x == 0)
    return env

# Wrap in vectorized env with frame stacking (must match training setup)
eval_env = DummyVecEnv([make_eval_env])
eval_env = VecFrameStack(eval_env, n_stack=4)

What does this code do?

PPO.load("ppo_car_racing") loads the trained neural network weights from the ppo_car_racing.zip file you saved in the previous step.

The make_eval_env() function creates a fresh CarRacing environment wrapped with RecordVideo to save an .mp4 of the first episode. The VecFrameStack with n_stack=4 must match training, because the model expects 4 stacked frames as input.

  • Add the evaluation loop and summary statistics below the setup code:
# Run 5 evaluation episodes with deterministic actions
episode_rewards = []
for episode in range(5):
    obs = eval_env.reset()
    total_reward = 0
    done = False
    while not done:
        # deterministic=True picks the best action without exploration
        action, _ = model.predict(obs, deterministic=True)
        obs, reward, dones, infos = eval_env.step(action)
        total_reward += reward[0]
        done = dones[0]
    episode_rewards.append(total_reward)
    print(f"Episode {episode + 1}: Reward = {total_reward:.1f}")

eval_env.close()

# Print summary statistics
mean_reward = np.mean(episode_rewards)
std_reward = np.std(episode_rewards)
print(f"\nMean reward over 5 episodes: {mean_reward:.1f} +/- {std_reward:.1f}")
print(f"Video saved to videos/trained/")

What does this code do?

The loop runs 5 complete episodes. In each episode, model.predict(obs, deterministic=True) asks the trained neural network for the best action given the current 4-frame stack.

Setting deterministic=True tells the policy to always pick the highest-probability action instead of sampling randomly. This gives you the agent's true performance without exploration noise.

After all 5 episodes, the script prints the mean and standard deviation of rewards so you can see how consistently the agent performs.

  • Save evaluate_agent.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before running it.

ⓧ I'd like to double check the full code

import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv, VecFrameStack
from gymnasium.wrappers import RecordVideo

model = PPO.load("ppo_car_racing")

def make_eval_env():
    env = gym.make("CarRacing-v2", render_mode="rgb_array", continuous=True)
    env = RecordVideo(env, video_folder="videos/trained", name_prefix="trained-agent", episode_trigger=lambda x: x == 0)
    return env

eval_env = DummyVecEnv([make_eval_env])
eval_env = VecFrameStack(eval_env, n_stack=4)

episode_rewards = []
for episode in range(5):
    obs = eval_env.reset()
    total_reward = 0
    done = False
    while not done:
        action, _ = model.predict(obs, deterministic=True)
        obs, reward, dones, infos = eval_env.step(action)
        total_reward += reward[0]
        done = dones[0]
    episode_rewards.append(total_reward)
    print(f"Episode {episode + 1}: Reward = {total_reward:.1f}")

eval_env.close()

mean_reward = np.mean(episode_rewards)
std_reward = np.std(episode_rewards)
print(f"\nMean reward over 5 episodes: {mean_reward:.1f} +/- {std_reward:.1f}")
print(f"Video saved to videos/trained/")
Run the evaluation and record video

With the script saved, you can now let your trained agent drive and record its performance on video.

  • Run the evaluation script by executing this command:
python evaluate_agent.py

The script takes about 1-2 minutes to complete 5 episodes. You should see per-episode reward printouts followed by a mean reward summary.

What should I see?

Each episode prints a reward score. A well-trained agent typically achieves rewards above 200 per episode. Your final output will look something like:

Episode 1: Reward = 312.5
Episode 2: Reward = 285.3
...
Mean reward over 5 episodes: 295.4 +/- 21.7
Video saved to videos/trained/

The exact numbers depend on your specific training run, but the mean should be above 200.

  • Confirm the video was saved by running:
ls videos/trained/

You should see a .mp4 file with the prefix trained-agent in the output.

Seeing a low reward or error?

If your mean reward is below 100, your model may not have trained long enough. You can retrain with more timesteps by editing train_ppo.py to use total_timesteps=200_000 and running it again.

If you see a FileNotFoundError, make sure ppo_car_racing.zip is in the same directory where you are running the script.

help me debug my evaluation script

Compare trained vs random agent performance

You now have video evidence of both your random agent and your trained agent. Comparing the two shows exactly how much the PPO algorithm improved driving ability through training.

  • Open the video file in videos/trained/ and watch the trained agent drive.
  • Open the video file in videos/random/ and watch the random agent for comparison.
  • Note the numerical improvement: your trained agent's mean reward (above 200) vs the random agent's typical score (around -50 to +50).

Why is this comparison important?

The random agent picks actions with no understanding of the track. It swerves off-road, brakes at the wrong time, and barely moves forward.

Your trained agent learned a policy that maps pixel observations to good steering, gas, and brake actions. The reward difference (often 200+ points) quantifies how much the neural network learned about driving from 100,000 timesteps of practice.

Play both videos side by side

Open both .mp4 files at the same time to see the contrast clearly. The random agent crashes almost immediately, while the trained agent hugs the track and completes portions of each lap.

Secret mission

Battle of the Algorithms - PPO vs SAC

You trained a PPO agent that learned to race. But is PPO the best algorithm for the job? In this secret mission, you will train a second agent using SAC (Soft Actor-Critic) and pit the two algorithms against each other on the same track.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your Fedora machine, so there are no ongoing cloud costs or subscriptions to worry about.

Decide whether to keep your project files for further experimentation, or delete them to free up disk space.

Resources you used:

  • Project folder containing Python scripts, trained model files (ppo_car_racing.zip, sac_car_racing.zip), TensorBoard logs, recorded videos, and reward plots.
  • Python virtual environment with PyTorch, Gymnasium, and Stable-Baselines3 installed.
  • System-level swig package installed via dnf.

Keep everything running

No action needed. Choose this if you want to keep experimenting with hyperparameters, training for more timesteps, or trying other Gymnasium environments.

  • Your trained models (ppo_car_racing.zip and sac_car_racing.zip) can be reloaded at any time with PPO.load() or SAC.load().
  • Your TensorBoard logs in tb_logs/ are available for further analysis.
  • The virtual environment stays on disk (approximately 2-3 GB with PyTorch included).
  • The system swig package remains installed and may be useful for other Box2D or C/C++ binding projects.

Pause - I'll come back to this later

Free up a small amount of memory by deactivating your virtual environment. Your files and models stay on disk so you can resume later.

  • If your virtual environment is still active, deactivate it by running:
deactivate
  • When you come back, reactivate the environment from your project folder by running source venv/bin/activate.
  • All your trained models, videos, and scripts will be exactly where you left them.

Delete - I don't want to use this again

Remove all project files, trained models, and the virtual environment. This frees up disk space but means you would need to retrain from scratch if you come back.

  • Deactivate your virtual environment first (if active) by running:
deactivate
  • Navigate to your Desktop and delete the entire project folder by running:
cd ~/Desktop
rm -rf self-driving-rl

Adjust the folder name

Replace self-driving-rl with whatever you named your project folder when you created it in Step 1.

  • This removes all Python scripts, trained model files, TensorBoard logs, recorded videos, reward plots, and the virtual environment.
  • Optionally, remove the system swig package by running:
sudo dnf remove swig
  • Confirm the removal when prompted. This is safe to remove if you do not plan to build other Box2D or C/C++ binding packages.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete deep reinforcement learning pipeline that taught a neural network to drive a car from nothing but raw pixel observations.

You've learned how to:

  • Built a full deep RL pipeline on Fedora Linux that trains a convolutional neural network to drive a car from raw 96x96 pixel observations.
  • Used PPO (Proximal Policy Optimization) with frame stacking, reward curves, and TensorBoard logging to track your agent's improvement from crashing immediately to completing laps.
  • Recorded and compared random vs trained agent performance with video evidence and numerical reward metrics.
  • Secret Mission: Compared PPO vs SAC, learning the key distinction between on-policy and off-policy reinforcement learning.

Ready to quiz yourself?