RL-Optimized Physical-to-Digital Sync

Train a PyTorch REINFORCE agent to correct noisy physical tracking signals.

Introduction

30 Second Summary

Every time you draw on a tablet or wave a hand in front of a sensor, the raw signal is noisy. Jitter, drift, and latency make the digital output look nothing like the smooth motion you intended.

In this project, you will build a reinforcement learning agent that learns to correct those noisy signals by trial and error. You will design a custom environment, train a neural network to discover the optimal correction policy, and deploy the whole system to the cloud.

What You'll Build

A working RL system where you run a training script, watch the reward numbers climb in real time as the agent figures out how to denoise tracking input, and then share the finished environment with anyone via a single URL.

By the end of this project, you'll have:

  • A containerized RL environment you can start, reset, and step through from any script, simulating real-world tracking noise with Gaussian jitter and sinusoidal drift.
  • A live training loop where you watch episode rewards climb from near-zero to high as your PyTorch policy network discovers intelligent correction behavior.
  • A deployed environment on the Hugging Face Hub that anyone can connect to and interact with using the OpenEnv client.
  • Secret Mission: Add trajectory-based delayed rewards so the agent optimizes full stroke quality rather than per-step corrections.

Are there any prerequisites?

You need Python 3.11+, Docker Desktop, and a code editor installed on your machine. Familiarity with basic Python classes (defining methods, using self) is enough to follow along.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This project combines custom reinforcement learning environments with PyTorch policy gradients, so it's worth being clear on your goals before the first line of code.

Set Up the RL Development Stack

Before you can train an agent to correct noisy signals, you need the tools that make it possible. OpenEnv provides the containerized environment framework. PyTorch provides the neural network engine. Together, they form the complete RL development stack.

In this step, you will install these tools and scaffold the project directory that OpenEnv uses to define custom environments. The scaffold gives you a ready-made file structure so you can focus on the RL logic rather than boilerplate.

In this step, get ready to:
  • Install uv, OpenEnv, and PyTorch CPU.
  • Scaffold the stroke sync environment with the OpenEnv CLI.
  • Verify Docker is running and the scaffold structure is valid.
Install uv, OpenEnv, and PyTorch

The RL stack requires three packages. uv is a fast Python package manager that OpenEnv uses internally for dependency locking. OpenEnv itself is the environment framework. PyTorch is the deep learning library you will use to build the policy network.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Install uv and openenv by running these commands:
pip install uv
pip install openenv

What do these packages do?

uv is a fast Python package manager. OpenEnv requires it to lock dependencies inside the scaffolded environment.

openenv installs the OpenEnv framework and the openenv CLI tool. You will use the CLI to scaffold, build, validate, and deploy environments.

  • Verify OpenEnv installed correctly by running:
openenv --version

You should see 0.4.1 printed in your terminal.

Seeing command not found?

If your terminal does not recognize openenv, your Python scripts directory may not be on your PATH. Try closing and reopening your terminal, then run the version check again.

If that does not help, run python -m openenv --version to confirm the package installed. Help me fix my OpenEnv PATH issue

  • Install PyTorch CPU by running this command:
pip install torch==2.12.1 --index-url https://download.pytorch.org/whl/cpu

Why CPU-only PyTorch?

The CPU build is smaller and faster to install. The policy network in this project is intentionally small (two 64-unit hidden layers), so it trains quickly on CPU without needing a GPU.

  • Confirm PyTorch is available by running:
python -c "import torch; print(torch.__version__)"

You should see 2.12.1+cpu printed in your terminal.

Seeing a ModuleNotFoundError?

Make sure you ran the install command with the --index-url flag exactly as shown. Without it, pip may pull a version incompatible with your system.

If you have multiple Python installations, confirm you are using Python 3.11+ by running python --version. Help me debug my PyTorch installation

Scaffold the stroke sync environment

OpenEnv provides a CLI command that generates the entire project structure for a custom environment. This saves you from manually creating the boilerplate files for the server, client, and models.

  • Scaffold the environment by running this command:
openenv init stroke_sync_env

What did this create?

The openenv init command creates a stroke_sync_env/ directory containing a complete environment template. Inside you will find:

  • models.py for defining your action and observation types.
  • client.py for the typed client that connects to the environment server.
  • server/app.py for the FastAPI entry point.
  • server/stroke_sync_environment.py for your environment logic (reset, step, reward).
  • openenv.yaml and a Dockerfile for containerized deployment.
  • Move into the new project directory by running:
cd stroke_sync_env
  • Confirm the scaffold created the expected files by running:
ls

You should see models.py, client.py, openenv.yaml, and a server/ directory listed in the output.

Don't see any files?

Make sure you ran cd stroke_sync_env after the init command. The files are inside the generated folder, not in your Desktop directory.

If the folder does not exist, check for error messages from the openenv init command. Help me fix my OpenEnv scaffold

Verify Docker is running

OpenEnv environments run inside Docker containers. Before going further, you need to confirm the Docker daemon is active on your machine.

  • Check that Docker is running by executing:
docker info

Before you run this, what do you expect to see if Docker Desktop is running correctly?

You should see a block of information about your Docker installation (server version, storage driver, number of containers). If you see this, your full RL development stack is ready.

Seeing 'Cannot connect to the Docker daemon'?

This means Docker Desktop is not running. Open Docker Desktop from your applications (press Cmd+Space on macOS or the Windows key on Windows, type Docker, and press Enter). Wait for the whale icon to stop animating, then run docker info again.

Help me get Docker running

  • Confirm the server directory also has the correct files by running:
ls server/

You should see app.py and stroke_sync_environment.py listed.

Your RL development stack is fully set up. You have OpenEnv for the environment framework, PyTorch for the neural network, and a scaffolded project ready for custom logic. Next up, you will fill in the environment with real stroke simulation and reward engineering.

Build the Stroke Sync Environment

Your scaffolded files are placeholders. The environment is the RL problem itself.

Without a well-designed observation space, action space, and reward signal, no agent can learn anything meaningful. In this step, you will define the data contracts, implement the simulation logic that generates noisy strokes, and wire everything into a running server.

In this step, get ready to:
  • Define Pydantic models for the action and observation spaces.
  • Implement the environment logic with noise simulation and reward calculation.
  • Wire up the FastAPI server and typed client, then verify the server starts.
Define the action and observation models

Every interaction between the RL agent and the environment has a strict shape. The agent sends a correction vector (how much to nudge the noisy point). The environment returns the current tracking state.

  • Open stroke_sync_env/models.py in your code editor.
  • Replace the placeholder comment with the following model definitions:
from pydantic import Field
from openenv.core.env_server.types import Action, Observation


class StrokeAction(Action):
    """Correction vector to apply to the noisy stroke point."""

    dx: float = Field(..., description="X-axis correction")
    dy: float = Field(..., description="Y-axis correction")


class StrokeObservation(Observation):
    """Current state of the stroke tracking."""

    noisy_x: float = Field(..., description="Noisy X position from simulated tracker")
    noisy_y: float = Field(..., description="Noisy Y position from simulated tracker")
    velocity_x: float = Field(..., description="X velocity of stroke")
    velocity_y: float = Field(..., description="Y velocity of stroke")
    noise_level: float = Field(..., description="Current noise magnitude")
    step_index: int = Field(..., description="Current step in the stroke")
    total_steps: int = Field(..., description="Total steps in the episode")

What does this code do?

StrokeAction extends OpenEnv's base Action class. It defines exactly two floats (dx, dy) that represent how much the agent wants to correct the noisy point.

StrokeObservation extends Observation and carries everything the agent can see: the noisy position, current velocity, noise magnitude, and step progress. These Pydantic models give OpenEnv type safety for every interaction.

  • Save models.py.
  • Verify the models import cleanly by running:
python -c "from stroke_sync_env.models import StrokeAction, StrokeObservation; print('Models OK')"

You should see Models OK printed with no errors.

Seeing a ModuleNotFoundError?

Make sure you are running the command from the parent directory that contains the stroke_sync_env folder (not from inside it). The import path stroke_sync_env.models expects the package folder to be a child of your current working directory.

help me fix a ModuleNotFoundError when importing stroke_sync_env.models

Implement the environment logic

The environment simulates a physical stylus drawing a smooth curve while a noisy tracker distorts it. The agent's job is to correct each noisy point back toward the true path.

  • Open stroke_sync_env/server/stroke_sync_environment.py in your editor.
  • Replace the placeholder with the class definition, constructor, and stroke generation method:
import math
import random
from uuid import uuid4

from openenv.core.env_server.interfaces import Environment
from openenv.core.env_server.types import State
from models import StrokeAction, StrokeObservation


class StrokeSyncEnvironment(Environment):
    def __init__(self):
        self._state = State(episode_id=str(uuid4()), step_count=0)
        self._total_steps = 50
        self._true_path = []
        self._step_index = 0
        self._noise_level = 0.1
        self._cumulative_reward = 0.0

    def _generate_stroke(self):
        """Generate a smooth reference stroke as a sine-based curve."""
        self._true_path = []
        for i in range(self._total_steps):
            t = i / self._total_steps
            x = t
            y = 0.3 * math.sin(2 * math.pi * t) + 0.2 * math.sin(4 * math.pi * t)
            self._true_path.append((x, y))

What does this code do?

The class extends OpenEnv's Environment interface, which requires reset(), step(), and a state property.

_generate_stroke() builds the ground-truth path the agent tries to recover. It uses a sine combination that produces a smooth, realistic curve with 50 discrete points.

  • Directly below _generate_stroke, add the noise simulation and the reset() method:
    def _add_noise(self, x, y):
        """Add simulated tracking noise: Gaussian jitter plus sinusoidal drift."""
        jitter_x = random.gauss(0, self._noise_level)
        jitter_y = random.gauss(0, self._noise_level)
        drift_x = 0.02 * math.sin(self._step_index * 0.3)
        drift_y = 0.01 * math.cos(self._step_index * 0.2)
        return x + jitter_x + drift_x, y + jitter_y + drift_y

    def reset(self) -> StrokeObservation:
        self._state = State(episode_id=str(uuid4()), step_count=0)
        self._step_index = 0
        self._cumulative_reward = 0.0
        self._noise_level = random.uniform(0.05, 0.15)
        self._generate_stroke()

        true_x, true_y = self._true_path[0]
        noisy_x, noisy_y = self._add_noise(true_x, true_y)

        return StrokeObservation(
            noisy_x=noisy_x,
            noisy_y=noisy_y,
            velocity_x=0.0,
            velocity_y=0.0,
            noise_level=self._noise_level,
            step_index=0,
            total_steps=self._total_steps,
            done=False,
            reward=0.0,
        )

What does this code do?

_add_noise() combines two types of tracking error: random Gaussian jitter (high-frequency) and sinusoidal drift (low-frequency). This mirrors real-world stylus and hand tracking noise.

reset() starts a fresh episode. It generates a new stroke, picks a random noise level between 0.05 and 0.15, and returns the first noisy observation. Each episode is slightly different because the noise level varies.

  • Below reset(), add the first half of the step() method that computes correction and reward:
    def step(self, action: StrokeAction) -> StrokeObservation:
        self._step_index += 1
        self._state.step_count = self._step_index

        done = self._step_index >= self._total_steps - 1

        true_x, true_y = self._true_path[min(self._step_index, len(self._true_path) - 1)]
        noisy_x, noisy_y = self._add_noise(true_x, true_y)

        corrected_x = noisy_x + action.dx
        corrected_y = noisy_y + action.dy

        distance = math.sqrt((corrected_x - true_x) ** 2 + (corrected_y - true_y) ** 2)
        reward = max(0.0, 1.0 - 10.0 * distance)

        self._cumulative_reward += reward

How does the reward work?

The reward formula max(0.0, 1.0 - 10.0 * distance) gives a clear gradient signal. A perfect correction (distance 0) earns 1.0. Any correction that leaves the point more than 0.1 units away scores 0. This makes the learning problem well-shaped for gradient-based optimization.

  • Immediately below the reward line, still inside step(), add the velocity computation, return statement, and the state property:
        velocity_x = 0.0
        velocity_y = 0.0
        if self._step_index > 0:
            prev_x, prev_y = self._true_path[self._step_index - 1]
            velocity_x = true_x - prev_x
            velocity_y = true_y - prev_y

        return StrokeObservation(
            noisy_x=noisy_x,
            noisy_y=noisy_y,
            velocity_x=velocity_x,
            velocity_y=velocity_y,
            noise_level=self._noise_level,
            step_index=self._step_index,
            total_steps=self._total_steps,
            done=done,
            reward=reward,
        )

    @property
    def state(self) -> State:
        return self._state

What does this code do?

The velocity tells the agent which direction the stroke is heading. This context helps it predict upcoming noise patterns.

The state property exposes the internal episode ID and step count to OpenEnv's session management layer.

  • Save stroke_sync_environment.py.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm your file has these sections in order:

  • Imports (math, random, uuid4, Environment, State, StrokeAction, StrokeObservation).
  • Class StrokeSyncEnvironment(Environment) with __init__ setting up 6 instance variables.
  • _generate_stroke() building 50 sine-based points.
  • _add_noise() adding jitter and drift.
  • reset() returning a fresh StrokeObservation.
  • step() computing correction, reward, velocity, and returning the next observation.
  • state property returning self._state.
Wire up the server and client

The environment logic is ready. Now you need to expose it through a FastAPI server and give it a typed client that agents will use to interact with it.

  • Open stroke_sync_env/server/app.py and replace the placeholder with:
from openenv.core.env_server import create_app
from models import StrokeAction, StrokeObservation
from .stroke_sync_environment import StrokeSyncEnvironment

app = create_app(
    StrokeSyncEnvironment, StrokeAction, StrokeObservation, env_name="stroke_sync_env"
)

What does this code do?

create_app takes your environment class (not an instance) so that each WebSocket session gets its own fresh environment. It automatically wires up /health, /reset, /step, and /state endpoints from your action and observation models.

  • Save app.py.
  • Start the environment server by running:
uv run --project . server

Before you check the result, what do you expect to see if the environment loaded correctly?

You should see FastAPI startup logs indicating the server is running on port 8000.

  • Confirm the health endpoint works by opening a new terminal and running:
curl http://localhost:8000/health

You should see a JSON response confirming the server is healthy.

Server won't start?

If you see an ImportError, double-check that stroke_sync_environment.py uses from models import StrokeAction, StrokeObservation (not a relative import with a dot). The OpenEnv server runner sets the Python path so that models.py is importable directly.

If you see Address already in use, another process is occupying port 8000. Stop it or check for lingering processes.

help me debug why my OpenEnv server won't start

Now implement the client that agents will use to interact with the server.

  • Open stroke_sync_env/client.py and replace the placeholder with:
from openenv.core.env_client import EnvClient
from openenv.core.client_types import StepResult
from openenv.core.env_server.types import State
from .models import StrokeAction, StrokeObservation


class StrokeSyncEnv(EnvClient[StrokeAction, StrokeObservation, State]):
    def _step_payload(self, action: StrokeAction) -> dict:
        return {"dx": action.dx, "dy": action.dy}

    def _parse_result(self, payload: dict) -> StepResult[StrokeObservation]:
        obs_data = payload.get("observation", {})
        obs = StrokeObservation(
            noisy_x=obs_data.get("noisy_x", 0.0),
            noisy_y=obs_data.get("noisy_y", 0.0),
            velocity_x=obs_data.get("velocity_x", 0.0),
            velocity_y=obs_data.get("velocity_y", 0.0),
            noise_level=obs_data.get("noise_level", 0.0),
            step_index=obs_data.get("step_index", 0),
            total_steps=obs_data.get("total_steps", 50),
            done=payload.get("done", False),
            reward=payload.get("reward", 0.0),
        )
        return StepResult(
            observation=obs,
            reward=payload.get("reward", 0.0),
            done=payload.get("done", False),
        )

    def _parse_state(self, payload: dict) -> State:
        return State(
            episode_id=payload.get("episode_id", ""),
            step_count=payload.get("step_count", 0),
        )

What does this code do?

StrokeSyncEnv extends OpenEnv's EnvClient with type parameters for your action, observation, and state. It bridges the WebSocket connection to your typed Python objects.

_step_payload() serializes the action for transmission. _parse_result() deserializes the server response back into a typed StepResult containing your StrokeObservation.

  • Save client.py.
  • Open stroke_sync_env/__init__.py and replace the placeholder with the package exports:
from .models import StrokeAction, StrokeObservation
from .client import StrokeSyncEnv
  • Save __init__.py.
  • In the terminal where the server is NOT running, verify the package imports work:
python -c "from stroke_sync_env import StrokeAction, StrokeObservation, StrokeSyncEnv; print('Package OK')"

You should see Package OK confirming all exports resolve.

Import still failing?

Check that __init__.py uses relative imports with dots (from .models and from .client) while client.py also uses from .models. The server-side files (stroke_sync_environment.py and app.py) use from models import without a dot because they run in a different context.

help me fix import errors in my OpenEnv package

Your environment is alive and responding to requests. Next up, you will connect a random agent to prove the environment works and see just how badly random corrections perform.

Run a Random Agent and Watch It Fail

Your environment is live and responding to requests. Now for the critical question: does correcting noisy strokes actually require intelligence, or could random guesses get the job done?

In this step, you'll write an agent that applies completely random correction vectors at every step. The reward signal will tell you exactly how well (or badly) random guessing performs.

In this step, get ready to:
  • Create a random agent script that connects to the running environment server.
  • Run 5 episodes and observe the consistently low rewards that prove intelligent behavior is needed.
Create the random agent script

The random agent lives outside the stroke_sync_env package because it is a consumer of the environment, not part of the environment itself.

  • Create a new file called random_agent.py in the root of your stroke_sync_env project directory (the same level as the stroke_sync_env/ folder).
  • Paste the following code into random_agent.py:
import random
from stroke_sync_env import StrokeAction, StrokeSyncEnv


def run_random_agent(num_episodes=5):
    # Connect to the running OpenEnv server
    client = StrokeSyncEnv(base_url="http://localhost:8000")

    with client:
        for episode in range(num_episodes):
            # Start a fresh episode
            result = client.reset()
            obs = result.observation
            total_reward = 0.0
            steps = 0

            while not obs.done:
                # Apply a completely random correction
                action = StrokeAction(
                    dx=random.uniform(-0.2, 0.2),
                    dy=random.uniform(-0.2, 0.2),
                )
                result = client.step(action)
                obs = result.observation
                total_reward += result.reward
                steps += 1

            print(f"Episode {episode + 1}: {steps} steps | Total reward: {total_reward:.2f}")


if __name__ == "__main__":
    run_random_agent()

What does this code do?

  • StrokeSyncEnv(base_url="http://localhost:8000") connects to the FastAPI server you started in the previous step.
  • client.reset() starts a fresh episode, generating a new reference stroke with new noise.
  • random.uniform(-0.2, 0.2) picks a correction that has no relationship to the actual noise. It pushes the point in an arbitrary direction.
  • The loop accumulates total reward across all 50 steps of the stroke. A perfect agent could score close to 50 (1.0 per step). A random agent will score far lower.
  • Save random_agent.py.
  • Confirm the file exists by running:
ls random_agent.py

You should see random_agent.py listed in the output.

File not found?

Make sure you created the file in the project root (the same directory that contains the stroke_sync_env/ folder), not inside the package itself.

help me find where to put random_agent.py in my project structure

Run the random agent and observe the results

The environment awards up to 1.0 reward per step when the corrected point lands exactly on the true path. Over 50 steps, a perfect agent could score close to 50. Before you run this, think about what score a random agent might achieve.

  • Make sure the environment server is still running in your other terminal (from the previous step).
  • Before running the script, predict: do you think random corrections will score closer to 0 or closer to 50?
  • Run the random agent by executing this command in your project root terminal:
python random_agent.py

You'll see output like this (exact numbers will vary due to randomness):

Episode 1: 49 steps | Total reward: 3.41
Episode 2: 49 steps | Total reward: 4.12
Episode 3: 49 steps | Total reward: 2.87
Episode 4: 49 steps | Total reward: 3.65
Episode 5: 49 steps | Total reward: 4.03

Total rewards land somewhere between 2 and 5 out of a possible 50. That is barely above zero.

Why are the rewards so low?

The reward function is max(0.0, 1.0 - 10.0 * distance). A correction only earns reward if it moves the point closer to the true path. Random corrections push the point in arbitrary directions, so most steps score 0.0.

The few points that do score happen purely by luck. The agent needs to learn which direction the noise is pushing and correct in the opposite direction. That is exactly the problem a neural network will solve.

Seeing a connection error?

The environment server must be running in a separate terminal. Switch back to that terminal and confirm it is still active. If it crashed, restart it by running uv run --project . server from the stroke_sync_env directory.

help me debug my random agent connection to the OpenEnv server

✔️ Awesome, I've got everything!

Great. Make sure you have saved random_agent.py and confirmed the low reward output.

ⓧ I'd like to double check the full code

Here is the complete random_agent.py file:

import random
from stroke_sync_env import StrokeAction, StrokeSyncEnv


def run_random_agent(num_episodes=5):
    client = StrokeSyncEnv(base_url="http://localhost:8000")

    with client:
        for episode in range(num_episodes):
            result = client.reset()
            obs = result.observation
            total_reward = 0.0
            steps = 0

            while not obs.done:
                action = StrokeAction(
                    dx=random.uniform(-0.2, 0.2),
                    dy=random.uniform(-0.2, 0.2),
                )
                result = client.step(action)
                obs = result.observation
                total_reward += result.reward
                steps += 1

            print(f"Episode {episode + 1}: {steps} steps | Total reward: {total_reward:.2f}")


if __name__ == "__main__":
    run_random_agent()

The numbers don't lie. Random corrections are nearly useless. Next up, you'll build the PyTorch neural network that learns to predict exactly which correction to apply at each step.

Build the PyTorch REINFORCE Agent

The random agent proved that corrections need to be intelligent. Random dx/dy values scored 2-5 out of 50 possible reward. The agent needs to learn which direction to push corrections based on what it observes about the noise.

In this step, you will build a PyTorch neural network that outputs correction vectors as probability distributions. The REINFORCE algorithm trains this network by reinforcing actions that led to high reward and discouraging actions that did not.

In this step, get ready to:
  • Build a PolicyNetwork that outputs Gaussian correction distributions.
  • Bridge OpenEnv observations to PyTorch tensors.
  • Implement the REINFORCE policy gradient training loop.
Build the PolicyNetwork

The policy network is the brain of your RL agent. It takes an observation (6 numbers describing the current noisy state) and outputs a probability distribution over correction vectors.

  • Create a new file called train_agent.py in the root of your stroke_sync_env project folder (the same folder where random_agent.py lives).
  • Add the imports and PolicyNetwork class by pasting this code:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Normal
from stroke_sync_env import StrokeAction, StrokeSyncEnv


class PolicyNetwork(nn.Module):
    """MLP policy that outputs correction vector distributions."""

    def __init__(self, input_size=6, hidden_size=64, output_size=2):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
        )
        self.mean_head = nn.Linear(hidden_size, output_size)
        self.log_std = nn.Parameter(torch.zeros(output_size))

    def forward(self, x):
        features = self.network(x)
        mean = self.mean_head(features)
        std = torch.exp(self.log_std)
        return mean, std

What does this network do?

The network has two hidden layers of 64 neurons with ReLU activations. It takes 6 inputs (noisy position, velocity, noise level, step progress) and produces two outputs.

  • mean_head predicts the average correction vector (dx, dy) the agent should apply.
  • log_std is a learnable parameter controlling how much the agent explores. Early in training, large std means wide exploration. As training progresses, the network narrows its distribution around good corrections.
Add the observation-to-tensor bridge

OpenEnv gives you typed Pydantic objects. PyTorch needs float tensors. This function is the bridge between the two frameworks.

  • Add the obs_to_tensor function below the PolicyNetwork class:
def obs_to_tensor(obs):
    """Convert StrokeObservation to a PyTorch tensor."""
    return torch.tensor(
        [
            obs.noisy_x,
            obs.noisy_y,
            obs.velocity_x,
            obs.velocity_y,
            obs.noise_level,
            obs.step_index / obs.total_steps,
        ],
        dtype=torch.float32,
    )

Why normalize step_index?

Dividing step_index by total_steps squashes it to the 0-1 range. Neural networks learn faster when all inputs are on similar scales. The other values (positions, velocities, noise) already fall near that range.

  • Save train_agent.py.
  • Verify the imports work by running this command:
python -c "from train_agent import PolicyNetwork, obs_to_tensor; print('Network and bridge loaded OK')"

You should see Network and bridge loaded OK printed to your terminal.

Seeing a ModuleNotFoundError?

  • Make sure you are running the command from the project root folder (the one containing stroke_sync_env/ and train_agent.py).
  • Confirm PyTorch is installed by running python -c "import torch; print(torch.__version__)". It should print 2.12.1.

Help me debug my PyTorch import error in train_agent.py

Implement the REINFORCE training loop

The training loop connects everything. Each episode, the agent collects experience by interacting with the environment, then updates its policy to favor actions that led to higher rewards.

  • Add the first part of the train() function below obs_to_tensor. This part sets up the optimizer and collects experience each episode:
def train(num_episodes=500, gamma=0.99, lr=1e-3):
    policy = PolicyNetwork()
    optimizer = optim.Adam(policy.parameters(), lr=lr)

    client = StrokeSyncEnv(base_url="http://localhost:8000")

    with client:
        for episode in range(num_episodes):
            result = client.reset()
            obs = result.observation

            log_probs = []
            rewards = []

            while not obs.done:
                state_tensor = obs_to_tensor(obs)
                mean, std = policy(state_tensor)
                dist = Normal(mean, std)
                action_tensor = dist.sample()
                log_prob = dist.log_prob(action_tensor).sum()

                action = StrokeAction(
                    dx=action_tensor[0].item(),
                    dy=action_tensor[1].item(),
                )

                result = client.step(action)
                obs = result.observation

                log_probs.append(log_prob)
                rewards.append(result.reward)

What happens in the collection loop?

  • The policy network sees the current observation and outputs a Gaussian distribution (mean + std) over correction vectors.
  • dist.sample() draws a random action from that distribution. Early on, the distribution is wide (exploring). Later, it narrows around good corrections.
  • log_prob records how likely the chosen action was under the current policy. This is what REINFORCE uses to compute gradients.
  • Now add the second part of the train() function directly below the collection loop. This computes returns, backpropagates the loss, and saves the model:
            # Compute discounted returns
            returns = []
            g = 0.0
            for r in reversed(rewards):
                g = r + gamma * g
                returns.insert(0, g)
            returns = torch.tensor(returns, dtype=torch.float32)

            # Normalize returns for stable gradients
            if returns.std() > 0:
                returns = (returns - returns.mean()) / (returns.std() + 1e-8)

            # Policy gradient loss
            loss = torch.tensor(0.0)
            for log_prob, g in zip(log_probs, returns):
                loss = loss - log_prob * g

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            total_reward = sum(rewards)
            if (episode + 1) % 25 == 0:
                print(f"Episode {episode + 1}/{num_episodes} | Reward: {total_reward:.2f}")

    print("Training complete!")
    torch.save(policy.state_dict(), "stroke_policy.pt")
    print("Model saved to stroke_policy.pt")


if __name__ == "__main__":
    train()

How does REINFORCE learn?

  • Discounted returns weight future rewards by gamma=0.99. A reward 10 steps in the future counts almost as much as an immediate reward.
  • Normalizing returns stabilizes training. Without normalization, a batch of all-high or all-low returns produces unhelpful gradient magnitudes.
  • The loss -log_prob * g increases the probability of actions that led to high returns and decreases it for actions with low returns. This is the core REINFORCE gradient.

Watch the indentation

The second block of code continues inside the for episode loop (3 levels of indentation). The print("Training complete!") lines and if __name__ block sit at the top level, outside the with client: block.

  • Save train_agent.py.

Before you run training, what do you expect to see? The random agent scored 2-5 per episode. Do you think the neural network will do better on its very first episode, or does it need time to learn?

  • Make sure the OpenEnv server is still running in your other terminal. Run training by executing this command:
python train_agent.py

The first print appears at episode 25. You should see a reward value printed that is already higher than the random agent's 2-5 range. Press Ctrl+C after you see the first print to stop training for now. The full 500-episode run happens in the next step.

Seeing a ConnectionRefusedError?

  • The OpenEnv server must be running on port 8000. Switch to the terminal where you started it and confirm it is still active. If it stopped, restart it with uv run --project . server from inside the stroke_sync_env directory.
  • If you see a syntax error instead, double-check that the indentation of the second code block matches the first. The returns = [] line should be at the same level as log_probs = [] above it.

Help me debug my REINFORCE training script

✔️ Awesome, I've got everything!

Great. Double check you have saved train_agent.py and that training printed an increasing reward.

ⓧ I'd like to double check the full code

Your train_agent.py should contain these sections in order. Scroll up and confirm each is present:

  • Imports (torch, torch.nn, torch.optim, torch.distributions.Normal, StrokeAction, StrokeSyncEnv).
  • PolicyNetwork class with __init__ (Sequential network, mean_head, log_std) and forward method.
  • obs_to_tensor() function returning a 6-element float32 tensor.
  • train() function with the episode collection loop (Normal distribution sampling, log_prob recording).
  • Discounted returns computation, normalization, policy gradient loss, optimizer step, and episode printing every 25 episodes.
  • if __name__ == "__main__": train() at the bottom.

Your agent is learning. Even that single data point at episode 25 shows the gradient signal flowing through the network. Next up, you will run the full 500-episode training and deploy the environment to Hugging Face Hub.

Train and Deploy to Hugging Face Hub

In the previous step, you built a REINFORCE agent with a policy network that outputs correction distributions. The agent has the architecture it needs to learn, but it has never actually trained.

Training is where everything comes together. Your reward signal, observation space, and policy gradient all get tested against reality. If rewards climb, it means the entire pipeline works. After confirming the agent learns, you will package and deploy the environment to Hugging Face Hub so anyone can interact with it.

In this step, get ready to:
  • Run the full 500-episode training loop and watch rewards climb.
  • Save and verify the trained model weights.
  • Build, validate, and deploy the environment to Hugging Face Hub.
Run the full training loop

The train() function in train_agent.py runs 500 episodes, printing a reward summary every 25 episodes. Each episode takes the agent through all 50 steps of a noisy stroke, collecting gradient signal along the way.

  • Make sure the OpenEnv server is still running on port 8000 in your other terminal.
  • In your second terminal (the one you used for random_agent.py), start training by running:
python train_agent.py

Training takes a few minutes on CPU. You will see output printed every 25 episodes showing the total reward for that episode.

What should I see?

Early episodes (1-50) print rewards around 3-8. This is barely better than random. By episode 200, rewards should reach 15-20. By episode 500, rewards climb to 30 or higher.

The agent is discovering that corrections should oppose the noise direction. It learns to use velocity and position context to predict where the true path is relative to the noisy reading.

Rewards staying flat near zero?

Make sure the OpenEnv server is still running. If it crashed, restart it with uv run --project . server from inside the stroke_sync_env directory.

If you see connection errors, confirm port 8000 is not blocked by another process.

Help me debug my training rewards

Verify the trained model

When training finishes, train_agent.py automatically saves the policy network weights to stroke_policy.pt using torch.save. You should see two confirmation lines printed in your terminal.

  • Confirm the model file exists by running:
ls stroke_policy.pt

You should see stroke_policy.pt listed. This file contains the trained weights for your PolicyNetwork.

  • Verify the model loads correctly by running:
python -c "import torch; from train_agent import PolicyNetwork; p = PolicyNetwork(); p.load_state_dict(torch.load('stroke_policy.pt')); print('Model loaded successfully')"

You should see Model loaded successfully printed with no errors.

File not found?

Make sure you are running the command from the same directory where train_agent.py lives (your project root, not inside stroke_sync_env/).

If training was interrupted before it finished, run python train_agent.py again. The script saves the model only after all 500 episodes complete.

Help me find my saved model file

Deploy to Hugging Face Hub

Your trained agent proves the environment works. Now you will package the environment as a Docker image and deploy it to Hugging Face Spaces so anyone can connect their own agent to it.

  • Navigate into the environment package directory by running:
cd stroke_sync_env
  • Build the Docker image for your environment by running:
openenv build

You should see Docker build output ending with a success message. This packages your environment, its dependencies, and the server entrypoint into a container image.

Docker build failing?

Make sure Docker Desktop is running. You can confirm by running docker info in your terminal. If you see a connection error, open Docker Desktop and wait for it to finish starting.

Help me fix my Docker build

  • Validate the environment structure and entrypoints by running:
openenv validate

You should see validation pass with no errors. This checks that your openenv.yaml, models, and server entrypoints are correctly configured.

  • Before you deploy, predict what you think will happen. Will openenv push need any authentication, or will it just work?
  • Deploy the environment to Hugging Face Spaces by running:
openenv push

What happens during push?

The openenv push command uploads your Docker image to a Hugging Face Space. If you are not already logged in, it will prompt you to authenticate via the Hugging Face Hub CLI. Follow the prompt to log in with your Hugging Face account.

Once the push completes, it prints the URL of your live Space. Anyone with that URL can connect an agent to your environment.

You should see a URL printed in your terminal pointing to your live Hugging Face Space. This is your deployed environment.

Push failing with authentication error?

Make sure you have a Hugging Face account. If the push prompts you to log in, follow the interactive prompt. You may need to create an access token on the Hugging Face website under your account settings.

Help me authenticate with Hugging Face for openenv push

Your RL pipeline is complete. You designed a custom environment, trained an agent that demonstrably learned, and deployed the whole thing as shareable infrastructure. Next up, the Secret Mission challenges you to rethink the reward signal itself.

Secret mission

Add Trajectory-Based Delayed Rewards

Your per-step reward agent optimizes each correction independently. It has no concept of the full stroke. What if you forced the agent to plan ahead by withholding all reward until the episode ends, then scoring the entire trajectory for smoothness, accuracy, and consistency?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on free-tier resources (local Docker and a free Hugging Face Space), so there are no ongoing costs.

Resources you used:

  • Local Docker image for the stroke_sync_env environment.
  • Hugging Face Space hosting the deployed environment (free tier).
  • Local project files in the stroke_sync_env directory on your Desktop.
  • Installed Python packages (openenv, torch, uv).

Keep everything running

No action needed. Choose this if you are still actively building or want to experiment further with your RL environment.

  • Your Hugging Face Space stays active on the free tier indefinitely.
  • Your local Docker image remains available for future openenv build and training runs.
  • Your installed packages (openenv, torch, uv) remain available for other projects.

Pause - I'll come back to this later

Stop running processes to free up memory while keeping all files intact for later.

  • Stop the local OpenEnv server if it is still running by pressing Ctrl+C in the terminal where it is active.
  • Stop any running Docker containers for this project by running:
docker stop $(docker ps -q --filter ancestor=stroke_sync_env)
  • Your Hugging Face Space continues running on the free tier at no cost.
  • Your project files and trained model remain on disk for when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Delete your Hugging Face Space by opening your Space page, clicking the three-dot menu near the top, selecting Settings, scrolling to the bottom, and clicking Delete this Space.
  • Remove the local Docker image by running:
docker rmi stroke_sync_env
  • Remove the project directory from your Desktop by running:
cd ~/Desktop
rm -rf stroke_sync_env
  • Uninstall the Python packages by running:
pip uninstall openenv torch uv -y

Nice Work!

Nice Work!

You just built a complete reinforcement learning pipeline from scratch. A PyTorch neural network learned to correct noisy physical tracking signals into smooth digital strokes, and you deployed the whole thing as a live OpenEnv environment on the Hugging Face Hub.

You've learned how to:

  • Design a custom RL environment with observation spaces, action spaces, and shaped reward signals that simulate real-world physical tracking noise.
  • Implement the REINFORCE policy gradient algorithm in PyTorch, connecting Gaussian action distributions and discounted returns to a live environment through OpenEnv's typed API.
  • Deploy a containerized RL environment to Hugging Face Hub using the OpenEnv CLI, proving an agent that climbed from near-zero reward to intelligent corrections over 500 training episodes.
  • Secret Mission: Added trajectory-based delayed rewards that score full stroke quality at episode end, producing qualitatively smoother strokes by forcing the agent to plan ahead.

Ready to quiz yourself?