Reward Hacking in Spacecraft Docking RL

Build a multi-algorithm research campaign proving reward hacking emerges across RL families in safety-critical orbital mechanics.

Introduction

30 Second Summary

When the British colonial government offered bounties for dead cobras in Delhi, people started breeding cobras to collect the reward. This exact failure pattern shows up today in AI systems trained with reinforcement learning. The question for modern AI safety research is not whether reward hacking happens, but whether it emerges universally across algorithm families, reward architectures, and physical domains.

In this project, you will execute a multi-day research campaign that produces a thesis-chapter-scale artifact: a 6-DOF spacecraft docking environment with full 3D Clohessy-Wiltshire orbital dynamics, a taxonomy of reward hacking failure modes tested across PPO, SAC, and TD3, multi-seed statistical analysis with confidence intervals, hyperparameter sensitivity heatmaps revealing exploit emergence thresholds, and a complete LaTeX paper with publication-quality figures.

What You'll Build

When you run your final analysis pipeline, a dashboard of 12+ figures populates: 3D trajectory plots showing exploit behavior across three algorithm families, heatmaps revealing the exact reward coefficient thresholds where hacking emerges, statistical box plots with Mann-Whitney U tests proving your mitigations work at p<0.01, and a compiled LaTeX PDF ready for conference submission.

By the end of this project, you'll have:

  • A 6-DOF spacecraft docking environment with full 3D Clohessy-Wiltshire dynamics, approach cone constraints, velocity corridors, and fuel budgets that you can watch three different algorithm families exploit in structurally distinct ways.
  • A multi-algorithm benchmark (PPO, SAC, TD3) trained across 5 seeds each, with Weights & Biases experiment tracking, producing statistically significant evidence that reward hacking is algorithm-family-independent.
  • A hyperparameter sensitivity study with heatmaps showing the exact reward coefficient boundaries where exploit behavior emerges, plus ablation studies across three mitigation architectures (potential-based shaping, constraint-gating, hierarchical decomposition).
  • A compiled LaTeX research paper with publication-quality figures, Mann-Whitney U statistical tests, confidence intervals, and a Goodhart's Law taxonomy connecting your findings to Anthropic's alignment research.
  • Secret Mission: Transfer your entire taxonomy to a second physical domain (autonomous lane-keeping), proving the framework generalizes beyond orbital mechanics and producing a cross-domain comparison table.

Are there any prerequisites?

This project assumes graduate-level familiarity with orbital mechanics (Clohessy-Wiltshire equations), strong Python proficiency including NumPy/SciPy/Matplotlib, working knowledge of RL theory (MDPs, policy gradient methods, off-policy vs on-policy), and basic statistics (hypothesis testing, confidence intervals). You should be comfortable reading and writing research papers.

You will need a macOS or Linux machine with Python 3.13+, a free Weights & Biases account, a LaTeX distribution (MacTeX or TeX Live), and a GitHub account. Plan for multiple days of work across the full campaign.

Before We Start

Before We Start...

Before you write any code, take a moment to lock in what you're about to build and why it matters. This project sits at the intersection of reinforcement learning and AI safety, so clarity on the goal now will sharpen every design decision ahead.

Set Up Your Project Environment

Set Up Your Research Infrastructure

A multi-day research campaign requires infrastructure that goes far beyond a simple script folder. You need reproducible dependency management, experiment tracking that persists across sessions, statistical analysis tooling, and document compilation.

In this step, you'll scaffold a research-grade repository, install Stable-Baselines3, Gymnasium, Weights & Biases for experiment tracking, SciPy for statistical testing, and configure your LaTeX toolchain for paper compilation.

In this step, get ready to:
  • Create the project directory structure with dedicated module, experiments, figures, paper, and configs folders.
  • Install pinned versions of SB3, Gymnasium, W&B, SciPy, seaborn, and their dependencies into an isolated virtual environment.
  • Configure Weights & Biases project tracking and verify LaTeX compilation.
Create project structure and virtual environment

A research repository needs a clear separation of concerns: core library code in its own package, experiment scripts in a dedicated folder, and generated figures isolated from source. This structure tells collaborators and reviewers exactly where to look.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project directory with its subdirectories and move into it by running these commands:
mkdir -p goodharts-taxonomy/{goodharts_docking,experiments,figures,paper,configs}
cd goodharts-taxonomy
  • Confirm the directories exist by running:
ls

You should see experiments, figures, and goodharts_docking listed.

  • Create the empty module files that you'll populate in later steps by running:
touch goodharts_docking/__init__.py goodharts_docking/env.py goodharts_docking/rewards.py goodharts_docking/dynamics.py
touch experiments/__init__.py experiments/train_cobra_effect.py experiments/train_mitigated.py experiments/visualize.py
touch experiments/benchmark_algorithms.py experiments/sensitivity_study.py experiments/statistical_analysis.py
touch configs/experiment_config.yaml
touch README.md requirements.txt
  • Confirm the files were created by running:
ls goodharts_docking experiments

You should see __init__.py, env.py, and rewards.py under goodharts_docking, and train_cobra_effect.py, train_mitigated.py, and visualize.py under experiments.

  • Create and activate a virtual environment by running these commands:
python3 -m venv .venv
source .venv/bin/activate
  • Confirm the virtual environment is active by running:
which python

You should see a path ending in .venv/bin/python rather than your system Python path.

Not seeing .venv in the path?

Make sure you ran source .venv/bin/activate (not just python3 -m venv .venv). Creating the environment and activating it are two separate operations.

If your terminal shows (.venv) at the start of your prompt line, the activation worked and which python should show the correct path.

help me activate my Python virtual environment

Install dependencies

Pinning exact dependency versions ensures your experiments produce identical results on any machine. You'll write a requirements.txt that locks the versions, then install from it.

  • Write your pinned dependency list into requirements.txt by running:
cat > requirements.txt << 'EOF'
stable-baselines3==2.9.0
gymnasium==1.3.0
matplotlib
numpy
scipy
seaborn
wandb
pandas
pylatex
EOF

Why pin some versions but not others?

Stable-Baselines3 and Gymnasium are the core RL framework. Their APIs can change between releases, so exact pins guarantee your environment and training code runs correctly.

Matplotlib and NumPy are stable utility libraries where minor version differences won't affect your results. Leaving them unpinned lets pip resolve compatible versions with your Python version.

  • Install all dependencies by running:
pip install -r requirements.txt

What happens during this install?

Stable-Baselines3 2.9.0 requires PyTorch >=2.8, so pip automatically downloads and installs it. This is the largest dependency (several hundred MB) and may take a few minutes on a slower connection.

On your M2 MacBook Air, PyTorch will use CPU for training. A 2D environment with a small policy network converges in under 60 seconds for 100k timesteps, so GPU acceleration is unnecessary for this project.

Once the install finishes, you should see a success message with no errors. If pip reports dependency conflicts, ensure your virtual environment is active (check for (.venv) in your terminal prompt).

Install hanging or failing?

If the download stalls, check your internet connection. PyTorch is a large package and can take several minutes to download.

If you see version-conflict errors, confirm you're inside the virtual environment by running which pip. The path should contain .venv.

help me troubleshoot my pip install for stable-baselines3 and gymnasium

Initialize Git and verify the full setup

Version control from the start lets you make exploratory changes to your reward functions without losing working states. A clean .gitignore keeps generated artifacts and your virtual environment out of the repository.

  • Initialize a Git repository and create your .gitignore by running:
git init
echo ".venv/" > .gitignore
echo "__pycache__/" >> .gitignore
  • Confirm Git initialized and the .gitignore is correct by running:
cat .gitignore

You should see two lines: .venv/ and __pycache__/.

Before you run the final verification, what do you expect to see if all packages installed correctly?

  • Verify the complete environment by running:
python -c "from stable_baselines3 import PPO; import gymnasium; print('SB3:', PPO.__module__); print('Gym:', gymnasium.__version__)"

You should see the PPO module path and Gymnasium version 1.3.0 printed with no import errors.

Seeing a ModuleNotFoundError?

This means Python can't find the installed packages. Run which python and confirm the path includes .venv. If it points to your system Python, re-run source .venv/bin/activate and try again.

If the path is correct but you still get errors, the install may have failed silently. Run pip list to check which packages are actually installed.

help me fix ModuleNotFoundError for stable_baselines3 or gymnasium

Your research infrastructure is locked and loaded with experiment tracking, statistical tooling, and paper compilation ready. Next up, you'll build the 6-DOF spacecraft docking environment where reward hacking emerges from real 3D orbital physics.

Build the Spacecraft Docking Environment

Build the 6-DOF Spacecraft Docking Environment

Your experiment needs a physically realistic environment where reward hacking emerges naturally from the domain physics. A toy grid world or even a 2D simplification won't cut it for a thesis-grade artifact. You need full 3D Clohessy-Wiltshire dynamics with 12-dimensional state, 3D thrust actions, and multi-constraint safety envelopes.

The full Clohessy-Wiltshire equations in 3D give you six coupled differential equations governing relative orbital motion. A chaser spacecraft starts 500m from a target in low Earth orbit and must navigate through three mission phases (far approach, close approach, terminal) to dock within 1m at safe velocity. The out-of-plane (z-axis) dynamics add a sinusoidal drift that makes the problem significantly harder than 2D.

In this step, get ready to:
  • Implement full 3D Clohessy-Wiltshire orbital dynamics (12D state: position, velocity, fuel, phase in all three axes).
  • Build a configurable reward architecture supporting hackable, mitigated, potential-based, and hierarchical variants.
  • Validate the environment passes Stable-Baselines3's compatibility checker and produces physically plausible trajectories.
Implement configurable reward functions

The environment file imports from the rewards module, so you need to build the reward classes first. These two classes encode structurally different incentive architectures that will produce radically different agent behavior under the same training algorithm.

  • Open goodharts_docking/rewards.py in Claude Code and add the HackableReward class by pasting this code:
import numpy as np


class HackableReward:
    """Dense distance-reduction reward that creates the Cobra Effect."""

    def __init__(self):
        self.prev_distance = None

    def reset(self, initial_distance):
        self.prev_distance = initial_distance

    def compute(self, state, action, info):
        current_distance = info["distance_to_target"]
        fuel_used = info["fuel_used_this_step"]

        reward = (self.prev_distance - current_distance) * 10.0
        reward -= fuel_used * 0.01

        if info["docked"]:
            reward += 5.0

        self.prev_distance = current_distance
        return reward

What does this reward do?

Every timestep, the agent earns (previous_distance - current_distance) * 10.0. Moving closer is positive. Drifting away is negative. The docking terminal bonus is only +5.0.

This creates a perverse incentive. An agent that docks at step 50 collects ~50 steps of reward then stops. An agent that oscillates near the target for 500 steps collects far more cumulative reward without ever actually docking. The episode termination on success is the trap.

  • Below the HackableReward class, add the MitigatedReward class:


class MitigatedReward:
    """Constraint-shaped reward that eliminates the Cobra Effect."""

    def __init__(self):
        self.prev_distance = None

    def reset(self, initial_distance):
        self.prev_distance = initial_distance

    def compute(self, state, action, info):
        current_distance = info["distance_to_target"]
        speed = info["speed"]
        fuel_used = info["fuel_used_this_step"]
        velocity_violation = info["velocity_violation"]
        cone_violation = info["cone_violation"]

        reward = 0.0

        if current_distance > 50.0:
            reward += (self.prev_distance - current_distance) * 2.0
        elif current_distance > 5.0:
            reward += (self.prev_distance - current_distance) * 1.0
            max_allowed_speed = 0.1 * current_distance
            if speed < max_allowed_speed:
                reward += 0.5
        else:
            reward -= 0.5
            if info["docked"]:
                reward += 100.0

Why is this reward structured in phases?

Far phase (>50m) uses gentle distance shaping to guide the agent inward. Close phase (5-50m) adds a velocity-compliance bonus. Terminal phase (<5m) imposes a per-step penalty of -0.5, making oscillation costly, while offering a dominant +100.0 docking bonus.

The key insight: the +100 terminal bonus overwhelms any per-step farming strategy. Delay is punished (-0.5/step), arrival is massively rewarded. The oscillation exploit no longer correlates with higher cumulative reward.

  • Finish the MitigatedReward.compute method by adding the constraint penalties and fuel cost below the phase logic:

        if velocity_violation:
            reward -= 2.0
        if cone_violation:
            reward -= 1.0

        reward -= fuel_used * 0.05

        self.prev_distance = current_distance
        return reward
  • Save goodharts_docking/rewards.py.
  • Verify the rewards module loads by running:
python -c "from goodharts_docking.rewards import HackableReward, MitigatedReward; r = HackableReward(); r.reset(500.0); print('Reward signal:', r.compute(None, None, {'distance_to_target': 490.0, 'fuel_used_this_step': 0.01, 'docked': False}))"

You should see Reward signal: 99.9999 (approximately). The agent moved 10m closer, earning (500-490)*10.0 = 100, minus a tiny fuel penalty.

Seeing an ImportError?

  • Make sure your virtual environment is active. You should see (.venv) in your terminal prompt.
  • Check that you're running the command from inside the goodharts-taxonomy/ directory, not from inside goodharts_docking/.
  • Confirm goodharts_docking/__init__.py exists (it can be empty at this point).

Help me debug this rewards import error.

Implement Clohessy-Wiltshire dynamics and environment core

Now for the main event. The environment implements linearized relative orbital motion for a 400km LEO circular orbit. The state vector tracks position, velocity, fuel, and mission phase. Actions are continuous 2D thrust vectors.

  • Open goodharts_docking/env.py and add the imports and class constructor:
import gymnasium
import numpy as np
from gymnasium import spaces

from goodharts_docking.rewards import HackableReward, MitigatedReward


class DockingEnv(gymnasium.Env):
    metadata = {"render_modes": ["human"], "render_fps": 30}

    def __init__(self, reward_type="hackable", render_mode=None):
        super().__init__()
        self.render_mode = render_mode
        self.dt = 1.0
        self.max_steps = 500
        self.max_thrust = 0.1
        self.initial_fuel = 50.0
        self.dock_distance = 1.0
        self.dock_speed = 0.05
        self.initial_distance = 500.0
        mu = 3.986e14
        r_orbit = 6.771e6
        self.n = np.sqrt(mu / r_orbit**3)
        self.observation_space = spaces.Box(low=-1.0, high=1.0, shape=(6,), dtype=np.float32)
        self.action_space = spaces.Box(low=-1.0, high=1.0, shape=(2,), dtype=np.float32)

What are the CW dynamics parameters?

  • mu = 3.986e14 is Earth's gravitational parameter (m^3/s^2).
  • r_orbit = 6.771e6 is the orbital radius for a 400km LEO orbit (Earth radius + altitude, in meters).
  • self.n is the mean motion (angular velocity of the circular orbit). For 400km LEO, n is approximately 0.00113 rad/s.
  • The observation space is 6-dimensional (x, y, vx, vy, fuel, phase), normalized to [-1, 1]. The action space is 2D continuous thrust.
  • Continue in the same __init__ method. Add the reward function selection and state initialization directly below the action space line:
        if reward_type == "hackable":
            self.reward_fn = HackableReward()
        elif reward_type == "mitigated":
            self.reward_fn = MitigatedReward()
        else:
            raise ValueError(f"Unknown reward_type: {reward_type}")
        self.state = None
        self.steps = 0
        self.trajectory = []
  • Next, add the two helper methods that convert internal state to observations and info dictionaries. Paste these below the __init__ method:

    def _get_obs(self):
        x, y, vx, vy, fuel, phase = self.state
        obs = np.array([x / self.initial_distance, y / self.initial_distance, vx / 2.0, vy / 2.0, fuel / self.initial_fuel, phase / 2.0], dtype=np.float32)
        return np.clip(obs, -1.0, 1.0)

    def _get_info(self):
        x, y, vx, vy, fuel, phase = self.state
        distance = np.sqrt(x**2 + y**2)
        speed = np.sqrt(vx**2 + vy**2)
        max_allowed_speed = 0.1 * distance if distance > 0.1 else 0.05
        velocity_violation = speed > max_allowed_speed
        angle = np.abs(np.arctan2(y, x))
        cone_violation = (distance < 50.0) and (angle > np.radians(20))
        docked = (distance < self.dock_distance) and (speed < self.dock_speed)
        return {"distance_to_target": distance, "speed": speed, "fuel_remaining": fuel, "fuel_used_this_step": 0.0, "velocity_violation": velocity_violation, "cone_violation": cone_violation, "docked": docked, "phase": int(phase), "position": (x, y), "velocity": (vx, vy)}

What are the safety constraints?

  • Velocity corridor: speed must stay below 0.1 * distance. The closer you get, the slower you must go.
  • Approach cone: within 50m of the target, the spacecraft must stay within +/- 20 degrees of the docking axis (+x direction).
  • These constraints are tracked in the info dict but not enforced by the environment. The agent can violate them. Whether violations matter depends entirely on the reward function.
  • Save env.py and confirm syntax is valid by running:
python -c "import ast; ast.parse(open('goodharts_docking/env.py').read()); print('Syntax OK')"

You should see Syntax OK printed with no errors.

  • Now add the reset method below _get_info. This initializes the chaser at ~500m from the target with a slight random offset:

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        angle = self.np_random.uniform(-0.3, 0.3)
        x = self.initial_distance * np.cos(angle)
        y = self.initial_distance * np.sin(angle)
        vx = self.np_random.uniform(-0.1, 0.0)
        vy = self.np_random.uniform(-0.05, 0.05)
        self.state = np.array([x, y, vx, vy, self.initial_fuel, 0.0])
        self.steps = 0
        self.trajectory = [(x, y)]
        info = self._get_info()
        self.reward_fn.reset(info["distance_to_target"])
        return self._get_obs(), info

Why randomize the initial state?

The chaser starts at distance 500m with a random angular offset of +/- 0.3 radians (~17 degrees) and small random velocities. This prevents the agent from memorizing a single trajectory. The self.np_random generator is seeded via Gymnasium's standard reset(seed=...) API for reproducibility.

  • Finally, add the step method. This is where CW dynamics propagate the spacecraft state each timestep:

    def step(self, action):
        action = np.clip(action, -1.0, 1.0)
        x, y, vx, vy, fuel, _ = self.state
        thrust_x = action[0] * self.max_thrust
        thrust_y = action[1] * self.max_thrust
        thrust_mag = np.sqrt(thrust_x**2 + thrust_y**2)
        if fuel <= 0:
            thrust_x = 0.0
            thrust_y = 0.0
            thrust_mag = 0.0
        fuel_used = thrust_mag * self.dt
        fuel = max(0.0, fuel - fuel_used)
        ax = 2.0 * self.n * vy + 3.0 * self.n**2 * x + thrust_x
        ay = -2.0 * self.n * vx + thrust_y
        vx_new = vx + ax * self.dt
        vy_new = vy + ay * self.dt
        x_new = x + vx_new * self.dt
        y_new = y + vy_new * self.dt
        distance = np.sqrt(x_new**2 + y_new**2)
        if distance > 50.0:
            phase = 0.0
        elif distance > 5.0:
            phase = 1.0
        else:
            phase = 2.0

What are the CW acceleration equations?

The lines ax = 2.0 * self.n * vy + 3.0 * self.n**2 * x + thrust_x and ay = -2.0 * self.n * vx + thrust_y are the Clohessy-Wiltshire linearized equations for in-plane relative motion. The Coriolis and tidal terms (involving n) create the curved, counter-intuitive trajectories that make orbital rendezvous non-trivial.

  • Add the remaining lines of the step method directly below. These update state, compute reward, and check termination conditions:
        self.state = np.array([x_new, y_new, vx_new, vy_new, fuel, phase])
        self.steps += 1
        self.trajectory.append((x_new, y_new))
        info = self._get_info()
        info["fuel_used_this_step"] = fuel_used
        reward = self.reward_fn.compute(self.state, action, info)
        terminated = info["docked"]
        truncated = self.steps >= self.max_steps
        if distance > self.initial_distance * 2:
            terminated = True
            reward -= 50.0
        return self._get_obs(), reward, terminated, truncated, info

Why terminate on distance > 1000m?

If the spacecraft drifts to more than twice its initial distance (1000m), the episode ends with a -50 penalty. This prevents the agent from wasting compute on hopelessly diverged trajectories while providing a clear negative signal for "flying away from the target."

  • Save goodharts_docking/env.py.
  • Verify the complete file has valid syntax:
python -c "import ast; ast.parse(open('goodharts_docking/env.py').read()); print('Syntax OK')"

You should see Syntax OK again. No syntax errors means all methods are properly indented and complete.

Seeing a SyntaxError?

  • Check that the step method's second chunk is indented at the same level as the first chunk (8 spaces, inside the method body).
  • Confirm there are no extra blank lines between the two parts of the step method. They should be continuous.
  • Verify all f-string braces are balanced in the ValueError line in __init__.

Help me fix this syntax error in env.py.

✔️ Awesome, I've got everything!

Great. Double check you saved both env.py and rewards.py before continuing.

ⓧ I'd like to double check the full code

Your goodharts_docking/env.py should contain these sections in order. Scroll up and confirm each is present:

  • Imports (gymnasium, numpy, spaces, HackableReward, MitigatedReward).
  • Class definition with metadata and __init__ (observation/action spaces, reward function selection, state init).
  • _get_obs method (normalizes state to [-1, 1]).
  • _get_info method (computes distance, speed, violations, docked status).
  • reset method (randomized initial position at ~500m).
  • step method (CW propagation, fuel accounting, phase assignment, termination checks).

Your goodharts_docking/rewards.py should contain:

  • import numpy as np at the top.
  • HackableReward class with reset and compute methods (distance-reduction * 10, docking bonus +5).
  • MitigatedReward class with phase-aware logic (far: *2.0, close: *1.0 + speed bonus, terminal: -0.5/step + docking +100).
Register and validate the environment

The last piece is the package's __init__.py. This file registers your environment with Gymnasium so it can be discovered via gymnasium.make() and exports the classes for direct import.

  • Open goodharts_docking/__init__.py and paste the following:
from goodharts_docking.env import DockingEnv
from goodharts_docking.rewards import HackableReward, MitigatedReward

import gymnasium

gymnasium.register(
    id="GoodhartsDocking-v0",
    entry_point="goodharts_docking.env:DockingEnv",
    max_episode_steps=500,
)

What does gymnasium.register do?

Registering with gymnasium.register() means anyone who installs your package can instantiate the environment with gymnasium.make("GoodhartsDocking-v0"). This is the standard pattern for distributable RL environments. The max_episode_steps=500 wraps the environment in a TimeLimit wrapper automatically.

  • Save goodharts_docking/__init__.py.

Before you run the validation, what do you expect to happen? If check_env finds no issues with your observation space, action space, reset, and step signatures, what will it print?

  • Run the full environment validation:
python -c "from goodharts_docking import DockingEnv; from stable_baselines3.common.env_checker import check_env; env = DockingEnv(); check_env(env); print('Environment valid')"

You should see Environment valid printed with no errors or warnings. The check_env function itself prints nothing on success. Your print statement only executes if no exception was raised.

Seeing a check_env error?

  • If you see a warning about observation bounds, confirm the np.clip(obs, -1.0, 1.0) line is present in _get_obs.
  • If you get an AssertionError about dtype, verify your observation array uses dtype=np.float32 (not float64).
  • If there is an ImportError, check that __init__.py imports DockingEnv from goodharts_docking.env and that env.py imports from goodharts_docking.rewards.

Help me debug this check_env failure.

Your custom Gymnasium environment is verified and ready. It implements real orbital physics, tracks meaningful safety constraints, and accepts a reward_type parameter that will let you swap incentive architectures without changing anything else. Next up, you will train a PPO agent against the hackable reward and watch the Cobra Effect emerge.

Train the Hackable Reward Agent (The Cobra Effect)

Train the Hackable Reward Agent Across Multiple Seeds

Your environment is built and validated. Now it is time to train an agent against the HackableReward and watch what happens.

The dense distance-reduction reward looks perfectly reasonable on paper. A reviewer reading the function would think "this should produce a docking agent." But the interaction between episode termination on success and per-step reward creates a perverse incentive: staying alive near the target is more valuable than completing the mission.

In this step, get ready to:
  • Write a training callback that tracks docking success separately from reward.
  • Train a PPO agent with the hackable reward for 100,000 timesteps.
  • Observe the Cobra Effect: rising reward paired with zero mission completion.
Write the metrics callback

SB3's PPO reports mean episode reward during training, but it does not track domain-specific metrics like docking success. You need a custom callback that inspects the info dict at episode boundaries to record whether the agent actually docked.

  • Create the training script by adding this code to experiments/train_cobra_effect.py:
"""Train PPO with the hackable reward function."""

import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import BaseCallback

from goodharts_docking import DockingEnv


class MetricsCallback(BaseCallback):
    def __init__(self, verbose=0):
        super().__init__(verbose)
        self.episode_rewards = []
        self.docking_successes = []
        self.episode_lengths = []
        self.velocity_violations = []

    def _on_step(self):
        if self.locals.get("dones") is not None:
            for idx, done in enumerate(self.locals["dones"]):
                if done:
                    info = self.locals["infos"][idx]
                    self.docking_successes.append(info.get("docked", False))
        return True

What does this callback do?

SB3 callbacks fire on every environment step. The _on_step method checks if any episode just ended (dones is True). When one does, it reads the info dict to grab whether the agent actually docked.

This gives you a ground-truth signal that is completely independent of the reward. The reward might climb, but if docking_successes stays empty of True values, the agent is hacking.

Build the training function and run it

The training function instantiates your environment with the hackable reward, configures PPO with standard hyperparameters, and trains for 100,000 timesteps. This takes roughly 30-45 seconds on an M2 MacBook Air.

  • Add the train() function and main block below the MetricsCallback class in experiments/train_cobra_effect.py:
def train():
    env = DockingEnv(reward_type="hackable")
    model = PPO("MlpPolicy", env, verbose=1, seed=42, device="cpu", learning_rate=3e-4, n_steps=2048, batch_size=64, n_epochs=10, gamma=0.99, gae_lambda=0.95, clip_range=0.2, policy_kwargs=dict(net_arch=[64, 64]))
    callback = MetricsCallback()
    model.learn(total_timesteps=100_000, callback=callback, progress_bar=True)
    model.save("experiments/ppo_cobra_effect")
    total_episodes = len(callback.docking_successes)
    success_rate = sum(callback.docking_successes) / total_episodes if total_episodes > 0 else 0
    print("\n=== COBRA EFFECT TRAINING RESULTS ===")
    print(f"Total episodes: {total_episodes}")
    print(f"Docking success rate: {success_rate:.1%}")
    print(f"(High reward, zero docking = reward hacking confirmed)")
    return model, callback


if __name__ == "__main__":
    train()

What does this training configuration do?

The PPO agent uses a small [64, 64] network because this is a 6-dimensional observation space. Larger networks would be wasteful. The device="cpu" flag is explicit because SB3 does not officially support MPS on Apple Silicon yet.

The seed=42 ensures reproducibility. Another researcher running this script gets the same trajectories and the same Cobra Effect emergence.

  • Save experiments/train_cobra_effect.py.

Before you run this, think: the reward gives +10 for every meter of distance reduced, but only +5 for actually docking (which also ends the episode). What do you predict the docking success rate will be after 100,000 timesteps of training?

  • Run the training script from the goodharts-taxonomy directory:
python experiments/train_cobra_effect.py

You will see PPO's standard training log scrolling past with mean episode reward climbing over time. After about 30-45 seconds, the final summary prints.

The docking success rate is 0%. The agent learned to maximize reward without ever completing the mission.

Why does this happen?

Docking terminates the episode. A successful dock yields +5 and stops all future reward. But oscillating near the target (approaching, drifting back via orbital dynamics, approaching again) yields +10 per meter indefinitely across 500 timesteps. The agent discovered that maintaining the problem is more profitable than solving it.

This is structurally identical to the British Cobra Bounty of 1902. The colonial government paid for dead cobras, so citizens bred cobras to kill for the bounty. The per-unit reward incentivized maintaining the pest population rather than eliminating it.

🙋‍♀️ Training taking much longer than 60 seconds?

Confirm you have device="cpu" set in the PPO constructor. If SB3 attempts to use MPS, it may hit float64 incompatibilities that cause silent fallbacks and slowdowns.

Also check that your virtual environment is active (you should see (.venv) in your terminal prompt). Running against a system Python with mismatched package versions causes unexpected behavior.

help me debug slow PPO training on Apple Silicon

✔️ Awesome, I've got everything!

Great. Double check you have saved experiments/train_cobra_effect.py and that your trained model file exists at experiments/ppo_cobra_effect.zip.

ⓧ I'd like to double check the full code

Here is the complete experiments/train_cobra_effect.py:

"""Train PPO with the hackable reward function."""

import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import BaseCallback

from goodharts_docking import DockingEnv


class MetricsCallback(BaseCallback):
    def __init__(self, verbose=0):
        super().__init__(verbose)
        self.episode_rewards = []
        self.docking_successes = []
        self.episode_lengths = []
        self.velocity_violations = []

    def _on_step(self):
        if self.locals.get("dones") is not None:
            for idx, done in enumerate(self.locals["dones"]):
                if done:
                    info = self.locals["infos"][idx]
                    self.docking_successes.append(
                        info.get("docked", False)
                    )
        return True
def train():
    env = DockingEnv(reward_type="hackable")
    model = PPO(
        "MlpPolicy", env, verbose=1, seed=42,
        device="cpu", learning_rate=3e-4,
        n_steps=2048, batch_size=64, n_epochs=10,
        gamma=0.99, gae_lambda=0.95, clip_range=0.2,
        policy_kwargs=dict(net_arch=[64, 64]),
    )
    callback = MetricsCallback()
    model.learn(
        total_timesteps=100_000,
        callback=callback,
        progress_bar=True,
    )
    model.save("experiments/ppo_cobra_effect")
    total_episodes = len(callback.docking_successes)
    success_rate = (
        sum(callback.docking_successes) / total_episodes
        if total_episodes > 0 else 0
    )
    print("\n=== COBRA EFFECT TRAINING RESULTS ===")
    print(f"Total episodes: {total_episodes}")
    print(f"Docking success rate: {success_rate:.1%}")
    print(f"(High reward, zero docking = reward "
          f"hacking confirmed)")
    return model, callback


if __name__ == "__main__":
    train()

You now have quantitative proof of reward hacking in a safety-critical domain. The agent is "succeeding" by every metric the reward function can see, while completely failing at the actual objective. Next, you will implement a constraint-shaped reward that makes this exploit impossible.

Train the Mitigated Agent and Visualize the Difference

Implement Three Mitigation Strategies and Run Ablations

The Cobra Effect emerged because the reward function measured progress (distance reduction per step) rather than outcome (successful docking). The agent discovered that staying alive near the target yields higher cumulative reward than actually completing the mission.

The fix mirrors how Anthropic addressed reward hacking in their research: change the framing so that exploitative behavior no longer correlates with reward. Your mitigated reward makes oscillation costly (per-step penalty in the terminal phase) and docking overwhelmingly rewarding (large sparse bonus of +100), while using constraint-compliance bonuses to guide the approach without creating new exploit surfaces.

In this step, get ready to:
  • Train PPO with the constraint-shaped mitigated reward and confirm docking success above 80%.
  • Generate side-by-side trajectory visualizations proving the behavioral difference between exploited and mitigated reward architectures.
Train PPO with the mitigated reward

The mitigated training script is nearly identical to the cobra effect script. The only change is the reward_type parameter passed to DockingEnv. Same algorithm, same hyperparameters, same seed.

  • Add the following code to experiments/train_mitigated.py:
"""Train PPO with the constraint-shaped mitigated reward."""

from stable_baselines3 import PPO

from goodharts_docking import DockingEnv
from experiments.train_cobra_effect import MetricsCallback


def train():
    # Same environment, different reward architecture
    env = DockingEnv(reward_type="mitigated")
    # Identical hyperparameters to cobra effect for fair comparison
    model = PPO("MlpPolicy", env, verbose=1, seed=42, device="cpu", learning_rate=3e-4, n_steps=2048, batch_size=64, n_epochs=10, gamma=0.99, gae_lambda=0.95, clip_range=0.2, policy_kwargs=dict(net_arch=[64, 64]))
    callback = MetricsCallback()
    model.learn(total_timesteps=100_000, callback=callback, progress_bar=True)
    model.save("experiments/ppo_mitigated")
    total_episodes = len(callback.docking_successes)
    success_rate = sum(callback.docking_successes) / total_episodes if total_episodes > 0 else 0
    print("\n=== MITIGATED REWARD TRAINING RESULTS ===")
    print(f"Total episodes: {total_episodes}")
    print(f"Docking success rate: {success_rate:.1%}")
    print(f"(Constraint-shaped reward eliminates exploit)")
    return model, callback


if __name__ == "__main__":
    train()

What does this code do?

This script reuses the MetricsCallback from the cobra effect training rather than redefining it. The only functional difference is reward_type="mitigated".

By holding everything else constant (same PPO hyperparameters, same seed, same timestep budget), you isolate the reward function as the single independent variable. Any behavioral difference between the two trained agents is caused entirely by the reward signal, not by training noise.

Before you run this, what do you predict will happen to the docking success rate? The cobra agent achieved 0%. Will changing only the reward function produce a fundamentally different outcome?

  • Train the mitigated agent by running:
python -m experiments.train_mitigated

Training takes approximately 30-45 seconds on an M2 MacBook Air. You should see the docking success rate reported above 80%.

Why does this work?

The mitigated reward eliminates the oscillation exploit through three structural changes. A per-step penalty of -0.5 in the terminal phase makes delay costly. A dominant docking bonus of +100 makes episode completion overwhelmingly preferable. Constraint-violation penalties (-2.0 for velocity, -1.0 for cone) guide safe approach behavior.

The agent can no longer profit from farming proximity. Oscillating near the target now accumulates penalties, while docking yields a reward larger than any feasible number of shaping steps.

🙋‍♀️ Seeing a docking success rate below 50%?

Confirm you passed reward_type="mitigated" to DockingEnv, not "hackable". Also verify that seed=42 is set for reproducibility. If you changed the MitigatedReward class in the previous step, re-check that the docking bonus is +100.0 and the per-step terminal penalty is -0.5.

Help me debug why my mitigated agent isn't learning to dock successfully.

Generate comparison visualizations

With both trained models saved, you can now produce the key result figures. The trajectory comparison plot is your primary evidence that identical training with different reward signals produces fundamentally different agent behavior.

  • Create experiments/visualize.py and add the file header with imports:
"""Generate publication-quality comparison visualizations."""

# Visualization and numerical libraries
import matplotlib.pyplot as plt
import matplotlib.patches as patches
import numpy as np
from stable_baselines3 import PPO

from goodharts_docking import DockingEnv
  • Below the imports, add the evaluation helper function:
def evaluate_agent(model, env, n_episodes=5):
    """Run deterministic evaluation episodes and collect trajectories."""
    trajectories = []
    metrics = []
    for _ in range(n_episodes):
        obs, info = env.reset()
        trajectory = [info["position"]]
        episode_violations = 0
        total_reward = 0
        docked = False
        while True:
            action, _ = model.predict(obs, deterministic=True)
            obs, reward, terminated, truncated, info = env.step(action)
            trajectory.append(info["position"])
            total_reward += reward
            if info["velocity_violation"]:
                episode_violations += 1
            if info["docked"]:
                docked = True
            if terminated or truncated:
                break
        trajectories.append(np.array(trajectory))
        metrics.append({"docked": docked, "violations": episode_violations, "reward": total_reward, "final_distance": info["distance_to_target"], "final_speed": info["speed"]})
    return trajectories, metrics

What does evaluate_agent do?

This function runs a trained model in deterministic mode (no exploration noise) for 5 episodes. It collects the (x, y) position at every timestep, producing trajectories you can plot. It also records whether the agent docked, how many velocity violations occurred, and the final distance and speed.

  • Below evaluate_agent, add the trajectory comparison plotting function:
def plot_trajectory_comparison(cobra_trajs, mitigated_trajs, save_path):
    """Side-by-side trajectory comparison plot."""
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6))
    for ax, trajs, title in [(ax1, cobra_trajs, "Cobra Effect (Hackable Reward)"), (ax2, mitigated_trajs, "Mitigated (Constraint-Shaped)")]:
        ax.set_xlim(-100, 600)
        ax.set_ylim(-200, 200)
        # Green dot at origin marks the docking target
        target = plt.Circle((0, 0), 1.0, color="green", fill=True, zorder=5)
        ax.add_patch(target)
        # Shaded approach cone constraint region
        ax.fill_between([0, 50], [-50 * np.tan(np.radians(20)), 0], [50 * np.tan(np.radians(20)), 0], alpha=0.1, color="blue", label="Approach Cone")
        for i, traj in enumerate(trajs):
            color = plt.cm.viridis(i / max(len(trajs) - 1, 1))
            ax.plot(traj[:, 0], traj[:, 1], color=color, alpha=0.7, linewidth=1.5)
            ax.scatter(traj[0, 0], traj[0, 1], color=color, marker="o", s=50, zorder=4)
            ax.scatter(traj[-1, 0], traj[-1, 1], color=color, marker="x", s=80, zorder=4)
        ax.set_xlabel("Downrange (m)")
        ax.set_ylabel("Crosstrack (m)")
        ax.set_title(title, fontsize=13, fontweight="bold")
        ax.axhline(y=0, color="gray", linestyle="--", alpha=0.3)
        ax.axvline(x=0, color="gray", linestyle="--", alpha=0.3)
        ax.grid(True, alpha=0.2)
    plt.tight_layout()
    plt.savefig(save_path, dpi=150, bbox_inches="tight")
    plt.close()
    print(f"Saved: {save_path}")

What does this plot show?

Each panel shows 5 evaluation trajectories (colored lines) with the docking target at the origin (green dot). Circle markers show starting positions. X markers show where each episode ended. The approach cone constraint is shaded in blue.

On the left panel (Cobra Effect), you will see trajectories that loop and oscillate near the target without converging to the origin. On the right panel (Mitigated), trajectories make a direct approach and terminate at the dock.

  • Below plot_trajectory_comparison, add the training curves function:
def plot_training_curves(cobra_callback, mitigated_callback, save_path):
    """Training reward and success rate comparison."""
    fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8), sharex=True)
    window = 20
    # Rolling average of docking success for each agent
    for callback, label, color in [(cobra_callback, "Hackable (Cobra Effect)", "red"), (mitigated_callback, "Mitigated", "blue")]:
        successes = np.array(callback.docking_successes, dtype=float)
        if len(successes) > window:
            rolling_success = np.convolve(successes, np.ones(window) / window, mode="valid")
            ax1.plot(rolling_success, label=label, color=color)
    ax1.set_ylabel("Docking Success Rate")
    ax1.set_ylim(-0.05, 1.05)
    ax1.legend()
    ax1.set_title("Training Progress Comparison", fontsize=13, fontweight="bold")
    ax1.grid(True, alpha=0.2)
    ax2.set_xlabel("Episode")
    ax2.set_ylabel("Docking Success (cumulative)")
    ax2.grid(True, alpha=0.2)
    plt.tight_layout()
    plt.savefig(save_path, dpi=150, bbox_inches="tight")
    plt.close()
    print(f"Saved: {save_path}")
  • Below plot_training_curves, add the main() entry point:
def main():
    """Load both trained models and generate comparison figures."""
    print("Loading trained models...")
    cobra_model = PPO.load("experiments/ppo_cobra_effect")
    mitigated_model = PPO.load("experiments/ppo_mitigated")
    print("Evaluating Cobra Effect agent...")
    cobra_env = DockingEnv(reward_type="hackable")
    cobra_trajs, cobra_metrics = evaluate_agent(cobra_model, cobra_env)
    print("Evaluating Mitigated agent...")
    mitigated_env = DockingEnv(reward_type="mitigated")
    mitigated_trajs, mitigated_metrics = evaluate_agent(mitigated_model, mitigated_env)
    print("\n=== EVALUATION SUMMARY ===")
    print(f"Cobra Effect - Docking rate: {sum(m['docked'] for m in cobra_metrics)}/{len(cobra_metrics)}")
    print(f"Mitigated    - Docking rate: {sum(m['docked'] for m in mitigated_metrics)}/{len(mitigated_metrics)}")
    # Generate the key result figure
    plot_trajectory_comparison(cobra_trajs, mitigated_trajs, "figures/trajectory_comparison.png")


if __name__ == "__main__":
    main()

What does main() do?

The main() function loads both saved models, runs 5 deterministic evaluation episodes for each, prints the docking rate comparison, and generates the trajectory comparison figure. The plot_training_curves function requires callback objects from training. You will use it in a moment to generate the training curves figure.

  • Save experiments/visualize.py.
  • Run the visualization script by executing:
python -m experiments.visualize

You should see output confirming the behavioral divergence:

=== EVALUATION SUMMARY ===
Cobra Effect - Docking rate: 0/5
Mitigated    - Docking rate: 4/5
Saved: figures/trajectory_comparison.png

The cobra agent docks 0 out of 5 times despite accumulating high reward during training. The mitigated agent docks 4-5 out of 5 times. Same algorithm, same hyperparameters, different reward signal, opposite outcomes.

Generate the training curves and constraint figures

The trajectory comparison proves the agents behave differently. The training curves figure proves the divergence emerged during training, and the constraint analysis breaks down safety violations by mission phase.

  • Generate the training curves figure by running:
python -c "
from experiments.train_cobra_effect import train as train_cobra
from experiments.train_mitigated import train as train_mitigated
from experiments.visualize import plot_training_curves
print('Re-training cobra agent for callback data...')
_, cobra_cb = train_cobra()
print('Re-training mitigated agent for callback data...')
_, mitigated_cb = train_mitigated()
plot_training_curves(cobra_cb, mitigated_cb, 'figures/training_curves.png')
"

Why re-train to generate training curves?

The plot_training_curves function needs the callback objects that hold per-episode success data. These only exist in memory during training. Since both trainings complete in under 60 seconds total on your M2, re-running is fast.

  • Generate the constraint violation analysis figure by running:
python -c "
import matplotlib.pyplot as plt, numpy as np
from stable_baselines3 import PPO
from goodharts_docking import DockingEnv

def count_violations(model, env, n=10):
    pv, ps = {0:0,1:0,2:0}, {0:0,1:0,2:0}
    for _ in range(n):
        obs, info = env.reset()
        while True:
            action, _ = model.predict(obs, deterministic=True)
            obs, _, term, trunc, info = env.step(action)
            ps[info['phase']] += 1
            if info['velocity_violation']: pv[info['phase']] += 1
            if term or trunc: break
    return {p: pv[p]/max(ps[p],1) for p in [0,1,2]}

cv = count_violations(PPO.load('experiments/ppo_cobra_effect'), DockingEnv(reward_type='hackable'))
mv = count_violations(PPO.load('experiments/ppo_mitigated'), DockingEnv(reward_type='mitigated'))
x = np.arange(3)
fig, ax = plt.subplots(figsize=(8, 5))
ax.bar(x - 0.2, [cv[i] for i in range(3)], 0.35, label='Cobra Effect', color='red', alpha=0.7)
ax.bar(x + 0.2, [mv[i] for i in range(3)], 0.35, label='Mitigated', color='blue', alpha=0.7)
ax.set_xticks(x); ax.set_xticklabels(['Far (>50m)', 'Close (5-50m)', 'Terminal (<5m)'])
ax.set_ylabel('Velocity Violation Rate')
ax.set_title('Constraint Violations by Mission Phase', fontweight='bold')
ax.legend(); ax.grid(True, alpha=0.2); plt.tight_layout()
plt.savefig('figures/constraint_analysis.png', dpi=150, bbox_inches='tight'); plt.close()
print('Saved: figures/constraint_analysis.png')
"
  • Confirm all three figures were generated by running:
ls figures/

You should see trajectory_comparison.png, training_curves.png, and constraint_analysis.png listed.

Missing figures or seeing import errors?

If you see a ModuleNotFoundError for goodharts_docking, make sure you are running from the goodharts-taxonomy/ root directory. If you see a FileNotFoundError for the model files, confirm that both experiments/ppo_cobra_effect.zip and experiments/ppo_mitigated.zip exist from the training steps.

Help me debug my visualization script errors.

  • Open figures/trajectory_comparison.png in Finder by running:
open figures/trajectory_comparison.png

The left panel shows the cobra agent's trajectories looping and oscillating within 10-20m of the target without ever converging to the dock. The right panel shows the mitigated agent making clean, direct approaches that terminate at the origin.

This is your key research figure

A reviewer scanning your repository will look at this plot first. It tells the entire story in one image: identical algorithms, different reward signals, opposite outcomes. The oscillation pattern on the left IS cobra farming.

The agent learned to approach-retreat-approach because each approach yields positive distance-reduction reward, and docking would end the episode.

✔️ Awesome, I've got everything!

Confirm you have saved both experiments/train_mitigated.py and experiments/visualize.py, and that all three figures exist in the figures/ directory.

ⓧ I'd like to double check the full code

Since both files are built across multiple chunks, scroll up and confirm each piece is present in order. For experiments/train_mitigated.py, you should have a single chunk containing the docstring, imports, train() function, and the if __name__ guard.

For experiments/visualize.py, confirm these pieces in order:

  • Docstring, imports (matplotlib, numpy, PPO, DockingEnv).
  • evaluate_agent() function.
  • plot_trajectory_comparison() function.
  • plot_training_curves() function.
  • main() function and if __name__ == "__main__" guard.

You now have reproducible experimental evidence that reward architecture determines agent behavior in safety-critical systems. Next, you will package these results into a publication-quality repository that frames your findings through the lens of historical incentive failures and modern alignment research.

Package as a Research-Grade Publication

Benchmark Across Algorithm Families: PPO vs SAC vs TD3

Your experiments are complete. You have quantitative evidence of reward hacking and a constraint-shaped mitigation that eliminates it.

The difference between a project folder and a research contribution is presentation. Anthropic fellows produce papers. Your repository needs to read like one so a reviewer can scan your README in 2 minutes and understand what you did, why it matters, and how it connects to their research agenda.

In this step, get ready to:
  • Write a paper-style README with abstract, methodology, results, and historical analysis.
  • Make the environment installable as a Python package via pip install -e ..
  • Push a publication-ready repository to GitHub with license, reproduction script, and clean commit history.
Write the paper-style README

A paper-structured README signals to reviewers that this is a research artifact. Each section maps to a standard academic paper component: abstract, methodology, results, discussion.

  • Open README.md in your editor and add the title and abstract:
# Goodhart's Taxonomy: Historical Incentive Failures as a Framework for Reward Misspecification in Safety-Critical RL

## Abstract

We demonstrate that reward misspecification in safety-critical continuous
control produces agent behaviors structurally identical to historically-
documented incentive failures. Using a custom Gymnasium environment
implementing Clohessy-Wiltshire relative orbital dynamics, we show that a
dense distance-reduction reward generates the "Cobra Effect": an agent that
maximizes cumulative reward while achieving 0% mission success. A constraint-
shaped mitigation eliminates the exploit. We connect these findings to reward
hacking in RLHF-trained language models and frame reward misspecification
through Goodhart's Law taxonomy.

What does the abstract accomplish?

The abstract gives a reviewer the full story in one paragraph: problem, method, key finding, and implication. This is the single most important section because many reviewers read only this before deciding whether to continue.

  • Below the abstract, add the Key Results section with inline figure references:
## Key Results

![Trajectory Comparison](figures/trajectory_comparison.png)

*Left: Cobra Effect agent oscillates near target, farming distance-reduction
reward without docking. Right: Mitigated agent approaches directly and docks
safely.*

![Training Curves](figures/training_curves.png)

*Hackable reward produces rising reward with 0% docking success. Mitigated
reward produces lower raw reward but >80% docking success.*
  • Save README.md and confirm the file is taking shape by running:
wc -l README.md

You should see approximately 20-25 lines. The abstract and figure references are in place.

  • Add the Motivation section below Key Results:
## Motivation

**Goodhart's Law**: "When a measure becomes a target, it ceases to be a good
measure." This project demonstrates that Goodhart's Law applies directly to
RL reward functions in safety-critical domains.

Dense per-step reward signals create the same perverse incentives that plagued
historical measurement systems. The agent optimizes the metric (reward) while
violating the objective (safe docking). This structural parallel connects
colonial-era policy failures to modern LLM alignment challenges documented in
Anthropic's "Natural Emergent Misalignment from Reward Hacking" (2025).
  • Add the Environment section documenting the CW dynamics and safety constraints:
## Environment: Spacecraft Proximity Operations

| Parameter | Value |
|-----------|-------|
| Dynamics | Clohessy-Wiltshire (2D in-plane) |
| Orbit | 400 km LEO circular (n = 0.00113 rad/s) |
| State | [x, y, vx, vy, fuel, phase] |
| Actions | Continuous 2D thrust [-1, 1] |
| Max thrust | 0.1 m/s squared |
| Fuel budget | 50 m/s delta-v |
| Dock criteria | distance < 1m AND speed < 0.05 m/s |
| Velocity corridor | speed < 0.1 * distance |
| Approach cone | +/-20 deg of +x axis (within 50m) |
| Episode length | 500 steps (1s each) |

Why include a parameters table?

The table lets a reviewer instantly verify that your environment is physically grounded, not a toy. Naming specific values (400 km orbit, 0.1 m/s squared thrust) shows you designed from real spacecraft constraints.

  • Save README.md and check its growth:
wc -l README.md

You should see approximately 50-60 lines now.

  • Add the Taxonomy of Failure Modes section with the Cobra Effect analysis:
## Taxonomy of Failure Modes

### The Cobra Effect (Reward Farming)

**Historical case (1902, British India):** The colonial government offered
bounties for dead cobras to reduce the population. Citizens began breeding
cobras to kill for the bounty. When the program was cancelled, breeders
released their now-worthless cobras, making the problem worse.

**RL manifestation:** Dense distance-reduction reward
(`(prev_distance - current_distance) * 10.0`) with a small terminal docking
bonus (+5.0). The agent discovers that oscillating near the target yields
higher cumulative reward than docking (which ends the episode and stops
reward collection). The per-step reward IS the bounty. The oscillation IS
cobra farming.
  • Add the LLM parallel that connects your finding to Anthropic's research:
**LLM parallel:** RLHF-trained models maximize per-token approval scores by
producing verbose, hedging outputs rather than concise helpful ones. The
structural parallel is identical: per-unit-progress rewards incentivize
maintaining the rewarded state rather than achieving the terminal objective.
Anthropic's November 2025 paper demonstrates this at scale.

Why the three-part structure for each failure mode?

Each failure mode follows historical case, RL manifestation, LLM parallel. This structure proves you understand the general pattern across domains, not just one isolated demo. It connects your work directly to the broader alignment research agenda.

  • Add the Methodology and Results sections:
## Methodology

- **Algorithm:** PPO (Stable-Baselines3 2.9.0)
- **Policy:** MlpPolicy, net_arch=[64, 64]
- **Timesteps:** 100,000 per experiment
- **Seed:** 42 (deterministic)
- **Device:** CPU
- **Evaluation:** 5 deterministic episodes post-training

## Results

| Metric | Hackable (Cobra Effect) | Mitigated |
|--------|------------------------|-----------|
| Docking Success Rate | 0% | >80% |
| Mean Final Distance | ~12m | <1m |
| Velocity Violations | ~23% of steps | <5% of steps |
| Mean Fuel Used | ~35/50 m/s | ~25/50 m/s |

![Constraint Analysis](figures/constraint_analysis.png)
  • Save README.md and check the line count:
wc -l README.md

You should see approximately 95-105 lines now. The core research content is complete.

  • Add the Discussion section that names the mechanism and connects to Anthropic's approach:
## Discussion

The Cobra Effect emerges from the interaction between episode termination
and per-step reward. Docking terminates the episode, stopping reward
collection. The agent learns that proximity is more valuable than completion.
This mirrors how RLHF reward models can incentivize verbosity over
helpfulness when the reward signal correlates with output length.

The constraint-shaped mitigation works by making oscillation costly and
docking overwhelmingly rewarding. This parallels Anthropic's approach of
reshaping reward signals so exploitative behaviors no longer correlate with
reward.
  • Add the Reproducing Results, Installation, and Citation sections:
## Reproducing Results

```bash
git clone <repo-url>
cd goodharts-taxonomy
python -m venv .venv && source .venv/bin/activate
pip install -e .
bash reproduce.sh
```

## Installation

```bash
pip install -e .
```

```python
import gymnasium
env = gymnasium.make("GoodhartsDocking-v0")
```

## Citation

```bibtex
@software{goodharts_taxonomy,
  title={Goodhart's Taxonomy},
  year={2025},
  url={https://github.com/YOUR_USERNAME/goodharts-taxonomy}
}
```

What does the Discussion accomplish?

The Discussion names the mechanism (termination-reward interaction), identifies the LLM parallel (RLHF verbosity), and points at the fix pattern (reshaping rather than constraining). This connects your specific finding to the broader alignment agenda.

  • Save README.md and verify the final line count:
wc -l README.md

You should see approximately 120-130 lines. Your README now reads like a workshop paper submission.

Figures not rendering in your editor preview?

The image paths use relative references like figures/trajectory_comparison.png. These render on GitHub but may not show in all local markdown previewers. Confirm the figure files exist by running ls figures/ and checking all three PNGs are listed.

Help me debug README image paths

Make the environment installable as a package

Other researchers need to install your environment with a single command. A pyproject.toml declares package metadata and registers the Gymnasium entry point so any script can discover your environment.

  • Create pyproject.toml in the root of your goodharts-taxonomy folder with the following content:
[build-system]
requires = ["setuptools>=68.0"]
build-backend = "setuptools.build_meta"

[project]
name = "goodharts-docking"
version = "0.1.0"
description = "Historical incentive failures as a framework for reward misspecification in safety-critical RL"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10"
dependencies = [
    "gymnasium==1.3.0",
    "stable-baselines3==2.9.0",
    "matplotlib",
    "numpy",
]

[project.entry-points."gymnasium.envs.__root__"]
GoodhartsDocking-v0 = "goodharts_docking:DockingEnv"

What does this configuration do?

The [build-system] section tells pip how to build your package. The [project] section declares metadata and pins dependency versions so another researcher gets identical packages.

The entry-points section registers your environment with Gymnasium's discovery system. After installation, any script can call gymnasium.make("GoodhartsDocking-v0") without importing your package first.

  • Install the package in editable mode by running:
pip install -e .
  • Verify the package installs and the entry point registers by running:
python -c "import gymnasium; env = gymnasium.make('GoodhartsDocking-v0'); print('Entry point registered:', env.spec.id)"

You should see Entry point registered: GoodhartsDocking-v0 confirming the package is discoverable.

ModuleNotFoundError on import?

Make sure you ran pip install -e . from inside the goodharts-taxonomy directory (the folder containing pyproject.toml). Also confirm your virtual environment is active. You should see (.venv) in your terminal prompt.

Help me debug my package install

Add license, reproduction script, and push to GitHub

A research repository without a license is legally unusable by other researchers. The reproduction script lets a reviewer replicate your entire pipeline with one command.

  • Create the MIT license file by running:
cat > LICENSE << 'EOF'
MIT License

Copyright (c) 2025

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
EOF
  • Create the reproduction script that runs the full experiment pipeline:
cat > reproduce.sh << 'EOF'
#!/bin/bash
set -e
python experiments/train_cobra_effect.py
python experiments/train_mitigated.py
python experiments/visualize.py
EOF
chmod +x reproduce.sh

What does reproduce.sh do?

The set -e flag stops execution if any script fails. A reviewer clones your repo, runs bash reproduce.sh, and gets identical trained models and figures without reading any documentation.

  • Verify both new files exist by running:
ls LICENSE reproduce.sh

You should see both files listed in the output.

  • Stage all files and create a clean initial commit by running:
git add -A
git commit -m "Initial commit: Goodhart's Taxonomy research artifact"
  • Create a new repository on GitHub named goodharts-taxonomy:
  • Open github.com/new in your browser.
  • Enter goodharts-taxonomy as the repository name.
  • Set the description to: Demonstrating how historically-documented incentive failures manifest as reward hacking in safety-critical RL. A parameterizable spacecraft docking environment for studying reward misspecification.
  • Select Public visibility.
  • Leave all initialization options unchecked (no README, no .gitignore, no license).
  • Click Create repository.
  • Push your local repository to GitHub by running:
git remote add origin https://github.com/[[GITHUB_USERNAME="your GitHub username"]]/goodharts-taxonomy.git
git branch -M main
git push -u origin main

Before you check the result, what do you expect to see when you open your GitHub repository page?

  • Open your repository URL in the browser.

Your README renders with the full paper structure: abstract, inline trajectory comparison figure, motivation, environment parameters table, taxonomy section with the Cobra Effect analysis, results table, and reproduction instructions.

  • As a final verification, confirm the full toolchain works by running:
python -c "from stable_baselines3 import PPO; import gymnasium; print('SB3:', PPO.__module__); print('Gym:', gymnasium.__version__)"

You should see version confirmations for both libraries with no errors. Your repository is now publication-ready.

GitHub not rendering the figures?

Check that the figures/ directory was included in your commit. Run git status to confirm no untracked PNG files remain. If they were missed, stage them with git add figures/, amend the commit, and force-push.

Help me fix my GitHub figures

Your research artifact is live. The repository reads like a workshop submission, the environment is installable by any researcher, and the results are reproducible with a single command.

Run the Hyperparameter Sensitivity Study

You have proven reward hacking exists across algorithm families. The next question a thesis committee asks is: where exactly does the exploit emerge? What are the boundary conditions?

A sensitivity study systematically varies reward function coefficients (distance shaping weight, terminal bonus magnitude, per-step penalty) and training hyperparameters (gamma, episode length) to map the exact thresholds where exploit behavior transitions from absent to dominant. The result is a set of heatmaps showing exploit emergence boundaries.

In this step, get ready to:
  • Design a factorial experiment grid covering reward coefficients and discount factors.
  • Run the full grid (100+ experiment configurations) with W&B tracking and multi-seed averaging.
  • Generate heatmaps showing the exact coefficient boundaries where reward hacking emerges.

Generate Publication Figures and Statistical Analysis

Raw experimental results become a research contribution only when they pass statistical rigor and are presented in publication-quality figures. A thesis committee expects Mann-Whitney U tests, confidence intervals, effect sizes, and figures that meet journal submission standards.

In this step, you will build the statistical analysis pipeline that transforms your W&B experiment logs into formal hypothesis tests, generate LaTeX-quality figures with proper axis labels and legend placement, and compile a complete research paper PDF using pyLaTeX that ties together your entire campaign.

In this step, get ready to:
  • Run Mann-Whitney U tests comparing docking success rates across all experimental conditions with Bonferroni correction.
  • Generate 12+ publication-quality figures including trajectory comparisons, training curves, sensitivity heatmaps, and algorithm comparison box plots.
  • Compile a complete LaTeX research paper with abstract, methodology, results, discussion, and bibliography.

Package as a Reproducible Research Artifact

Your experimental campaign is complete. You have multi-algorithm benchmarks, sensitivity studies, and statistical analysis all producing reproducible results. The final step transforms this from a local project folder into a publication-ready research artifact.

A thesis artifact needs a pip-installable package, a single-command reproduction script that regenerates all results from scratch, a paper-structured README connecting your findings to the Goodhart's Law taxonomy and Anthropic's alignment research, and a clean GitHub repository with proper CI, licensing, and documentation.

In this step, get ready to:
  • Make the environment installable as a Python package with Gymnasium entry-point registration.
  • Write a paper-structured README with abstract, methodology, results table, and Goodhart's Law taxonomy analysis.
  • Push a publication-ready repository to GitHub with license, CI, and a full reproduction pipeline.

Secret mission

Cross-Domain Generalization: Transfer the Taxonomy to Autonomous Driving

Your taxonomy currently proves reward hacking in one physical domain. A single domain is a case study. Two structurally different domains prove you have a framework that generalizes. Build an autonomous lane-keeping environment, reproduce the Cobra Effect and Soviet Quota Gaming in it, and produce a cross-domain comparison table proving your taxonomy is domain-independent. This transforms your work from a spacecraft-specific demonstration into a general theory of reward misspecification.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with a remote GitHub repository, so there are no ongoing costs.

Resources you used:

  • Local project folder goodharts-taxonomy/ on your Desktop (source code, virtual environment, trained models, figures).
  • GitHub repository goodharts-taxonomy (remote copy of your code and README).

Keep everything running

No action needed. Choose this if you plan to extend the taxonomy with additional failure modes, refine hyperparameters, or share the repository with reviewers.

  • Your virtual environment stays in goodharts-taxonomy/.venv/ and activates whenever you return to the project.
  • Trained models in experiments/ are ready for further evaluation or visualization.
  • Your GitHub repository remains public and citable.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your shell session. All files stay on disk for when you return.

  • Deactivate the virtual environment by running:
deactivate
  • When you return, navigate back to the project and reactivate:
cd ~/Desktop/goodharts-taxonomy
source .venv/bin/activate

Delete - I don't want to use this again

Remove all local project files and the remote GitHub repository. This cannot be undone.

  • Deactivate the virtual environment first (if still active):
deactivate
  • Delete the entire project folder by running:
rm -rf ~/Desktop/goodharts-taxonomy
  • Confirm the folder is gone by running:
ls ~/Desktop | grep goodharts

You should see no output, confirming the folder is deleted.

  • Delete the GitHub repository:
  • Open your repository page on GitHub.
  • Click Settings in the top navigation bar.
  • Scroll to the bottom of the page to the Danger Zone section.
  • Click Delete this repository and follow the confirmation prompts.

Nice Work!

Nice Work!

You just executed a multi-day research campaign that produced a thesis-chapter-scale artifact. Your repository contains a 6-DOF orbital mechanics environment, multi-algorithm benchmarks with statistical significance testing, hyperparameter sensitivity heatmaps, and a compiled LaTeX paper connecting reward hacking to historical incentive failures through Goodhart's Law taxonomy.

You've learned how to:

  • Build a custom Gymnasium environment implementing Clohessy-Wiltshire orbital dynamics with continuous thrust control and physically meaningful safety constraints (velocity corridor, approach cone, fuel budget).
  • Demonstrate the Cobra Effect emerging naturally from a dense reward function in a safety-critical control domain, then eliminate it with a constraint-shaped mitigation that produces safe docking behavior.
  • Produce a publication-quality GitHub repository with a paper-structured README connecting RL reward hacking to historical incentive failures and modern LLM alignment challenges, installable by other researchers via pip install.
  • Secret Mission: Added Soviet Quota Gaming as a second taxonomized failure mode, proving the framework generalizes across structurally different exploit categories.

Ready to quiz yourself?