Build an RL Task for Coding Agents
Design, containerize, and grade a coding challenge for AI agents.
Introduction
30 Second Summary
Companies building AI coding agents need to know if those agents can actually write good code. They test them with structured challenges that score the agent's output automatically.
In this project, you will build a complete RL task package that tests a coding agent's ability to make coordinated changes across multiple files, structured exactly like the tasks Mechanize sells to AI labs.
What You'll Build
A self-contained, containerized task package containing a multi-file Python application (the environment), a natural language prompt, and an automated grading system that scores a coding agent's work.
By the end of this project, you'll have:
- A containerized RL task with a prompt, environment, and automated grader structured like the ones Mechanize engineers build daily.
- Firsthand experience directing a coding agent (Codex) to build software and evaluating its output for correctness.
- A documented failure analysis showing where a frontier coding agent struggles with your task and why.
- Secret Mission: Design a second task targeting a different agent weakness (state management) and compare failure patterns.
Are there any prerequisites?
You'll need Docker installed, access to OpenAI Codex, and Google Antigravity IDE installed on your machine. Step 1 walks you through verifying each of these.
Set Up the Project Structure
You've got your tools installed and understand how a Mechanize RL task is structured. Now it's time to create the actual project layout that mirrors what Mechanize ships to AI labs.
You'll use Codex to scaffold this directory structure. This mirrors the real Mechanize workflow: engineers direct coding agents to handle routine tasks, then review what the agent produces.
In this step, get ready to:
- Create the project directory structure using Codex.
- Review and understand the purpose of each file and directory.
Scaffold the project directory with Codex
A Mechanize task package always follows the same layout: an environment directory for the codebase, a grader directory for tests, a prompt file, a scoring script, and a Dockerfile to containerize everything. Instead of creating these manually, you'll prompt Codex to do it for you.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Antigravity and press Enter to open Google Antigravity IDE.
What is Google Antigravity IDE?
Google Antigravity is an agent-first development platform. It combines a familiar code editor (the Editor View) with a dedicated agent interface (the Manager Surface) where you can spawn and observe coding agents working on tasks.
It supports multiple AI models including OpenAI's GPT-OSS. In this project, you'll use it to interact with Codex for scaffolding and code generation.
- Open the integrated terminal by pressing Ctrl+` (macOS and Windows).
- Move to your Desktop by running this command:
cd ~/Desktop- Open the Chat panel by pressing Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), then type Open Chat and select it.
- In the Chat panel, enter the following prompt to direct Codex to create the project structure:
Create a project directory called mechanize-task on my Desktop with this structure:
- mechanize-task/environment/ (empty directory)
- mechanize-task/grader/ (empty directory)
- mechanize-task/prompt.md (empty file)
- mechanize-task/Dockerfile (empty file)
- mechanize-task/score.py (empty file)
Use mkdir and touch commands in the terminal. Do not add any content to the files.What does this prompt do?
You're directing Codex to run terminal commands that create the folder layout. The agent will execute mkdir to create directories and touch to create empty placeholder files.
This is how Mechanize engineers work daily. They describe what they want, the agent executes it, and they review the result.
- Review the commands Codex proposes. If the agent asks for permission to run terminal commands, click Run or approve the execution.
- Verify the structure was created by running these commands:
cd ~/Desktop/mechanize-task
lsYou should see Dockerfile, environment, grader, prompt.md, and score.py in the output.
Don't see all the files?
If Codex created a different structure or missed some files, you can create the missing pieces manually. Run any of these commands as needed:
mkdir -p environment grader
touch prompt.md Dockerfile score.pyAgents occasionally interpret prompts differently. Reviewing and correcting agent output is part of the Mechanize workflow. Help me troubleshoot my project structure
Review the project layout
Now that the structure exists, take a moment to understand what each piece is for. Knowing the purpose of every component is critical because you'll be designing a challenge that depends on how these pieces connect.
- Run this command to see the full directory tree:
find . -type f -o -type d | sortYou should see the complete layout of your mechanize-task directory with all files and subdirectories listed.
What is each piece for?
- environment/ holds the codebase that an agent gets dropped into. This is the "world" the agent works in.
- grader/ contains the pytest suite that scores the agent's work automatically.
- prompt.md is the natural language task description you hand to the agent. It describes what feature to implement without specifying how.
- score.py runs the grader and outputs a numerical score (0.0 to 1.0). This is the reward signal in RL terms.
- Dockerfile packages everything into a container so the task is reproducible on any machine.
Why containerize the task?
Docker containers ensure that every agent evaluation runs in an identical environment. Without containerization, differences in system Python versions or installed packages could make grading inconsistent.
Mechanize bundles every task into a container so AI labs can run thousands of evaluations reliably.
Your project skeleton is ready. Next up, you'll fill the environment/ directory with a multi-file Python app that becomes the codebase agents will work in.
Build the Environment
Your project structure is scaffolded and ready. Now it's time to build the actual codebase that a coding agent will be dropped into.
In a Mechanize task, the "environment" is a real, working application. Agents don't get toy single-file scripts. They get codebases with multiple files, dependencies between those files, and established patterns they need to follow. You'll direct Codex to build this codebase, then review it the same way a Mechanize engineer reviews agent output.
In this step, get ready to:
- Use Codex to generate a multi-file Python task tracker inside environment/.
- Run the app locally to verify it works end-to-end.
- Identify the cross-file gap that will become your challenge for agents.
Use Codex to generate the task tracker app
You need a multi-file Python app that has clear structure and real data flow between files. This is the codebase an agent will later be asked to modify.
- In Google Antigravity IDE, activate Agent Mode by pressing Cmd+Shift+A (macOS) or Ctrl+Shift+A (Windows/Linux).
- Give Codex the following prompt to create your task tracker application:
Create a multi-file Python task tracker app inside the environment/ directory.
The app needs three files:
1. models.py - A Task dataclass with fields: title (str), priority (using a
Priority IntEnum with LOW=1, MEDIUM=2, HIGH=3), status (using a Status
class with TODO, IN_PROGRESS, DONE constants), and task_id (auto-generated
8-character hex string from uuid4).
2. storage.py - A TaskStore class with an internal _tasks list and methods:
add(task), get(task_id), list_all(), and delete(task_id).
3. app.py - A CLI interface with a menu loop offering options:
1. Add task (prompts for title and priority)
2. List tasks (shows all in insertion order)
3. Delete task (by ID)
4. Quit
Use only the standard library. No external dependencies.Why give Codex such a detailed prompt?
Mechanize engineers spend significant time crafting precise prompts for coding agents. A vague prompt like "make a task app" would produce code that's too simple or structured differently than what you need. The specificity here mirrors the level of detail you'd use when working with agents professionally.
After Codex finishes generating, open each file in the environment/ directory and review them. Your three files should match the structure below.
Open environment/models.py. This defines the data structures your app works with:
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Optional
import uuid
# Priority levels as integers for easy sorting later
class Priority(IntEnum):
LOW = 1
MEDIUM = 2
HIGH = 3
# Status constants for task state
class Status:
TODO = "todo"
IN_PROGRESS = "in_progress"
DONE = "done"
# The core Task data structure
@dataclass
class Task:
title: str
priority: Priority = Priority.MEDIUM
status: str = Status.TODO
task_id: str = field(default_factory=lambda: uuid.uuid4().hex[:8])What does this file do?
Priority is an IntEnum so each level has a numeric value (1, 2, 3). This makes priorities comparable and sortable. The Task dataclass auto-generates an 8-character hex ID so each task is uniquely addressable.
Notice that Status is a plain class with string constants, not an Enum. This is a deliberate design choice in the codebase that an agent will need to respect when making changes.
Open environment/storage.py. This is the data layer that manages the task list:
from models import Task
from typing import List, Optional
class TaskStore:
def __init__(self):
# In-memory list that holds all tasks
self._tasks: List[Task] = []
def add(self, task: Task) -> None:
self._tasks.append(task)
def get(self, task_id: str) -> Optional[Task]:
for task in self._tasks:
if task.task_id == task_id:
return task
return None
def list_all(self) -> List[Task]:
# Returns a copy of the list in insertion order
return list(self._tasks)
def delete(self, task_id: str) -> bool:
for i, task in enumerate(self._tasks):
if task.task_id == task_id:
self._tasks.pop(i)
return True
return FalseWhat does this file do?
TaskStore wraps a plain Python list with CRUD operations. The list_all() method returns tasks in insertion order. There is no sorting capability yet.
This is the file that will need the most significant changes when you design the task prompt later. An agent will need to add a new method here without breaking the existing ones.
Open environment/app.py. This is the CLI interface that ties the other files together:
import sys
from models import Task, Priority
from storage import TaskStore
def main():
store = TaskStore()
while True:
print("\n--- Task Tracker ---")
print("1. Add task")
print("2. List tasks")
print("3. Delete task")
print("4. Quit")
choice = input("Choose an option: ").strip()
if choice == "1":
title = input("Task title: ").strip()
priority_input = input("Priority (low/medium/high) [medium]: ").strip().lower()
priority_map = {"low": Priority.LOW, "medium": Priority.MEDIUM, "high": Priority.HIGH}
priority = priority_map.get(priority_input, Priority.MEDIUM)
task = Task(title=title, priority=priority)
store.add(task)
print(f"Added task: {task.title} (id: {task.task_id})")
elif choice == "2":
tasks = store.list_all()
if not tasks:
print("No tasks.")
else:
for t in tasks:
print(f" [{t.task_id}] {t.title} | priority={t.priority.name} | status={t.status}")
elif choice == "3":
task_id = input("Task ID to delete: ").strip()
if store.delete(task_id):
print("Deleted.")
else:
print("Task not found.")
elif choice == "4":
print("Goodbye!")
sys.exit(0)
else:
print("Invalid choice.")
if __name__ == "__main__":
main()What does this file do?
app.py imports from both models.py and storage.py, creating a dependency chain across all three files. The menu loop handles user input, creates Task objects, and passes them to TaskStore.
Option 2 calls store.list_all() which returns tasks in insertion order only. Later, your task prompt will ask an agent to add a fifth menu option for sorted listing. That requires changes here AND in storage.py.
Codex output doesn't match exactly?
Codex may produce slightly different code each time. The key elements to verify: Priority uses IntEnum with values 1, 2, 3. TaskStore has add, get, list_all, and delete methods. The CLI has four menu options. If your output differs significantly, replace the file contents with the reference code above.
✔️ Awesome, I've got everything!
Great. Make sure all three files are saved in your environment/ directory.
ⓧ I'd like to double check the full code
Here are all three files for reference. Make sure your environment/ directory contains these exact files:
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Optional
import uuid
class Priority(IntEnum):
LOW = 1
MEDIUM = 2
HIGH = 3
class Status:
TODO = "todo"
IN_PROGRESS = "in_progress"
DONE = "done"
@dataclass
class Task:
title: str
priority: Priority = Priority.MEDIUM
status: str = Status.TODO
task_id: str = field(default_factory=lambda: uuid.uuid4().hex[:8])from models import Task
from typing import List, Optional
class TaskStore:
def __init__(self):
self._tasks: List[Task] = []
def add(self, task: Task) -> None:
self._tasks.append(task)
def get(self, task_id: str) -> Optional[Task]:
for task in self._tasks:
if task.task_id == task_id:
return task
return None
def list_all(self) -> List[Task]:
return list(self._tasks)
def delete(self, task_id: str) -> bool:
for i, task in enumerate(self._tasks):
if task.task_id == task_id:
self._tasks.pop(i)
return True
return Falseimport sys
from models import Task, Priority
from storage import TaskStore
def main():
store = TaskStore()
while True:
print("\n--- Task Tracker ---")
print("1. Add task")
print("2. List tasks")
print("3. Delete task")
print("4. Quit")
choice = input("Choose an option: ").strip()
if choice == "1":
title = input("Task title: ").strip()
priority_input = input("Priority (low/medium/high) [medium]: ").strip().lower()
priority_map = {"low": Priority.LOW, "medium": Priority.MEDIUM, "high": Priority.HIGH}
priority = priority_map.get(priority_input, Priority.MEDIUM)
task = Task(title=title, priority=priority)
store.add(task)
print(f"Added task: {task.title} (id: {task.task_id})")
elif choice == "2":
tasks = store.list_all()
if not tasks:
print("No tasks.")
else:
for t in tasks:
print(f" [{t.task_id}] {t.title} | priority={t.priority.name} | status={t.status}")
elif choice == "3":
task_id = input("Task ID to delete: ").strip()
if store.delete(task_id):
print("Deleted.")
else:
print("Task not found.")
elif choice == "4":
print("Goodbye!")
sys.exit(0)
else:
print("Invalid choice.")
if __name__ == "__main__":
main()Run the app locally to verify it works
Before you can design a challenge around this codebase, you need to confirm it actually runs. You also need to understand how it behaves firsthand.
- Open a terminal in Google Antigravity IDE by pressing Ctrl+` (backtick).
- Run the task tracker by executing this command:
python3 environment/app.py- The app displays a menu. Choose option 1 to add a task.
- Enter Write docs as the title.
- Enter low for the priority.
- Add a second task with title Fix login bug and priority high.
- Add a third task with title Update deps and priority medium.
- Choose option 2 to list all tasks.
You should see all three tasks listed in the order you added them, not sorted by priority.
What should I see?
Your terminal output should look something like this (IDs will differ):
[a3f2c1e8] Write docs | priority=LOW | status=todo
[7b9d4e2a] Fix login bug | priority=HIGH | status=todo
[1c8e5f3b] Update deps | priority=MEDIUM | status=todoThe tasks appear in insertion order. "Write docs" is first because you added it first, even though it has the lowest priority.
- Choose option 4 to quit the app.
Seeing a ModuleNotFoundError?
If you see ModuleNotFoundError: No module named 'models', the command is running from the wrong directory. The app uses relative imports, so python3 needs to be invoked from inside environment/ or with the file path as shown above.
Try running from your project root with the full path: python3 environment/app.py. If that still fails, cd into the environment/ directory and run python3 app.py instead.
Help me fix my ModuleNotFoundError when running the task tracker app.
Identify the gap you will exploit
Now that you've seen the app in action, look at it through the lens of a task designer. Your job is to find a feature gap that requires coordinated changes across multiple files.
- Notice that option 2 ("List tasks") shows tasks in insertion order only. There is no way to view tasks sorted by priority.
- Think about what adding priority-based sorting would require.
Why cross-file coordination makes a good task
Adding priority-based sorting requires changes in ALL THREE files. The storage.py file needs a new list_sorted() method. The app.py file needs a new menu option that calls it. The models.py file already has Priority as an IntEnum (which enables sorting), but an agent might not realize that and try to add redundant sorting logic.
Single-file changes are too easy for today's coding agents. Mechanize explicitly warns that "the most common failure mode for new hires is to underestimate what today's coding agents are already capable of." Cross-file coordination is a known weakness because agents often lose context when switching between files.
- Make a mental note of these three change points. In the next step, you'll write a prompt that asks an agent to implement this feature without spelling out which files to touch.
Your environment is built and verified. Next up, you'll write the task prompt that will challenge a coding agent to modify this codebase.
Write the Task Prompt
You have a working task tracker codebase and you know exactly where it falls short: tasks only display in insertion order. Now you shift from building software to designing a challenge around that software.
This is the creative design work that Mechanize engineers do daily. You will write the natural language prompt that tells a coding agent what feature to implement, without spelling out how to implement it.
In this step, get ready to:
- Write a task prompt describing a priority-sorting feature.
- Review the prompt for ambiguity using a theory-of-mind lens.
- Add edge cases that raise the difficulty for coding agents.
Write your task prompt from scratch
Mechanize engineers write prompts by hand, not with AI. The prompt defines what the agent must accomplish while deliberately avoiding implementation details.
Why write this by hand?
If you let Codex write the prompt, it might leak implementation hints that make the task trivially easy for another agent. Writing by hand forces you to think about what behavior to specify versus what to leave unspecified.
This is the skill Mechanize interviews for: designing challenges that genuinely test agent capabilities.
- Create a new file called prompt.md in the root of your mechanize-task folder (alongside the environment/ directory).
- Paste the following content into prompt.md:
# Task: Add Priority-Based Sorting to the Task Tracker
## Overview
The task tracker application currently lists tasks in the order they were added.
Your job is to add the ability to list tasks sorted by priority (highest first).
## Requirements
1. Add a `list_sorted` method to `TaskStore` in `storage.py` that returns tasks
sorted by priority from highest (3) to lowest (1).
2. When two tasks have the same priority level, they must maintain their original
insertion order (stable sort).
3. Add a new menu option "5. List tasks by priority" in `app.py` that calls this
sorted listing.
4. The existing "2. List tasks" option must continue to work exactly as before
(insertion order).
## Expected Behavior
Given tasks added in this order:
- "Write docs" (LOW)
- "Fix login bug" (HIGH)
- "Update deps" (MEDIUM)
- "Add tests" (HIGH)
Option 2 (List tasks) should show them in insertion order:
1. Write docs
2. Fix login bug
3. Update deps
4. Add tests
Option 5 (List tasks by priority) should show:
1. Fix login bug (HIGH)
2. Add tests (HIGH)
3. Update deps (MEDIUM)
4. Write docs (LOW)
Note: "Fix login bug" appears before "Add tests" because it was added first
(stable sort within same priority).
## Constraints
- Do not modify the Task dataclass fields in models.py
- Do not change the behavior of existing menu options
- Do not use any external libraries beyond the standard libraryWhat does each section do?
- Overview tells the agent the current limitation and what to add.
- Requirements specifies the technical deliverables (method names, file locations) without saying how to implement them.
- Expected Behavior gives concrete input/output examples so the grader can verify correctness.
- Constraints defines what the agent must NOT break, preventing easy shortcuts that bypass the real challenge.
- Save prompt.md.
- Confirm the file exists by running:
ls prompt.mdYou should see prompt.md listed in the output.
File not found?
- Make sure you created the file inside the mechanize-task folder, not inside environment/.
- Check that your terminal's working directory is mechanize-task by running pwd.
Why can't I see my prompt.md file?
✔️ Awesome, I've got everything!
Great. Double check you have saved prompt.md before moving on.
ⓧ I'd like to double check the full code
# Task: Add Priority-Based Sorting to the Task Tracker
## Overview
The task tracker application currently lists tasks in the order they were added.
Your job is to add the ability to list tasks sorted by priority (highest first).
## Requirements
1. Add a `list_sorted` method to `TaskStore` in `storage.py` that returns tasks
sorted by priority from highest (3) to lowest (1).
2. When two tasks have the same priority level, they must maintain their original
insertion order (stable sort).
3. Add a new menu option "5. List tasks by priority" in `app.py` that calls this
sorted listing.
4. The existing "2. List tasks" option must continue to work exactly as before
(insertion order).
## Expected Behavior
Given tasks added in this order:
- "Write docs" (LOW)
- "Fix login bug" (HIGH)
- "Update deps" (MEDIUM)
- "Add tests" (HIGH)
Option 2 (List tasks) should show them in insertion order:
1. Write docs
2. Fix login bug
3. Update deps
4. Add tests
Option 5 (List tasks by priority) should show:
1. Fix login bug (HIGH)
2. Add tests (HIGH)
3. Update deps (MEDIUM)
4. Write docs (LOW)
Note: "Fix login bug" appears before "Add tests" because it was added first
(stable sort within same priority).
## Constraints
- Do not modify the Task dataclass fields in models.py
- Do not change the behavior of existing menu options
- Do not use any external libraries beyond the standard libraryReview your prompt for ambiguity
Mechanize engineers review every prompt with a "theory of mind" lens. You put yourself in the agent's position and ask: how could this be misread?
- Read through your prompt.md from top to bottom as if you were a coding agent seeing this codebase for the first time.
- Ask yourself these questions for each requirement:
- Could an agent satisfy this requirement with a shortcut that technically passes but misses the intent?
- Is there any ambiguity about which file to edit or which method to create?
- Would an agent know to preserve existing functionality, or does the prompt need to say so explicitly?
What is theory-of-mind prompting?
Theory of mind means modelling what another entity knows, believes, and might misunderstand. When you review a task prompt this way, you anticipate how an agent might misinterpret vague language.
For example, "sort tasks by priority" without specifying direction could mean lowest-first or highest-first. Your prompt says "highest first" explicitly to close that ambiguity.
- Tighten any wording you find ambiguous. If you used the reference prompt above, notice that it already specifies "highest (3) to lowest (1)" and names exact files.
Verify your edge cases increase difficulty
A prompt that is too easy won't reveal anything about an agent's capabilities. Mechanize's guidance is direct: "Take what you were initially planning to ask the agent to do, make it ten times harder, and you'll probably have a good task idea."
- Check that your prompt includes the stable sort requirement (Requirement 2 in the reference).
- Verify you have the concrete example in Expected Behavior showing that "Fix login bug" appears before "Add tests" because it was added first.
- Confirm the Constraints section explicitly forbids modifying the Task dataclass fields.
Why does stable sort matter as an edge case?
Python's built-in sorted() is guaranteed stable. An agent that uses sorted() with the right key gets stability for free.
However, agents often implement custom sorting algorithms or mutate the internal list in place. Both approaches commonly break insertion-order stability. This subtlety separates trivial tasks from ones that actually test agent capabilities.
Why require cross-file coordination?
Your prompt requires changes in storage.py (new method), app.py (new menu option), and understanding of models.py (Priority enum values). Single-file changes are too easy for today's frontier models.
Cross-file coordination is a known weakness because agents must maintain consistency across multiple files simultaneously. This is exactly what Mechanize means by "underestimate what today's coding agents are already capable of."
Your prompt is locked and loaded. Next up, you will build the automated grader that scores whatever an agent produces when it reads this prompt.
Build the Automated Grader
Your prompt is polished and ready to challenge an agent. Now you need the third piece of the Mechanize task anatomy: the automated grader.
The grader is what makes an RL task useful at scale. Without it, a human would need to read every agent submission and judge correctness manually. Your grader uses pytest to check the agent's work, and a scoring script converts test results into a single number between 0.0 and 1.0.
In this step, get ready to:
- Write a pytest suite that tests whether the priority-sorting feature works correctly.
- Create a scoring script that runs the tests and outputs a numerical reward signal.
- Containerize the entire task package with Docker.
Direct Codex to write the pytest suite
The grader needs to test whether an agent correctly implemented priority-based sorting. You will direct Codex to write these tests, then review its output to confirm the tests actually validate the right behavior.
- Open the Codex agent panel in Google Antigravity IDE.
- Give Codex this prompt: "Write a pytest file at grader/test_task.py that tests a list_sorted method on the TaskStore class in environment/storage.py. The method should return tasks sorted by priority (highest first) with stable ordering for equal priorities. Write tests covering: method existence, basic priority sorting, stable sort for same priority, original list_all still returns insertion order, empty list, single task, all same priority preserves order, and that sorting does not mutate the internal list."
- Review what Codex produces. Compare it against the reference below to make sure all 8 test cases are covered and the path manipulation at the top is correct.
Your grader/test_task.py should look like this:
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "environment"))
from models import Task, Priority
from storage import TaskStore
def test_list_sorted_method_exists():
store = TaskStore()
assert hasattr(store, "list_sorted"), "TaskStore must have a list_sorted method"
def test_basic_priority_sorting():
store = TaskStore()
store.add(Task(title="Low task", priority=Priority.LOW))
store.add(Task(title="High task", priority=Priority.HIGH))
store.add(Task(title="Medium task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "High task"
assert sorted_tasks[1].title == "Medium task"
assert sorted_tasks[2].title == "Low task"
def test_stable_sort_same_priority():
store = TaskStore()
store.add(Task(title="First high", priority=Priority.HIGH))
store.add(Task(title="Second high", priority=Priority.HIGH))
store.add(Task(title="Third high", priority=Priority.HIGH))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "First high"
assert sorted_tasks[1].title == "Second high"
assert sorted_tasks[2].title == "Third high"
def test_original_list_unchanged():
store = TaskStore()
store.add(Task(title="Second", priority=Priority.HIGH))
store.add(Task(title="First", priority=Priority.LOW))
original = store.list_all()
assert original[0].title == "Second"
assert original[1].title == "First"
def test_empty_list():
store = TaskStore()
sorted_tasks = store.list_sorted()
assert sorted_tasks == []
def test_single_task():
store = TaskStore()
store.add(Task(title="Only task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert len(sorted_tasks) == 1
assert sorted_tasks[0].title == "Only task"
def test_all_same_priority_preserves_order():
store = TaskStore()
store.add(Task(title="A", priority=Priority.MEDIUM))
store.add(Task(title="B", priority=Priority.MEDIUM))
store.add(Task(title="C", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "A"
assert sorted_tasks[1].title == "B"
assert sorted_tasks[2].title == "C"
def test_sorted_does_not_mutate_internal_list():
store = TaskStore()
store.add(Task(title="Low", priority=Priority.LOW))
store.add(Task(title="High", priority=Priority.HIGH))
store.list_sorted()
original = store.list_all()
assert original[0].title == "Low"
assert original[1].title == "High"What do these tests check?
- sys.path.insert(0, ...) at the top adds the environment/ folder to Python's import path so the tests can find models.py and storage.py.
- test_basic_priority_sorting confirms HIGH tasks appear before MEDIUM, and MEDIUM before LOW.
- test_stable_sort_same_priority ensures tasks with equal priority keep their original insertion order. This is the subtle requirement that trips up many agents.
- test_sorted_does_not_mutate_internal_list catches a common agent mistake: sorting the internal _tasks list in-place instead of returning a new sorted copy.
Why review Codex's output?
Mechanize engineers never blindly accept agent-generated code. A test that asserts the wrong thing gives you a false signal. For example, if a test only checks that results are sorted but not that they are stably sorted, an agent could pass with a naive implementation that breaks the spec.
Your job is to verify that each test targets a distinct failure mode. If Codex missed any of the 8 cases, add them manually.
- Save grader/test_task.py.
- Run the tests against the current codebase (which does NOT have list_sorted yet) by running:
python3 -m pytest grader/ --tb=short -qYou should see all 8 tests fail. This is the correct result because the feature has not been implemented yet.
Seeing a ModuleNotFoundError?
Make sure you are running the command from the mechanize-task/ project root, not from inside the grader/ folder. The sys.path.insert line in the test file builds a relative path from its own location to the environment/ directory.
Also confirm that pytest is installed. Run pip install pytest==9.0.3 if needed.
help me debug this ModuleNotFoundError in my pytest grader
✔️ Awesome, I've got everything!
Great. Double check you have saved grader/test_task.py and all 8 tests appear in the file.
ⓧ I'd like to double check the full code
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "environment"))
from models import Task, Priority
from storage import TaskStore
def test_list_sorted_method_exists():
store = TaskStore()
assert hasattr(store, "list_sorted"), "TaskStore must have a list_sorted method"
def test_basic_priority_sorting():
store = TaskStore()
store.add(Task(title="Low task", priority=Priority.LOW))
store.add(Task(title="High task", priority=Priority.HIGH))
store.add(Task(title="Medium task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "High task"
assert sorted_tasks[1].title == "Medium task"
assert sorted_tasks[2].title == "Low task"
def test_stable_sort_same_priority():
store = TaskStore()
store.add(Task(title="First high", priority=Priority.HIGH))
store.add(Task(title="Second high", priority=Priority.HIGH))
store.add(Task(title="Third high", priority=Priority.HIGH))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "First high"
assert sorted_tasks[1].title == "Second high"
assert sorted_tasks[2].title == "Third high"
def test_original_list_unchanged():
store = TaskStore()
store.add(Task(title="Second", priority=Priority.HIGH))
store.add(Task(title="First", priority=Priority.LOW))
original = store.list_all()
assert original[0].title == "Second"
assert original[1].title == "First"
def test_empty_list():
store = TaskStore()
sorted_tasks = store.list_sorted()
assert sorted_tasks == []
def test_single_task():
store = TaskStore()
store.add(Task(title="Only task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert len(sorted_tasks) == 1
assert sorted_tasks[0].title == "Only task"
def test_all_same_priority_preserves_order():
store = TaskStore()
store.add(Task(title="A", priority=Priority.MEDIUM))
store.add(Task(title="B", priority=Priority.MEDIUM))
store.add(Task(title="C", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "A"
assert sorted_tasks[1].title == "B"
assert sorted_tasks[2].title == "C"
def test_sorted_does_not_mutate_internal_list():
store = TaskStore()
store.add(Task(title="Low", priority=Priority.LOW))
store.add(Task(title="High", priority=Priority.HIGH))
store.list_sorted()
original = store.list_all()
assert original[0].title == "Low"
assert original[1].title == "High"Create the scoring script
In reinforcement learning terms, the grader produces the reward signal. Your scoring script runs the pytest suite as a subprocess, counts how many tests passed, and outputs a score from 0.0 (nothing works) to 1.0 (everything passes).
- Open the empty score.py file in your editor and paste the following code:
import subprocess
import sys
import re
def run_grader():
result = subprocess.run(
[sys.executable, "-m", "pytest", "grader/", "--tb=short", "-q"],
capture_output=True,
text=True,
)
output = result.stdout + result.stderr
print(output)
match = re.search(r"(\d+) passed", output)
passed = int(match.group(1)) if match else 0
match_failed = re.search(r"(\d+) failed", output)
failed = int(match_failed.group(1)) if match_failed else 0
total = passed + failed
if total == 0:
score = 0.0
else:
score = passed / total
print(f"\n{'='*40}")
print(f"SCORE: {score:.2f} ({passed}/{total} tests passed)")
print(f"{'='*40}")
return score
if __name__ == "__main__":
run_grader()What does this script do?
- subprocess.run(...) launches pytest as a child process, capturing all terminal output.
- sys.executable ensures the script uses the same Python interpreter that is running score.py itself. This avoids version mismatches inside the Docker container.
- The regex parses pytest's summary line to extract the number of passed and failed tests, then divides passed by total to get a 0.0-1.0 score.
- If no tests run at all (total is 0), the score defaults to 0.0 instead of causing a division-by-zero error.
- Save score.py.
- Verify the script works by running:
python3 score.pyYou should see all 8 tests listed as FAILED, followed by a score line reading SCORE: 0.00 (0/8 tests passed).
Seeing 0 tests collected?
Make sure you are running python3 score.py from the mechanize-task/ project root. The script passes grader/ as a relative path to pytest, so it must be run from the directory that contains the grader/ folder.
help me debug why my score.py is not collecting any tests
✔️ Awesome, I've got everything!
Great. Double check you have saved score.py and confirmed the 0.00 score output.
ⓧ I'd like to double check the full code
import subprocess
import sys
import re
def run_grader():
result = subprocess.run(
[sys.executable, "-m", "pytest", "grader/", "--tb=short", "-q"],
capture_output=True,
text=True,
)
output = result.stdout + result.stderr
print(output)
match = re.search(r"(\d+) passed", output)
passed = int(match.group(1)) if match else 0
match_failed = re.search(r"(\d+) failed", output)
failed = int(match_failed.group(1)) if match_failed else 0
total = passed + failed
if total == 0:
score = 0.0
else:
score = passed / total
print(f"\n{'='*40}")
print(f"SCORE: {score:.2f} ({passed}/{total} tests passed)")
print(f"{'='*40}")
return score
if __name__ == "__main__":
run_grader()Containerize everything with Docker
A real Mechanize task ships as a self-contained Docker container. This means any AI lab can pull the image, drop an agent inside, and run the grader without worrying about dependencies or environment differences.
- Open the empty Dockerfile in your editor and paste the following:
FROM python:3.14-slim
WORKDIR /task
COPY environment/ ./environment/
COPY grader/ ./grader/
COPY score.py ./
COPY prompt.md ./
RUN pip install --no-cache-dir pytest==9.0.3
CMD ["python", "score.py"]What does each Dockerfile instruction do?
- FROM python:3.14-slim starts from a minimal Python 3.14 image. The slim variant keeps the image small and avoids compatibility issues.
- WORKDIR /task sets the working directory inside the container. All subsequent commands run from here.
- The COPY lines bring in the environment code, grader tests, scoring script, and prompt.
- RUN pip install --no-cache-dir pytest==9.0.3 installs the exact version of pytest needed to run the grader.
- CMD ["python", "score.py"] sets the default command, so running the container immediately executes the grader and outputs a score.
- Save the Dockerfile.
- Build the Docker image by running:
docker build -t mechanize-task .- Once the build completes, run the container to verify the grader works:
docker run mechanize-task python score.pyYou should see all 8 tests fail and a final score of SCORE: 0.00 (0/8 tests passed). This is exactly what you want. The feature has not been implemented, so a perfect grader should report a score of 0.0.
Why is 0.0 the correct result?
The list_sorted method does not exist on TaskStore yet. Every test that calls it will raise an AttributeError and fail. A score of 0.0 confirms your grader is correctly detecting the absence of the feature.
If you saw a score above 0.0, something is wrong. Your tests might be importing a different file or asserting the wrong thing.
Docker build failing?
Make sure Docker Desktop is running. If you see a "Cannot connect to the Docker daemon" error, start Docker Desktop and wait for it to finish loading.
If the build fails during the COPY step, confirm you are running docker build from the mechanize-task/ project root where the Dockerfile, environment/, and grader/ all live.
help me fix my Docker build for the mechanize-task
✔️ Awesome, I've got everything!
Great. Double check you have saved your Dockerfile and confirmed the 0.00 score from the container.
ⓧ I'd like to double check the full code
FROM python:3.14-slim
WORKDIR /task
COPY environment/ ./environment/
COPY grader/ ./grader/
COPY score.py ./
COPY prompt.md ./
RUN pip install --no-cache-dir pytest==9.0.3
CMD ["python", "score.py"]Your complete RL task package is containerized and ready to score any agent submission. Next up, you will hand this task to Codex and see how a frontier coding agent performs.
Run Codex Against Your Task and Analyze Results
Your RL task package is fully containerized and ready to evaluate an agent. Now comes the moment of truth: you will hand your task to OpenAI Codex and see how it performs.
This is the core workflow Mechanize engineers follow every day. You prompt a coding agent, observe what it produces, score it with your automated grader, and then write a failure analysis explaining why it fell short.
In this step, get ready to:
- Give Codex your task prompt and environment files (without the grader).
- Run the automated grader against Codex's submission and record the score.
- Write a failure analysis documenting where and why the agent failed.
Give Codex the task prompt and environment files
Mechanize engineers never give agents the grading tests. The agent only sees the prompt and the environment it must modify. This mirrors real evaluation conditions where the agent must infer correctness from the prompt alone.
- Open Codex in your browser or IDE.
- Provide Codex with the contents of prompt.md along with the three environment files: environment/models.py, environment/storage.py, and environment/app.py.
- Ask Codex to implement the feature described in the prompt.
- Do NOT provide grader/test_task.py or score.py. The agent must work without seeing the tests.
Why withhold the tests?
In reinforcement learning evaluation, the grader is the hidden reward function. If an agent sees the tests, it can overfit to them without truly understanding the task. Mechanize tasks must evaluate genuine capability, not test-gaming.
- Wait for Codex to produce its implementation.
- Save Codex's modified versions of the files. Copy each modified file into your environment/ folder, replacing the originals.
Keep the originals safe
Before overwriting your environment files with Codex's output, consider making a backup copy of the originals. You can create a folder called environment-original/ and copy the three files there so you can restore them later if needed.
Take a moment to read through what Codex produced. Look at which files it changed and what approach it took. You need to understand the submission before you can analyze its failures.
Run the grader against Codex's submission
Now you will rebuild the Docker image with Codex's changes baked in, then run the scoring script to get a numerical result. This is the automated reward signal that tells you exactly how the agent performed.
- Rebuild the Docker image with Codex's changes by running this command from your mechanize-task/ folder:
docker build -t mechanize-task .What does this command do?
This rebuilds the mechanize-task image from your Dockerfile. Because the Dockerfile copies the environment/ folder into the container, rebuilding picks up Codex's modified files.
- Run the grader against Codex's submission by running:
docker run mechanize-task python score.pyWhat does this command do?
This spins up the container and runs score.py, which executes the pytest suite inside grader/test_task.py and outputs a score between 0.0 and 1.0 based on how many tests passed.
- Record the score that appears in your terminal output. You will see a line like SCORE: 0.75 (6/8 tests passed).
- Note which specific tests failed. The pytest output above the score line shows each failing test name and a short traceback.
What if Codex scored 1.0?
If Codex passes all 8 tests, your task may be too easy for frontier models. Mechanize's guidance is clear: "Take what you were initially planning to ask the agent to do, make it ten times harder, and you'll probably have a good task idea."
A perfect score is still a valid result to document. Your analysis would note that cross-file coordination at this complexity level is within Codex's current capability boundary.
Seeing errors unrelated to the tests?
If you see import errors or the container fails to start, Codex may have renamed files or changed import paths. Check that the three files in environment/ are still named models.py, storage.py, and app.py.
help me debug why my Docker container is failing to run the grader
Analyze the failure and write your analysis document
This is the most important part of the evaluation workflow. Mechanize engineers produce a failure analysis for every task they ship. The analysis explains not just what failed, but why. This demonstrates the evaluation mindset Mechanize hires for.
- Create a file called analysis.md in your mechanize-task/ folder.
- Write your failure analysis using the following structure as a guide:
# Failure Analysis: Priority Sorting Task
## Agent Score
<!-- Record the numerical score from score.py -->
Score: X.XX (N/8 tests passed)
## Failing Tests
<!-- List each test that failed by name -->
- test_name_here: brief description of what it checks
## Root-Cause Analysis
<!-- Explain WHY the agent failed, not just what failed -->
<!-- Consider these common failure modes: -->
<!-- - Missed the stable sort requirement -->
<!-- - Mutated the internal _tasks list instead of returning a new sorted list -->
<!-- - Broke existing menu options -->
<!-- - Edited the wrong file -->
<!-- - Misunderstood the cross-file coordination requirement -->
## Observations
<!-- What does this tell you about the agent's capabilities? -->
<!-- Did it understand the prompt? Did it coordinate across files correctly? -->What makes a strong failure analysis?
A strong analysis goes beyond "it failed test X." It explains the root cause. Did the agent misunderstand the prompt? Did it take a shortcut that technically satisfies one requirement but breaks another? Did it fail at cross-file coordination?
Compare what Codex produced against what your prompt asked for. Look at the diff between the original storage.py and Codex's version. Check whether the list_sorted method returns a new list or mutates the internal one. These details reveal the agent's reasoning gaps.
- Fill in each section of analysis.md with your actual observations from running the grader.
- Save analysis.md.
Common agent failure patterns to look for
- Missed stable sort: the agent sorts by priority but does not preserve insertion order for equal priorities.
- Mutated internal list: list_sorted sorts self._tasks in place instead of returning a new sorted list.
- Broke existing functionality: the agent changed how option 2 works while adding option 5.
- Wrong file edited: the agent added sorting logic to app.py instead of storage.py.
Not sure what to write in your analysis?
Look at the test names in the pytest output. Each test name tells you exactly what it checks (e.g., test_stable_sort_same_priority checks stable sorting). Map each failure back to a specific behavior Codex got wrong.
help me analyze Codex's failure patterns
✔️ Awesome, I've got everything!
Great. Double check that you have saved analysis.md in your mechanize-task/ folder with your score, failing tests, and root-cause explanation filled in.
ⓧ I'd like to double check the full code
Your project files should remain unchanged from the previous step. Only analysis.md is new. Here are the files that should still be in your project:
FROM python:3.14-slim
WORKDIR /task
COPY environment/ ./environment/
COPY grader/ ./grader/
COPY score.py ./
COPY prompt.md ./
RUN pip install --no-cache-dir pytest==9.0.3
CMD ["python", "score.py"]import subprocess
import sys
import re
def run_grader():
result = subprocess.run(
[sys.executable, "-m", "pytest", "grader/", "--tb=short", "-q"],
capture_output=True,
text=True,
)
output = result.stdout + result.stderr
print(output)
match = re.search(r"(\d+) passed", output)
passed = int(match.group(1)) if match else 0
match_failed = re.search(r"(\d+) failed", output)
failed = int(match_failed.group(1)) if match_failed else 0
total = passed + failed
if total == 0:
score = 0.0
else:
score = passed / total
print(f"\n{'='*40}")
print(f"SCORE: {score:.2f} ({passed}/{total} tests passed)")
print(f"{'='*40}")
return score
if __name__ == "__main__":
run_grader()import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "environment"))
from models import Task, Priority
from storage import TaskStore
def test_list_sorted_method_exists():
store = TaskStore()
assert hasattr(store, "list_sorted"), "TaskStore must have a list_sorted method"
def test_basic_priority_sorting():
store = TaskStore()
store.add(Task(title="Low task", priority=Priority.LOW))
store.add(Task(title="High task", priority=Priority.HIGH))
store.add(Task(title="Medium task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "High task"
assert sorted_tasks[1].title == "Medium task"
assert sorted_tasks[2].title == "Low task"
def test_stable_sort_same_priority():
store = TaskStore()
store.add(Task(title="First high", priority=Priority.HIGH))
store.add(Task(title="Second high", priority=Priority.HIGH))
store.add(Task(title="Third high", priority=Priority.HIGH))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "First high"
assert sorted_tasks[1].title == "Second high"
assert sorted_tasks[2].title == "Third high"
def test_original_list_unchanged():
store = TaskStore()
store.add(Task(title="Second", priority=Priority.HIGH))
store.add(Task(title="First", priority=Priority.LOW))
original = store.list_all()
assert original[0].title == "Second"
assert original[1].title == "First"
def test_empty_list():
store = TaskStore()
sorted_tasks = store.list_sorted()
assert sorted_tasks == []
def test_single_task():
store = TaskStore()
store.add(Task(title="Only task", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert len(sorted_tasks) == 1
assert sorted_tasks[0].title == "Only task"
def test_all_same_priority_preserves_order():
store = TaskStore()
store.add(Task(title="A", priority=Priority.MEDIUM))
store.add(Task(title="B", priority=Priority.MEDIUM))
store.add(Task(title="C", priority=Priority.MEDIUM))
sorted_tasks = store.list_sorted()
assert sorted_tasks[0].title == "A"
assert sorted_tasks[1].title == "B"
assert sorted_tasks[2].title == "C"
def test_sorted_does_not_mutate_internal_list():
store = TaskStore()
store.add(Task(title="Low", priority=Priority.LOW))
store.add(Task(title="High", priority=Priority.HIGH))
store.list_sorted()
original = store.list_all()
assert original[0].title == "Low"
assert original[1].title == "High"Secret mission
Design a Second Task Targeting a Different Weakness
Your first task exposed how agents struggle with cross-file coordination. But that is only one dimension of capability. Can you design a completely new task from scratch that targets a different agent weakness, run Codex against it, and compare the failure patterns?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Docker image mechanize-task (built locally).
- Local project directory mechanize-task/ on your Desktop.
- pytest package (installed via pip).
Keep everything running
No action needed. Choose this if you're still actively building or want to extend the project further.
- Your Docker image stays cached locally. It takes up minimal disk space.
- Your project files remain on your Desktop.
- pytest stays installed in your Python environment.
Pause - I'll come back to this later
Free up running processes while keeping your files intact.
- If Docker Desktop is running, quit it to free up memory. Your image will still be there next time you open it.
- Your project files and pytest installation remain untouched.
Delete - I don't want to use this again
Remove all project resources and start fresh.
- Remove any stopped containers from the project by running:
docker rm $(docker ps -a -q --filter ancestor=mechanize-task)- Remove the Docker image by running:
docker rmi mechanize-task- Delete the project directory. On macOS/Linux, run:
rm -rf ~/Desktop/mechanize-taskOn Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\mechanize-task- Uninstall pytest by running:
pip uninstall pytestConfirm with y when prompted.
Nice Work!
Nice Work!
Nice work! You just built a production-grade RL task package structured exactly like the ones Mechanize sells to AI labs.
You've learned how to:
- Build a complete, containerized RL task with a natural language prompt, a multi-file Python environment, and an automated pytest grader.
- Use Codex as a tool to build software and scaffold code, mirroring how Mechanize engineers work daily.
- Evaluate a coding agent's output against deterministic tests and write a failure analysis identifying specific failure modes and root causes.
- Secret Mission: Designed a second task targeting state management across function boundaries and compared failure patterns between both tasks.
Ready to quiz yourself?