Build a LeRobot Dataset Quality Auditor

Introduction

30 Second Summary

A robot can only learn as well as the examples it receives. A missing step or strange movement can waste hours of training.

In this project, you will build a command-line quality auditor for selected episodes from the public lerobot/pusht dataset using LeRobot. It turns structural failures and behavioral warnings into JSON and Markdown reports.

What You'll Build

One Windows command gives every selected episode a PASS, REVIEW, or FAIL verdict with evidence you can inspect in JSON or share in Markdown.

By the end of this project, you'll have:

• A verified dataset profile that makes the PushT collection's scale visible before you audit a frame. You can inspect its timing contract and required numeric fields from the terminal.
• A raw-versus-decoded demonstration that reproduces the missing-video limitation on purpose. You can show that raw numeric access preserves the fields your audit needs.
• A reusable episode quality gate that checks frame order, timestamps, terminal markers, finite values, and action behavior. Each episode receives a concrete status with reasons. The reports are ready for automation or human review.
• Secret Mission: Compare two non-overlapping episode cohorts. Use their warning patterns to decide whether an issue is systemic, threshold-related, or isolated.

Are there any prerequisites?

You need a 64-bit computer running Windows 10 or 11 with internet access.

All required software setup is included for this public read-only workflow, which needs no Hugging Face Hub account or GPU.

Before We Start

Before any setup, commit to auditing the public read-only lerobot/pusht dataset at revision v3.0 without a robot, GPU, or Hugging Face account. You are checking structural and behavioral anomalies before committing compute to ACT or SmolVLA training.

Set Up Your Windows Workspace

Robot dataset behavior can change when package versions drift. A quality finding only carries weight when someone else can reproduce the environment that produced it.

You'll use Miniforge3 to create an isolated Conda environment with Python 3.12. You'll install LeRobot 0.6.1 before opening the workspace in Visual Studio Code.

In this step, get ready to:

• Prepare Miniforge Prompt and Visual Studio Code on Windows.
• Create an isolated Conda environment with pinned Python and LeRobot versions.
• Build the local workspace and verify every installed version.

Prepare your Windows tools

Miniforge Prompt gives you a Windows shell with Conda available. Your Miniforge3 starting point determines whether you need the current installer.

✔️ I have Miniforge3 26.7.2-0

Miniforge3 26.7.2-0 is ready for the environment commands later in this step.

ⓧ I have an older Miniforge3

Updating Miniforge3 gives you the pinned base tool used throughout this project.

• Visit the official Miniforge release page.
• Download Miniforge3-Windows-x86_64.exe for release 26.7.2-0.
• Run the downloaded installer.
• Keep the default installation choices.
• Finish the installation.

Good progress. Miniforge Prompt is now ready to create the isolated environment.

ⓧ Miniforge3 is not installed

The Windows x86_64 installer adds Miniforge Prompt to your computer. It supports Windows 10 or later.

• Visit the official Miniforge release page.
• Download Miniforge3-Windows-x86_64.exe for release 26.7.2-0.
• Run the downloaded installer.
• Keep the default installation choices.
• Finish the installation.

Your Conda command shell is now available through Miniforge Prompt.

Visual Studio Code provides the editor for the scripts you build in later steps. The Windows User setup makes the code command available to new console windows.

✔️ Visual Studio Code is installed

Visual Studio Code is ready. The workspace command later in this step confirms that its command-line launcher is available.

ⓧ Visual Studio Code is not installed

The User setup installs Visual Studio Code without requiring administrator permissions.

• Visit the official Visual Studio Code Windows setup page.
• Download the 64-bit User setup installer.
• Run the downloaded installer.
• Keep the option that adds code to %PATH% enabled.
• Finish the installation.

Visual Studio Code is now ready. A fresh Miniforge Prompt loads its command-line launcher.

• Press the Windows key to open Windows search.
• Type Miniforge Prompt into the search field.
• Press Enter to open Miniforge Prompt.

You'll see a terminal window where every command in this project runs.

Create the isolated environment

A Conda environment keeps this auditor's packages separate from other Python projects. The environment name also gives you a clear signal that each command uses the pinned runtime.

The first installation can take several minutes while Conda and pip download the dataset dependencies. Ongoing package output means the installation is still working.

• Create lerobot-auditor and install the pinned dataset package by running these commands:

conda create -y -n lerobot-auditor python=3.12
conda activate lerobot-auditor
pip install "lerobot[dataset]==0.6.1"

What do these commands set up?

• The first conda command creates an environment named lerobot-auditor with Python 3.12.
• The activation command directs later commands into that isolated environment.
• The pip command installs lerobot[dataset]==0.6.1 with the packages needed to load datasets.

When the commands finish, you'll see the environment name at the beginning of the terminal prompt. The final installation output confirms that the packages were installed.

Installation stopped?

• Confirm that you are using Miniforge Prompt because a standard Windows terminal may not have conda available.
• Check your internet connection if package downloads stop before installation completes.
• Allow Hugging Face storage hosts through your network if a corporate proxy blocks package downloads.
• Ask for help with the exact output you see: Help me diagnose my Miniforge package installation.

Build and verify the workspace

The workspace keeps your inspection scripts and future reports together. Placing it on your Desktop also makes the project easy to find outside the terminal.

• Move to your Desktop by running this command:

cd /d %USERPROFILE%\Desktop

What does this command do?

The command moves Miniforge Prompt to the Desktop folder inside your Windows user profile. The /d option also changes drives when your Desktop lives on another drive.

• Create the lerobot-dataset-auditor folder by running these commands:

mkdir lerobot-dataset-auditor
cd lerobot-dataset-auditor

What do these commands do?

The first command creates the workspace folder on your Desktop. The second command makes that folder the current location for the rest of the project.

Your Miniforge Prompt path should now end with the workspace folder name.

Before you run the version check, do you expect every installed component to match the pinned environment?

• Verify the active environment by running this command:

lerobot-info

What does this check prove?

The lerobot-info command reports the active LeRobot and Python versions. It also shows whether the Datasets dependency is available.

You should see LeRobot version: 0.6.1 in the output.

You should also see Python 3.12. The Datasets line should contain a version number.

That locks in your setup. Your auditor now has the pinned runtime needed for repeatable dataset checks.

Versions do not match?

• Confirm that the environment name appears at the beginning of the Miniforge Prompt line.
• Run the activation command from the environment setup again if another Conda environment is active.
• Repeat the pinned package installation if the LeRobot version differs from 0.6.1.
• Ask for help interpreting your version output: Help me fix my LeRobot environment versions.

• Open the current workspace in Visual Studio Code by running this command:

code .

What does this command do?

The code launcher opens Visual Studio Code. The . tells it to use the current folder as the workspace.

You'll see Visual Studio Code open with lerobot-dataset-auditor as the workspace folder.

Visual Studio Code did not open?

• Confirm that you installed the Visual Studio Code User setup.
• Close Miniforge Prompt after the Visual Studio Code installation completes.
• Open a fresh Miniforge Prompt so it loads the updated %PATH%.
• Ask for help checking the launcher: Help me make the Visual Studio Code command available.

Your Windows workspace is pinned and verified. Next up, you'll use anonymous public access to the Hugging Face Hub to inspect the PushT metadata before loading frame rows.

Inspect the PushT Metadata

Your pinned LeRobot environment is ready. The next job is to learn the PushT data contract before any frame rows reach your auditor.

Metadata reveals the expected schema before you touch thousands of frame rows. It also defines the timing contract your later checks rely on.

In this step, get ready to:

• Create video_decode_demo.py with the PushT repository settings.
• Fetch the public v3.0 metadata anonymously.
• Confirm the dataset size, frame rate, and required feature names.

Create the metadata script

The LeRobotDatasetMetadata class reads a dataset's description without loading its frame rows. This keeps your first inspection focused on the contract the auditor must enforce.

• In Visual Studio Code, create video_decode_demo.py directly inside lerobot-dataset-auditor by using the file controls in the left sidebar.
• Paste the following imports and configuration into video_decode_demo.py:

import argparse
import shutil
from pathlib import Path

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

REPO_ID = "lerobot/pusht"
REVISION = "v3.0"
CACHE_DIR = Path(".video_demo_cache")

What Does This Code Prepare?

• The imports provide command-line parsing, local cache cleanup, path handling, and LeRobot dataset access.
• The LeRobotDatasetMetadata import provides the metadata reader used in this step.
• The REPO_ID and REVISION constants pin the inspection to lerobot/pusht at v3.0.
• The CACHE_DIR constant records the cache path used by the later dataset experiment. This metadata inspection does not create that cache.

• Save video_decode_demo.py.
• Confirm video_decode_demo.py is listed directly inside lerobot-dataset-auditor in the left sidebar.

Can't See the New File?

• Confirm the file is saved directly inside lerobot-dataset-auditor.
• Check that the filename ends with .py.
• Help me locate the script in my workspace.

Print the dataset contract

The script needs one entry point that requests the pinned metadata. Its output turns the dataset description into facts you can compare against the auditor's requirements.

• Add the following code below CACHE_DIR = Path(".video_demo_cache") in video_decode_demo.py:

def main() -> None:
    parser = argparse.ArgumentParser(
        description="Contrast decoded and raw LeRobot dataset access."
    )
    parser.add_argument(
        "--decode-video",
        action="store_true",
        help="Use decoded item access even though videos are intentionally omitted.",
    )
    args = parser.parse_args()

    metadata = LeRobotDatasetMetadata(REPO_ID, revision=REVISION)
    print(f"Dataset: {REPO_ID}")
    print(f"Revision: {REVISION}")
    print(f"Episodes: {metadata.total_episodes}")
    print(f"Frames: {metadata.total_frames}")
    print(f"FPS: {metadata.fps}")
    print(f"Features: {sorted(metadata.features)}")


if __name__ == "__main__":
    main()

What Does This Code Do?

• The command-line parser reserves the --decode-video option for the dataset access experiment.
• The metadata object reads the description of lerobot/pusht at revision v3.0.
• The print statements expose the repository, revision, episode count, frame count, frame rate, and sorted feature names.

• Save video_decode_demo.py.

✔️ Awesome, I've got everything!

Your video_decode_demo.py now contains the imports, pinned dataset settings, metadata reader, and print statements.

ⓧ I'd like to double check the full code

import argparse
import shutil
from pathlib import Path

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

REPO_ID = "lerobot/pusht"
REVISION = "v3.0"
CACHE_DIR = Path(".video_demo_cache")


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Contrast decoded and raw LeRobot dataset access."
    )
    parser.add_argument(
        "--decode-video",
        action="store_true",
        help="Use decoded item access even though videos are intentionally omitted.",
    )
    args = parser.parse_args()

    metadata = LeRobotDatasetMetadata(REPO_ID, revision=REVISION)
    print(f"Dataset: {REPO_ID}")
    print(f"Revision: {REVISION}")
    print(f"Episodes: {metadata.total_episodes}")
    print(f"Frames: {metadata.total_frames}")
    print(f"FPS: {metadata.fps}")
    print(f"Features: {sorted(metadata.features)}")


if __name__ == "__main__":
    main()

Verify the metadata contract

The first run contacts the public Hugging Face Hub to fetch lightweight metadata. Public dataset metadata needs no sign-in.

• Switch back to the Miniforge Prompt from earlier.

Before you run the script, which dataset facts do you expect the metadata object to reveal?

• Run the metadata inspection by entering this command:

python video_decode_demo.py

What Does This Run Prove?

The command runs the default metadata path. It does not construct a dataset, so .video_demo_cache remains unused.

• You'll see Episodes: 206.
• You'll see Frames: 25650.
• You'll see FPS: 10.
• The Features: line shows the sorted dataset schema.

• Find the Features: line in the output.
• Confirm it includes observation.state, action, episode_index, frame_index, timestamp, next.reward, next.done, and next.success.

Strong start. You now have the exact size, timing, and feature contract that the quality auditor must enforce.

Metadata Not Printing?

• Confirm the active Conda environment is lerobot-auditor.
• Check that your internet connection allows downloads from the public Hugging Face dataset service.
• Help me diagnose the metadata run.

The PushT metadata contract is confirmed. Next, you'll request a dataset item and observe which dependency path LeRobot chooses.

Expose the Video Decoding Limitation

Your PushT metadata check confirmed the schema for 206 episodes at 10 FPS. That gives your auditor a clear numeric contract.

Full item access in LeRobot follows the dataset's video features. This step tests that coupling with a disposable cache that omits video files.

In this step, get ready to:

• Download the non-video data for episode 0 into a fresh disposable cache.
• Add a branch that requests a decoded item from the dataset.
• Run the decoded-access branch to expose its dependency on an omitted video file.

Configure LeRobotDataset

The experiment needs a clean starting point on every run. A fresh .video_demo_cache keeps previously downloaded assets from changing the result.

• In video_decode_demo.py, find the metadata line that prints Features.
• Add the cache reset and dataset constructor directly below that line by pasting this code:

    if CACHE_DIR.exists():
        shutil.rmtree(CACHE_DIR)

    dataset = LeRobotDataset(
        REPO_ID,
        root=CACHE_DIR,
        episodes=[0],
        revision=REVISION,
        download_videos=False,
    )

What Does This Code Do?

• The CACHE_DIR.exists() check detects an earlier experiment cache.
• The shutil.rmtree(CACHE_DIR) call removes that cache before the next dataset download.
• The constructor selects episode 0 from revision v3.0.
• The download_videos=False setting keeps video assets out of the disposable cache.

• Save video_decode_demo.py.
• Populate the disposable cache by running this command in the Miniforge Prompt from earlier:

python video_decode_demo.py --decode-video

You'll see the same metadata output from the previous step. The command then returns to the prompt without requesting a sample.

• Switch back to the file tree in Visual Studio Code.
• Confirm that .video_demo_cache now exists inside lerobot-dataset-auditor.

Good progress. Your disposable cache now contains the selected episode's non-video data.

Did the Cache Download Fail?

• Confirm that Miniforge Prompt still shows the lerobot-auditor environment.
• Check that your internet connection allows public dataset downloads.
• Help me diagnose the dataset download.

Test dataset[0]

The cache is ready. The remaining question is what standard item access does when the metadata still advertises an image stream.

• In video_decode_demo.py, locate the closing parenthesis of the LeRobotDataset constructor.
• Add the decoded-access branch directly below the constructor by pasting this code:

    if args.decode_video:
        print("Mode: decoded item access")
        sample = dataset[0]

What Does This Branch Do?

• The args.decode_video condition enters the branch when the matching command-line flag is present.
• The mode message makes the selected access path visible before the item request.
• The dataset[0] expression requests the standard decoded item.

• Save video_decode_demo.py.

Before you run this, do you think decoded item access can complete using the video-free cache?

• Test the decoded-access branch by running this command in Miniforge Prompt:

python video_decode_demo.py --decode-video

You'll see the metadata followed by Mode: decoded item access. The command then exits through a traceback before sample receives a decoded row.

Why This Failure Matters

The failure is intentional. The dataset[0] request follows the decoded video path because the metadata includes an image feature.

The download_videos=False setting left that video asset out of the fresh cache. The traceback exposes the unnecessary dependency in a numeric audit.

Did the Branch Avoid the Failure?

• Confirm that the cache removal block appears before the LeRobotDataset constructor.
• Confirm that the constructor still contains download_videos=False.
• Confirm that the branch assigns dataset[0] to sample.
• Help me reproduce the decoded-access failure.

✔️ I reproduced the limitation

You have proved that decoded item access reaches for a video asset even when your audit only needs numeric data.

ⓧ I'd like to double check the full code

• Compare your saved video_decode_demo.py with this complete version for the current step:

import argparse
import shutil
from pathlib import Path

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

REPO_ID = "lerobot/pusht"
REVISION = "v3.0"
CACHE_DIR = Path(".video_demo_cache")


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Contrast decoded and raw LeRobot dataset access."
    )
    parser.add_argument(
        "--decode-video",
        action="store_true",
        help="Use decoded item access even though videos are intentionally omitted.",
    )
    args = parser.parse_args()

    metadata = LeRobotDatasetMetadata(REPO_ID, revision=REVISION)
    print(f"Dataset: {REPO_ID}")
    print(f"Revision: {REVISION}")
    print(f"Episodes: {metadata.total_episodes}")
    print(f"Frames: {metadata.total_frames}")
    print(f"FPS: {metadata.fps}")
    print(f"Features: {sorted(metadata.features)}")

    if CACHE_DIR.exists():
        shutil.rmtree(CACHE_DIR)

    dataset = LeRobotDataset(
        REPO_ID,
        root=CACHE_DIR,
        episodes=[0],
        revision=REVISION,
        download_videos=False,
    )

    if args.decode_video:
        print("Mode: decoded item access")
        sample = dataset[0]


if __name__ == "__main__":
    main()

The limitation is now reproducible. Next, you'll preserve the numeric fields while removing decoded video access from the audit path.

Read Raw Numeric Rows

Your decoded-access test proved that LeRobot follows the video path when normal item access sees an image feature.

LeRobot's get_raw_item() API returns the stored row without video decoding. This keeps the quality gate focused on tabular evidence.

In this step, get ready to:

• Read the first stored row through the raw access path.
• Print the filtered keys plus the action values and state values.

Read one row without decoding video

The default branch needs a row-access method that ignores the omitted video asset. The raw-row API provides that route.

• In video_decode_demo.py from earlier, find the decoded branch containing sample = dataset[0].
• Add the default branch directly below the decoded branch by pasting this code:

    else:
        print("Mode: raw numeric row")
        sample = dataset.get_raw_item(0)

What does this branch do?

• The else branch handles runs that omit --decode-video.
• The get_raw_item(0) call returns the first stored row without triggering video decoding.
• The mode message makes the selected access path visible in the terminal.

• Save video_decode_demo.py.
• Test the default branch in the Miniforge Prompt from earlier by running this command:

python video_decode_demo.py

What should you see?

You should see the metadata followed by Mode: raw numeric row. The script should return to the prompt without the video access failure.

Seeing a syntax error?

• Confirm that else: aligns with if args.decode_video:.
• Confirm that the two lines inside else: have one extra indentation level.
• Ask for help with the branch indentation.

Print the numeric fields

The terminal check needs a compact view of the fields used by the auditor. Filtering keys that begin with observation.image keeps the omitted image stream out of this check.

• In video_decode_demo.py, place the cursor after the completed if and else branches.
• Add the filtered output section by pasting this code:

    numeric_keys = sorted(
        key for key in sample if not key.startswith("observation.image")
    )
    print(f"Numeric keys: {numeric_keys}")
    print(f"Action: {sample['action'].tolist()}")
    print(f"State: {sample['observation.state'].tolist()}")

What does this output show?

• The numeric_keys expression excludes every key beginning with observation.image.
• The sorted() call gives the fields a stable display order.
• The .tolist() calls convert the action tensor and state tensor into readable Python lists.

• Save video_decode_demo.py.

Before you run the completed script, which fields do you expect to remain after the image key is filtered out?

• Verify the completed raw-row path in the Miniforge Prompt from earlier by running this command:

python video_decode_demo.py

What should you see?

You should see Mode: raw numeric row. The Numeric keys: line should exclude keys beginning with observation.image.

The Action: line should show two values. The State: line should also show two values.

That is the key dependency removed. Your script now reads audit-ready rows without video assets.

Still seeing video access?

• Confirm that your command omits the --decode-video flag.
• Check that the default branch uses dataset.get_raw_item(0).
• Ask for help tracing the selected branch.

✔️ Awesome, I've got everything!

Great work. Double check that you saved video_decode_demo.py after the successful raw-row run.

ⓧ I'd like to double check the full code

Compare your video_decode_demo.py with this completed version.

import argparse
import shutil
from pathlib import Path

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

REPO_ID = "lerobot/pusht"
REVISION = "v3.0"
CACHE_DIR = Path(".video_demo_cache")


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Contrast decoded and raw LeRobot dataset access."
    )
    parser.add_argument(
        "--decode-video",
        action="store_true",
        help="Use decoded item access even though videos are intentionally omitted.",
    )
    args = parser.parse_args()

    metadata = LeRobotDatasetMetadata(REPO_ID, revision=REVISION)
    print(f"Dataset: {REPO_ID}")
    print(f"Revision: {REVISION}")
    print(f"Episodes: {metadata.total_episodes}")
    print(f"Frames: {metadata.total_frames}")
    print(f"FPS: {metadata.fps}")
    print(f"Features: {sorted(metadata.features)}")

    if CACHE_DIR.exists():
        shutil.rmtree(CACHE_DIR)

    dataset = LeRobotDataset(
        REPO_ID,
        root=CACHE_DIR,
        episodes=[0],
        revision=REVISION,
        download_videos=False,
    )

    if args.decode_video:
        print("Mode: decoded item access")
        sample = dataset[0]
    else:
        print("Mode: raw numeric row")
        sample = dataset.get_raw_item(0)

    numeric_keys = sorted(
        key for key in sample if not key.startswith("observation.image")
    )
    print(f"Numeric keys: {numeric_keys}")
    print(f"Action: {sample['action'].tolist()}")
    print(f"State: {sample['observation.state'].tolist()}")


if __name__ == "__main__":
    main()

Your raw-row path now exposes the fields needed for numeric checks. Next up, you'll turn those rows into an episode-level quality report.

Build and Run the Quality Auditor

Your raw-row workflow now reads numeric fields without touching the omitted videos. That successful sample proves the access path works.

A single sample cannot expose episode-level gaps or broken terminal markers. This step turns the raw-row workflow into a reusable quality gate with structural failures, behavioral warnings, JSON output, plus Markdown output.

In this step, get ready to:

• Build reusable helpers for selecting episodes and converting tensor values.
• Validate episode structure, timing, terminal markers, finite values, action spikes, plus repeated actions.
• Audit ten PushT episodes and generate two evidence-based reports.

Create the auditor foundation

The auditor needs a stable data contract before it can judge an episode. Its configuration identifies the dataset revision plus the fields every raw row must contain.

• In the Visual Studio Code Explorer sidebar from earlier, create audit_dataset.py inside lerobot-dataset-auditor.
• Add the imports plus dataset contract to audit_dataset.py by pasting this code:

import argparse
import json
import math
import statistics
from collections import defaultdict
from pathlib import Path
from typing import Any

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

DEFAULT_REPO_ID = "lerobot/pusht"
DEFAULT_REVISION = "v3.0"
REQUIRED_KEYS = (
    "observation.state",
    "action",
    "episode_index",
    "frame_index",
    "timestamp",
    "next.reward",
    "next.done",
    "next.success",
)

What Does This Set Up?

• The standard-library imports provide CLI parsing, report serialization, numeric checks, robust statistics, grouping, paths, plus type hints.
• The LeRobot imports preserve the dataset metadata plus raw-row behavior you verified earlier.
• The REQUIRED_KEYS tuple defines the minimum evidence required for each episode audit.

• Add the scalar conversion helper below the required-key tuple:

def scalar(value: Any) -> Any:
    if hasattr(value, "detach"):
        value = value.detach().cpu()
    if hasattr(value, "item"):
        return value.item()
    return value

Why Convert Scalars?

LeRobot rows can contain tensor-backed scalar values. scalar() moves those values to the CPU before returning an ordinary Python value.

• Save audit_dataset.py.
• Check the first section for syntax errors by running this command:

python -m py_compile audit_dataset.py

A successful syntax check returns you to the prompt without an error.

Seeing a Syntax Error?

• Check that every name in REQUIRED_KEYS has matching quotation marks.
• Check that the closing parenthesis appears below next.success.
• Ask for help with the reported line number: Help me fix the first syntax error in audit_dataset.py.

• Add the vector conversion helper below scalar().

def flatten_numbers(value: Any) -> list[float]:
    if hasattr(value, "detach"):
        value = value.detach().cpu()
    if hasattr(value, "tolist"):
        value = value.tolist()
    if isinstance(value, (list, tuple)):
        flattened: list[float] = []
        for item in value:
            flattened.extend(flatten_numbers(item))
        return flattened
    return [float(value)]

What Does Flattening Preserve?

The audit compares actions plus states as flat lists of numbers. Recursive flattening supports tensor values as well as nested Python sequences.

• Add the episode-range parser below flatten_numbers().

def parse_episode_spec(spec: str, total_episodes: int) -> list[int]:
    selected: set[int] = set()
    for part in spec.split(","):
        part = part.strip()
        if not part:
            continue
        if "-" in part:
            start_text, end_text = part.split("-", maxsplit=1)
            start = int(start_text)
            end = int(end_text)
            if end < start:
                raise ValueError(f"Invalid descending range: {part}")
            selected.update(range(start, end + 1))
        else:
            selected.add(int(part))

    episodes = sorted(selected)
    if not episodes:
        raise ValueError("Select at least one episode.")
    invalid = [episode for episode in episodes if episode < 0 or episode >= total_episodes]
    if invalid:
        raise ValueError(
            f"Episode indices out of range 0-{total_episodes - 1}: {invalid}"
        )
    return episodes

How Are Episode Ranges Parsed?

• Comma-separated values let the CLI combine individual episodes with ranges.
• A set removes duplicate selections before the result is sorted.
• The final bounds check prevents requests outside the dataset metadata.

• Save audit_dataset.py.
• Check the new helpers for syntax errors by running this command:

python -m py_compile audit_dataset.py

You should return to the prompt without a traceback.

Is the Parser Failing to Compile?

• Confirm that the nested range checks use four spaces for each indentation level.
• Confirm that the final return episodes remains inside parse_episode_spec().
• Ask for help with the parser structure: Help me fix my parse_episode_spec function.

• Add the action-distance helper below parse_episode_spec().

def l2_distance(left: list[float], right: list[float]) -> float:
    if len(left) != len(right):
        return math.inf
    return math.sqrt(sum((a - b) ** 2 for a, b in zip(left, right, strict=True)))

Why Use L2 Distance?

L2 distance reduces a multi-value action change to one magnitude. A dimension mismatch returns infinity because the two vectors cannot be compared safely.

• Save audit_dataset.py.
• Check the completed foundation by running this command:

python -m py_compile audit_dataset.py

The prompt should return without a syntax error. Your input parsing plus numeric conversion foundation is now valid Python.

Seeing an Error in l2_distance?

• Check that strict=True sits inside the zip() call.
• Check that the final line has matching parentheses.
• Ask for help with the distance expression: Help me check the parentheses in l2_distance.

Audit episode structure and behavior

Each episode needs two layers of judgment. Structural defects become errors because they break the data contract. Unusual action behavior becomes a warning because its meaning depends on trajectory context.

• Start audit_episode() below l2_distance() with the episode inputs plus row ordering:

def audit_episode(
    episode_index: int,
    rows: list[dict[str, Any]],
    fps: int,
    timestamp_tolerance: float,
    spike_multiplier: float,
    stuck_epsilon: float,
    stuck_ratio_threshold: float,
) -> dict[str, Any]:
    rows.sort(key=lambda row: int(scalar(row["frame_index"])))
    missing_keys = [key for key in REQUIRED_KEYS if key not in rows[0]]

Why Sort Every Episode?

The checks depend on frame order. Sorting by frame_index makes cadence plus action-step comparisons deterministic.

• Continue the function with its result record:

    result: dict[str, Any] = {
        "episode_index": episode_index,
        "frame_count": len(rows),
        "duration_seconds": None,
        "max_timestamp_error_seconds": None,
        "cadence_violation_count": None,
        "action_step_median": None,
        "action_step_mad": None,
        "action_spike_threshold": None,
        "action_spike_count": None,
        "repeated_action_ratio": None,
        "mean_next_state_tracking_error": None,
        "max_reward": None,
        "final_success": None,
        "errors": [],
        "warnings": [],
        "status": "PASS",
    }

What Does the Result Record Hold?

Every episode receives the same report shape. Empty measurements begin as None so an early failure remains distinguishable from a measured zero.

• Save audit_dataset.py.
• Check the opening audit structure by running this command:

python -m py_compile audit_dataset.py

A clean return to the prompt confirms that the long function signature plus result dictionary are balanced.

Is the Result Dictionary Unbalanced?

• Confirm that each result field ends with a comma.
• Confirm that the closing brace stays indented inside audit_episode().
• Ask for help locating the unmatched delimiter: Help me find the unmatched delimiter in the audit result dictionary.

• Add the missing-key response plus normalized episode values below the result dictionary:

    if missing_keys:
        result["errors"].append(f"Missing required keys: {missing_keys}")
        result["status"] = "FAIL"
        return result

    frame_indices = [int(scalar(row["frame_index"])) for row in rows]
    timestamps = [float(scalar(row["timestamp"])) for row in rows]
    states = [flatten_numbers(row["observation.state"]) for row in rows]
    actions = [flatten_numbers(row["action"]) for row in rows]
    rewards = [float(scalar(row["next.reward"])) for row in rows]
    done_flags = [bool(scalar(row["next.done"])) for row in rows]
    success_flags = [bool(scalar(row["next.success"])) for row in rows]

    result["duration_seconds"] = timestamps[-1] - timestamps[0]
    result["max_reward"] = max(rewards)
    result["final_success"] = success_flags[-1]

Why Normalize the Row Values?

The raw dataset can expose tensors or Python values. These comprehensions convert each field into a predictable form before validation.

• Add the frame-continuity plus finite-number checks below the summary measurements:

    expected_frames = list(range(len(rows)))
    if frame_indices != expected_frames:
        result["errors"].append("Frame indices are not contiguous from zero.")

    all_numeric = [value for vector in states + actions for value in vector]
    all_numeric.extend(timestamps)
    all_numeric.extend(rewards)
    if not all(math.isfinite(value) for value in all_numeric):
        result["errors"].append("One or more numeric values are NaN or infinite.")

What Makes These Structural Checks?

• Contiguous frame indices prove that no position is missing from the episode sequence.
• Finite-number validation catches values that would poison training calculations.

• Save audit_dataset.py.
• Check the structural section by running this command:

python -m py_compile audit_dataset.py

The prompt should return without an error. The function can now reject missing fields, broken frame sequences, plus invalid numeric values.

Seeing an Indentation Error?

• Keep every new line at the same four-space function indentation unless it belongs inside an if block.
• Keep the two error-appending lines nested beneath their matching conditions.
• Ask for help comparing indentation levels: Help me correct the indentation in audit_episode.

• Add the timestamp cadence checks below the finite-number check:

    timestamp_steps = [
        current - previous for previous, current in zip(timestamps, timestamps[1:])
    ]
    if any(step <= 0 for step in timestamp_steps):
        result["errors"].append("Timestamps are not strictly increasing.")

    expected_step = 1.0 / fps
    timestamp_errors = [abs(step - expected_step) for step in timestamp_steps]
    max_timestamp_error = max(timestamp_errors, default=0.0)
    cadence_violations = sum(
        error > timestamp_tolerance for error in timestamp_errors
    )
    result["max_timestamp_error_seconds"] = max_timestamp_error
    result["cadence_violation_count"] = cadence_violations
    if cadence_violations:
        result["errors"].append(
            f"{cadence_violations} timestamp steps exceed the tolerance."
        )

How Is Timing Validated?

The expected interval comes from the dataset frame rate. The PushT metadata reports 10 fps, so each neighboring timestamp should remain close to one tenth of a second.

The tolerance separates harmless floating-point drift from a cadence violation.

• Add the terminal-marker checks plus action-step calculation below the cadence checks:

    if not done_flags[-1]:
        result["errors"].append("The final frame is not marked done.")
    premature_done_count = sum(done_flags[:-1])
    if premature_done_count:
        result["errors"].append(
            f"{premature_done_count} non-final frames are marked done."
        )

    action_steps = [
        l2_distance(previous, current)
        for previous, current in zip(actions, actions[1:])
    ]

Why Check Terminal Placement?

An episode should terminate on its final frame. Earlier terminal markers split one trajectory into conflicting segments.

The action-step list measures how far each action moves from its predecessor.

• Save audit_dataset.py.
• Check the timing plus terminal-marker sections by running this command:

python -m py_compile audit_dataset.py

You should return to the prompt without an error. The structural contract now covers timing plus episode termination.

Is the Timestamp Section Failing?

• Check that timestamps[1:] includes the colon inside the brackets.
• Check that the multiline error append closes both parentheses.
• Ask for help with the cadence calculation: Help me debug the timestamp checks.

• Add the robust action statistics below action_steps.

    if action_steps:
        action_step_median = statistics.median(action_steps)
        action_step_mad = statistics.median(
            abs(step - action_step_median) for step in action_steps
        )
        action_spike_threshold = action_step_median + spike_multiplier * max(
            action_step_mad, stuck_epsilon
        )
        action_spike_count = sum(
            step > action_spike_threshold for step in action_steps
        )
        repeated_action_ratio = sum(
            step <= stuck_epsilon for step in action_steps
        ) / len(action_steps)
    else:
        action_step_median = 0.0
        action_step_mad = 0.0
        action_spike_threshold = 0.0
        action_spike_count = 0
        repeated_action_ratio = 0.0

Why Use Median and MAD?

The median describes a typical action step without being pulled toward extreme movements. Median absolute deviation measures the usual spread around that center.

The resulting spike threshold stays robust when a small number of action changes are unusually large.

• Store the action measurements plus warning messages below the statistics block:

    result["action_step_median"] = action_step_median
    result["action_step_mad"] = action_step_mad
    result["action_spike_threshold"] = action_spike_threshold
    result["action_spike_count"] = action_spike_count
    result["repeated_action_ratio"] = repeated_action_ratio

    if action_spike_count:
        result["warnings"].append(
            f"Detected {action_spike_count} robust action-step outlier(s)."
        )
    if repeated_action_ratio > stuck_ratio_threshold:
        result["warnings"].append(
            "Repeated-action ratio exceeds the project threshold."
        )

Why Are These Findings Warnings?

An action spike can represent a genuine correction. Repeated actions can represent a deliberate hold.

The auditor flags both patterns for review without declaring the trajectory invalid.

• Save audit_dataset.py.
• Check the action-statistics section by running this command:

python -m py_compile audit_dataset.py

A clean return confirms that the robust statistics plus warning branches compile.

Seeing an Error in the Statistics Block?

• Confirm that the else line aligns with if action_steps.
• Confirm that each multiline calculation closes before the next assignment begins.
• Ask for help with the branch structure: Help me fix the action-statistics branches.

• Finish audit_episode() with tracking error plus final status selection:

    tracking_errors = [
        l2_distance(action, next_state)
        for action, next_state in zip(actions, states[1:])
    ]
    result["mean_next_state_tracking_error"] = (
        statistics.fmean(tracking_errors) if tracking_errors else 0.0
    )

    if result["errors"]:
        result["status"] = "FAIL"
    elif result["warnings"]:
        result["status"] = "REVIEW"

    return result

How Is the Final Status Chosen?

• Any structural error produces FAIL.
• A warning without an error produces REVIEW.
• An episode with no findings keeps its initial PASS status.

• Save audit_dataset.py.
• Check the completed episode auditor by running this command:

python -m py_compile audit_dataset.py

You should return to the prompt without an error. One call to audit_episode() can now classify an entire trajectory.

Is audit_episode Still Failing to Compile?

• Confirm that return result remains inside the function.
• Use the first reported line number because later errors can be side effects of the same typo.
• Ask for help with the completed function: Help me debug audit_episode from the first syntax error.

Generate reports and run the audit

Episode classifications become useful when both people and automation can consume them. The next function creates a readable table while the main workflow preserves the same evidence in a machine-readable report.

• Start build_markdown() below audit_episode() with the report summary plus table headings:

def build_markdown(report: dict[str, Any]) -> str:
    lines = [
        "# LeRobot Dataset Audit",
        "",
        f"Dataset: {report['dataset']['repo_id']}",
        f"Revision: {report['dataset']['revision']}",
        f"FPS: {report['dataset']['fps']}",
        f"Episodes audited: {report['summary']['episodes_audited']}",
        f"Frames audited: {report['summary']['frames_audited']}",
        f"PASS: {report['summary']['pass_count']}",
        f"REVIEW: {report['summary']['review_count']}",
        f"FAIL: {report['summary']['fail_count']}",
        "",
        "| Episode | Frames | Duration (s) | Cadence issues | Action spikes | Repeated action ratio | Status |",
        "| ---: | ---: | ---: | ---: | ---: | ---: | :--- |",
    ]

What Starts the Markdown Report?

The opening lines identify the dataset plus aggregate status counts. The final two strings define a Markdown table for episode-level evidence.

• Add the per-episode table rows below the lines list:

    for episode in report["episodes"]:
        duration = episode["duration_seconds"]
        duration_text = "-" if duration is None else f"{duration:.2f}"
        cadence = episode["cadence_violation_count"]
        cadence_text = "-" if cadence is None else str(cadence)
        spikes = episode["action_spike_count"]
        spikes_text = "-" if spikes is None else str(spikes)
        repeated = episode["repeated_action_ratio"]
        repeated_text = "-" if repeated is None else f"{repeated:.3f}"
        lines.append(
            f"| {episode['episode_index']} | {episode['frame_count']} | "
            f"{duration_text} | {cadence_text} | {spikes_text} | "
            f"{repeated_text} | {episode['status']} |"
        )

How Are Missing Measurements Displayed?

An early structural failure can leave later measurements empty. The table displays a hyphen for those unavailable values while preserving the episode status.

• Save audit_dataset.py.
• Check the Markdown table builder by running this command:

python -m py_compile audit_dataset.py

The prompt should return without an error. The report builder can now format summary values plus episode rows.

Is the Markdown Table Failing?

• Check that dictionary keys inside each formatted string use single quotation marks.
• Check that the three adjacent formatted strings remain inside lines.append().
• Ask for help with the formatted strings: Help me fix the Markdown table strings.

• Finish build_markdown() with detailed findings plus the interpretation note:

    lines.extend(["", "## Findings", ""])
    for episode in report["episodes"]:
        findings = episode["errors"] + episode["warnings"]
        if findings:
            lines.append(
                f"- Episode {episode['episode_index']}: " + "; ".join(findings)
            )
        else:
            lines.append(f"- Episode {episode['episode_index']}: no findings.")

    lines.extend(
        [
            "",
            "## Interpretation Note",
            "",
            "Action spikes and repeated-action warnings are project heuristics, not universal robotics standards. Review the trajectory context before excluding data.",
            "",
        ]
    )
    return "\n".join(lines)

Why Include an Interpretation Note?

The report keeps structural errors separate from project heuristics. This protects unusual but valid trajectories from automatic exclusion.

• Start main() below build_markdown() with the command-line arguments:

def main() -> None:
    parser = argparse.ArgumentParser(
        description="Audit raw numeric rows from a LeRobot dataset."
    )
    parser.add_argument("--repo-id", default=DEFAULT_REPO_ID)
    parser.add_argument("--revision", default=DEFAULT_REVISION)
    parser.add_argument("--episodes", default="0-9")
    parser.add_argument("--output-dir", default="reports")
    parser.add_argument("--timestamp-tolerance", type=float, default=1e-4)
    parser.add_argument("--spike-multiplier", type=float, default=6.0)
    parser.add_argument("--stuck-epsilon", type=float, default=1e-6)
    parser.add_argument("--stuck-ratio-threshold", type=float, default=0.25)
    args = parser.parse_args()

What Can the CLI Control?

The CLI accepts the dataset identity, revision, episode selection, output location, plus warning thresholds. Defaults make the verified ten-episode audit available without extra configuration.

• Save audit_dataset.py.
• Check the completed report builder plus CLI parser by running this command:

python -m py_compile audit_dataset.py

You should return to the prompt without an error.

Is the CLI Parser Failing?

• Check that every argument name begins with two hyphens.
• Check that each default value matches its surrounding quotation marks or numeric type.
• Ask for help with the parser arguments: Help me debug the main parser.

• Add metadata loading plus the video-free dataset constructor below args = parser.parse_args().

    metadata = LeRobotDatasetMetadata(args.repo_id, revision=args.revision)
    selected_episodes = parse_episode_spec(
        args.episodes, metadata.total_episodes
    )
    dataset = LeRobotDataset(
        args.repo_id,
        episodes=selected_episodes,
        revision=args.revision,
        download_videos=False,
    )

Why Keep Videos Disabled?

The auditor only needs tabular state, action, timing, reward, plus terminal fields. download_videos=False preserves the numeric-only path you proved in the previous step.

• Add raw-row grouping plus per-episode audit calls below the dataset constructor:

    grouped_rows: dict[int, list[dict[str, Any]]] = defaultdict(list)
    for index in range(len(dataset)):
        row = dataset.get_raw_item(index)
        episode_index = int(scalar(row["episode_index"]))
        grouped_rows[episode_index].append(row)

    episode_reports = [
        audit_episode(
            episode_index,
            grouped_rows[episode_index],
            metadata.fps,
            args.timestamp_tolerance,
            args.spike_multiplier,
            args.stuck_epsilon,
            args.stuck_ratio_threshold,
        )
        for episode_index in selected_episodes
    ]

How Are Frames Grouped?

Each raw row is assigned to a list keyed by its episode_index value. The list comprehension then audits every requested episode with the same thresholds.

• Save audit_dataset.py.
• Check the raw-row loading workflow by running this command:

python -m py_compile audit_dataset.py

A clean return confirms that metadata loading, raw-row grouping, plus episode auditing compile together.

Seeing an Error in grouped_rows?

• Confirm that grouped_rows uses square brackets for dictionary access.
• Confirm that the audit arguments follow the function signature order.
• Ask for help with the grouping loop: Help me debug the raw-row grouping workflow.

The report dictionary is one large construct, so you will add it in two connected parts. Paste both parts before running another syntax check.

• Begin the report dictionary below episode_reports with dataset plus configuration evidence:

    report = {
        "dataset": {
            "repo_id": args.repo_id,
            "revision": args.revision,
            "fps": metadata.fps,
            "total_episodes": metadata.total_episodes,
            "total_frames": metadata.total_frames,
            "cache_root": str(dataset.root),
        },
        "configuration": {
            "episodes": selected_episodes,
            "download_videos": False,
            "timestamp_tolerance": args.timestamp_tolerance,
            "spike_multiplier": args.spike_multiplier,
            "stuck_epsilon": args.stuck_epsilon,
            "stuck_ratio_threshold": args.stuck_ratio_threshold,
        },

What Evidence Is Recorded First?

The report records the dataset identity plus cache location. It also preserves every threshold needed to reproduce the classification.

• Complete the same report dictionary with aggregate counts plus episode details:

        "summary": {
            "episodes_audited": len(episode_reports),
            "frames_audited": sum(
                episode["frame_count"] for episode in episode_reports
            ),
            "pass_count": sum(
                episode["status"] == "PASS" for episode in episode_reports
            ),
            "review_count": sum(
                episode["status"] == "REVIEW" for episode in episode_reports
            ),
            "fail_count": sum(
                episode["status"] == "FAIL" for episode in episode_reports
            ),
        },
        "episodes": episode_reports,
    }

What Does the Summary Measure?

The summary counts audited episodes plus frames. Separate status totals show how many episodes passed, need review, or failed.

• Save audit_dataset.py after both report parts are present.
• Check the completed report dictionary by running this command:

python -m py_compile audit_dataset.py

The prompt should return without an error. The in-memory report now contains metadata, configuration, summary counts, plus episode findings.

Is the Report Dictionary Incomplete?

• Confirm that the second block begins directly after the configuration section.
• Confirm that the final closing brace aligns with report = {.
• Ask for help balancing the nested dictionaries: Help me balance the report dictionary.

• Add output-directory creation plus report writing below the report dictionary:

    output_dir = Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)
    json_path = output_dir / "audit_report.json"
    markdown_path = output_dir / "audit_report.md"

    json_path.write_text(json.dumps(report, indent=2), encoding="utf-8")
    markdown_path.write_text(build_markdown(report), encoding="utf-8")

How Are the Reports Written?

The output directory is created when needed. The same report object becomes indented JSON plus a human-readable Markdown document.

• Finish main() with its terminal summary plus script entry point:

    print(f"Audited {report['summary']['episodes_audited']} episodes.")
    print(f"Frames: {report['summary']['frames_audited']}")
    print(
        "Status counts: "
        f"PASS={report['summary']['pass_count']}, "
        f"REVIEW={report['summary']['review_count']}, "
        f"FAIL={report['summary']['fail_count']}"
    )
    print(f"Dataset cache: {dataset.root}")
    print(f"Wrote {json_path}")
    print(f"Wrote {markdown_path}")


if __name__ == "__main__":
    main()

What Does the Terminal Summary Prove?

The summary confirms the audit scope plus status counts. It also exposes the local dataset cache path and both generated report paths.

• Save the completed audit_dataset.py.
• Check the complete file for syntax errors by running this command:

python -m py_compile audit_dataset.py

You should return to the prompt without an error. The complete auditor is ready to process dataset rows.

Does the Final Syntax Check Fail?

• Start with the first line number reported by the compiler.
• Compare the surrounding function against the full-file reference below.
• Ask for help with the exact compiler output: Help me fix the final syntax error in audit_dataset.py.

Before you run the full audit, how many episode reports do you expect from the range 0-9?

• Run the verified ten-episode audit with this command:

python audit_dataset.py --repo-id lerobot/pusht --episodes 0-9 --output-dir reports

The first run downloads the selected tabular data into the local Hugging Face Hub cache. It can take longer than later runs while that cache fills.

You should see Audited 10 episodes. plus status counts. You should also see the dataset cache path followed by paths for reports\audit_report.json plus reports\audit_report.md.

Did the Audit Stop Before Writing Reports?

• Confirm that Miniforge Prompt still shows the active lerobot-auditor environment.
• Confirm that the prompt remains inside lerobot-dataset-auditor.
• Ask for help with the final traceback: Help me debug the auditor run.

✔️ Awesome, I've got everything!

Strong work. Your auditor now classifies ten episodes and writes evidence for both automation and human review.

• Confirm that reports\audit_report.json appears in the Visual Studio Code Explorer sidebar.
• Confirm that reports\audit_report.md appears beside it.

ⓧ I'd like to double check the full code

Compare audit_dataset.py with this complete file. Every function plus threshold should match exactly.

import argparse
import json
import math
import statistics
from collections import defaultdict
from pathlib import Path
from typing import Any

from lerobot.datasets import LeRobotDataset, LeRobotDatasetMetadata

DEFAULT_REPO_ID = "lerobot/pusht"
DEFAULT_REVISION = "v3.0"
REQUIRED_KEYS = (
    "observation.state",
    "action",
    "episode_index",
    "frame_index",
    "timestamp",
    "next.reward",
    "next.done",
    "next.success",
)


def scalar(value: Any) -> Any:
    if hasattr(value, "detach"):
        value = value.detach().cpu()
    if hasattr(value, "item"):
        return value.item()
    return value


def flatten_numbers(value: Any) -> list[float]:
    if hasattr(value, "detach"):
        value = value.detach().cpu()
    if hasattr(value, "tolist"):
        value = value.tolist()
    if isinstance(value, (list, tuple)):
        flattened: list[float] = []
        for item in value:
            flattened.extend(flatten_numbers(item))
        return flattened
    return [float(value)]


def parse_episode_spec(spec: str, total_episodes: int) -> list[int]:
    selected: set[int] = set()
    for part in spec.split(","):
        part = part.strip()
        if not part:
            continue
        if "-" in part:
            start_text, end_text = part.split("-", maxsplit=1)
            start = int(start_text)
            end = int(end_text)
            if end < start:
                raise ValueError(f"Invalid descending range: {part}")
            selected.update(range(start, end + 1))
        else:
            selected.add(int(part))

    episodes = sorted(selected)
    if not episodes:
        raise ValueError("Select at least one episode.")
    invalid = [episode for episode in episodes if episode < 0 or episode >= total_episodes]
    if invalid:
        raise ValueError(
            f"Episode indices out of range 0-{total_episodes - 1}: {invalid}"
        )
    return episodes


def l2_distance(left: list[float], right: list[float]) -> float:
    if len(left) != len(right):
        return math.inf
    return math.sqrt(sum((a - b) ** 2 for a, b in zip(left, right, strict=True)))


def audit_episode(
    episode_index: int,
    rows: list[dict[str, Any]],
    fps: int,
    timestamp_tolerance: float,
    spike_multiplier: float,
    stuck_epsilon: float,
    stuck_ratio_threshold: float,
) -> dict[str, Any]:
    rows.sort(key=lambda row: int(scalar(row["frame_index"])))
    missing_keys = [key for key in REQUIRED_KEYS if key not in rows[0]]

    result: dict[str, Any] = {
        "episode_index": episode_index,
        "frame_count": len(rows),
        "duration_seconds": None,
        "max_timestamp_error_seconds": None,
        "cadence_violation_count": None,
        "action_step_median": None,
        "action_step_mad": None,
        "action_spike_threshold": None,
        "action_spike_count": None,
        "repeated_action_ratio": None,
        "mean_next_state_tracking_error": None,
        "max_reward": None,
        "final_success": None,
        "errors": [],
        "warnings": [],
        "status": "PASS",
    }

    if missing_keys:
        result["errors"].append(f"Missing required keys: {missing_keys}")
        result["status"] = "FAIL"
        return result

    frame_indices = [int(scalar(row["frame_index"])) for row in rows]
    timestamps = [float(scalar(row["timestamp"])) for row in rows]
    states = [flatten_numbers(row["observation.state"]) for row in rows]
    actions = [flatten_numbers(row["action"]) for row in rows]
    rewards = [float(scalar(row["next.reward"])) for row in rows]
    done_flags = [bool(scalar(row["next.done"])) for row in rows]
    success_flags = [bool(scalar(row["next.success"])) for row in rows]

    result["duration_seconds"] = timestamps[-1] - timestamps[0]
    result["max_reward"] = max(rewards)
    result["final_success"] = success_flags[-1]

    expected_frames = list(range(len(rows)))
    if frame_indices != expected_frames:
        result["errors"].append("Frame indices are not contiguous from zero.")

    all_numeric = [value for vector in states + actions for value in vector]
    all_numeric.extend(timestamps)
    all_numeric.extend(rewards)
    if not all(math.isfinite(value) for value in all_numeric):
        result["errors"].append("One or more numeric values are NaN or infinite.")

    timestamp_steps = [
        current - previous for previous, current in zip(timestamps, timestamps[1:])
    ]
    if any(step <= 0 for step in timestamp_steps):
        result["errors"].append("Timestamps are not strictly increasing.")

    expected_step = 1.0 / fps
    timestamp_errors = [abs(step - expected_step) for step in timestamp_steps]
    max_timestamp_error = max(timestamp_errors, default=0.0)
    cadence_violations = sum(
        error > timestamp_tolerance for error in timestamp_errors
    )
    result["max_timestamp_error_seconds"] = max_timestamp_error
    result["cadence_violation_count"] = cadence_violations
    if cadence_violations:
        result["errors"].append(
            f"{cadence_violations} timestamp steps exceed the tolerance."
        )

    if not done_flags[-1]:
        result["errors"].append("The final frame is not marked done.")
    premature_done_count = sum(done_flags[:-1])
    if premature_done_count:
        result["errors"].append(
            f"{premature_done_count} non-final frames are marked done."
        )

    action_steps = [
        l2_distance(previous, current)
        for previous, current in zip(actions, actions[1:])
    ]
    if action_steps:
        action_step_median = statistics.median(action_steps)
        action_step_mad = statistics.median(
            abs(step - action_step_median) for step in action_steps
        )
        action_spike_threshold = action_step_median + spike_multiplier * max(
            action_step_mad, stuck_epsilon
        )
        action_spike_count = sum(
            step > action_spike_threshold for step in action_steps
        )
        repeated_action_ratio = sum(
            step <= stuck_epsilon for step in action_steps
        ) / len(action_steps)
    else:
        action_step_median = 0.0
        action_step_mad = 0.0
        action_spike_threshold = 0.0
        action_spike_count = 0
        repeated_action_ratio = 0.0

    result["action_step_median"] = action_step_median
    result["action_step_mad"] = action_step_mad
    result["action_spike_threshold"] = action_spike_threshold
    result["action_spike_count"] = action_spike_count
    result["repeated_action_ratio"] = repeated_action_ratio

    if action_spike_count:
        result["warnings"].append(
            f"Detected {action_spike_count} robust action-step outlier(s)."
        )
    if repeated_action_ratio > stuck_ratio_threshold:
        result["warnings"].append(
            "Repeated-action ratio exceeds the project threshold."
        )

    tracking_errors = [
        l2_distance(action, next_state)
        for action, next_state in zip(actions, states[1:])
    ]
    result["mean_next_state_tracking_error"] = (
        statistics.fmean(tracking_errors) if tracking_errors else 0.0
    )

    if result["errors"]:
        result["status"] = "FAIL"
    elif result["warnings"]:
        result["status"] = "REVIEW"

    return result


def build_markdown(report: dict[str, Any]) -> str:
    lines = [
        "# LeRobot Dataset Audit",
        "",
        f"Dataset: {report['dataset']['repo_id']}",
        f"Revision: {report['dataset']['revision']}",
        f"FPS: {report['dataset']['fps']}",
        f"Episodes audited: {report['summary']['episodes_audited']}",
        f"Frames audited: {report['summary']['frames_audited']}",
        f"PASS: {report['summary']['pass_count']}",
        f"REVIEW: {report['summary']['review_count']}",
        f"FAIL: {report['summary']['fail_count']}",
        "",
        "| Episode | Frames | Duration (s) | Cadence issues | Action spikes | Repeated action ratio | Status |",
        "| ---: | ---: | ---: | ---: | ---: | ---: | :--- |",
    ]

    for episode in report["episodes"]:
        duration = episode["duration_seconds"]
        duration_text = "-" if duration is None else f"{duration:.2f}"
        cadence = episode["cadence_violation_count"]
        cadence_text = "-" if cadence is None else str(cadence)
        spikes = episode["action_spike_count"]
        spikes_text = "-" if spikes is None else str(spikes)
        repeated = episode["repeated_action_ratio"]
        repeated_text = "-" if repeated is None else f"{repeated:.3f}"
        lines.append(
            f"| {episode['episode_index']} | {episode['frame_count']} | "
            f"{duration_text} | {cadence_text} | {spikes_text} | "
            f"{repeated_text} | {episode['status']} |"
        )

    lines.extend(["", "## Findings", ""])
    for episode in report["episodes"]:
        findings = episode["errors"] + episode["warnings"]
        if findings:
            lines.append(
                f"- Episode {episode['episode_index']}: " + "; ".join(findings)
            )
        else:
            lines.append(f"- Episode {episode['episode_index']}: no findings.")

    lines.extend(
        [
            "",
            "## Interpretation Note",
            "",
            "Action spikes and repeated-action warnings are project heuristics, not universal robotics standards. Review the trajectory context before excluding data.",
            "",
        ]
    )
    return "\n".join(lines)


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Audit raw numeric rows from a LeRobot dataset."
    )
    parser.add_argument("--repo-id", default=DEFAULT_REPO_ID)
    parser.add_argument("--revision", default=DEFAULT_REVISION)
    parser.add_argument("--episodes", default="0-9")
    parser.add_argument("--output-dir", default="reports")
    parser.add_argument("--timestamp-tolerance", type=float, default=1e-4)
    parser.add_argument("--spike-multiplier", type=float, default=6.0)
    parser.add_argument("--stuck-epsilon", type=float, default=1e-6)
    parser.add_argument("--stuck-ratio-threshold", type=float, default=0.25)
    args = parser.parse_args()

    metadata = LeRobotDatasetMetadata(args.repo_id, revision=args.revision)
    selected_episodes = parse_episode_spec(
        args.episodes, metadata.total_episodes
    )
    dataset = LeRobotDataset(
        args.repo_id,
        episodes=selected_episodes,
        revision=args.revision,
        download_videos=False,
    )

    grouped_rows: dict[int, list[dict[str, Any]]] = defaultdict(list)
    for index in range(len(dataset)):
        row = dataset.get_raw_item(index)
        episode_index = int(scalar(row["episode_index"]))
        grouped_rows[episode_index].append(row)

    episode_reports = [
        audit_episode(
            episode_index,
            grouped_rows[episode_index],
            metadata.fps,
            args.timestamp_tolerance,
            args.spike_multiplier,
            args.stuck_epsilon,
            args.stuck_ratio_threshold,
        )
        for episode_index in selected_episodes
    ]

    report = {
        "dataset": {
            "repo_id": args.repo_id,
            "revision": args.revision,
            "fps": metadata.fps,
            "total_episodes": metadata.total_episodes,
            "total_frames": metadata.total_frames,
            "cache_root": str(dataset.root),
        },
        "configuration": {
            "episodes": selected_episodes,
            "download_videos": False,
            "timestamp_tolerance": args.timestamp_tolerance,
            "spike_multiplier": args.spike_multiplier,
            "stuck_epsilon": args.stuck_epsilon,
            "stuck_ratio_threshold": args.stuck_ratio_threshold,
        },
        "summary": {
            "episodes_audited": len(episode_reports),
            "frames_audited": sum(
                episode["frame_count"] for episode in episode_reports
            ),
            "pass_count": sum(
                episode["status"] == "PASS" for episode in episode_reports
            ),
            "review_count": sum(
                episode["status"] == "REVIEW" for episode in episode_reports
            ),
            "fail_count": sum(
                episode["status"] == "FAIL" for episode in episode_reports
            ),
        },
        "episodes": episode_reports,
    }

    output_dir = Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)
    json_path = output_dir / "audit_report.json"
    markdown_path = output_dir / "audit_report.md"

    json_path.write_text(json.dumps(report, indent=2), encoding="utf-8")
    markdown_path.write_text(build_markdown(report), encoding="utf-8")

    print(f"Audited {report['summary']['episodes_audited']} episodes.")
    print(f"Frames: {report['summary']['frames_audited']}")
    print(
        "Status counts: "
        f"PASS={report['summary']['pass_count']}, "
        f"REVIEW={report['summary']['review_count']}, "
        f"FAIL={report['summary']['fail_count']}"
    )
    print(f"Dataset cache: {dataset.root}")
    print(f"Wrote {json_path}")
    print(f"Wrote {markdown_path}")


if __name__ == "__main__":
    main()

Secret mission

Compare Two Episode Cohorts

Your first audit describes one group of PushT episodes. In this mission, you will audit a second independent group and decide whether any warning represents a wider pattern or an isolated finding.

Clean Up Your Resources

Clean Up Your Resources

Everything in this project lives on your Windows computer, so no cloud resources are accumulating charges. Choose whether to keep your local resources, pause your work, or delete everything.

Resources you used:

• A local Conda environment named lerobot-auditor.
• The local lerobot-dataset-auditor folder containing video_decode_demo.py, audit_dataset.py, cohort-comparison.md, the reports folder, and the .video_demo_cache folder.
• A local Hugging Face Hub dataset cache at the location printed by the auditor.

Keep everything running

No action is needed. Choose this option if you plan to reuse the auditor for another set of episodes.

• Keep the lerobot-dataset-auditor folder so your scripts, reports, and cohort comparison remain available.
• Keep the lerobot-auditor environment so the pinned dataset dependencies remain ready.
• Leave the local dataset caches in place to avoid downloading the same public PushT data again.
• Continue working in Visual Studio Code and the existing Miniforge Prompt when you return.

Pause - I'll come back to this later

Shut down the open applications while keeping every local file. Your project stays ready for another session without incurring any cost.

• Close Visual Studio Code.
• Close the existing Miniforge Prompt.
• Leave the lerobot-dataset-auditor folder unchanged.
• Leave the lerobot-auditor environment unchanged.
• Leave the local dataset caches unchanged.

Delete - I don't want to use this again

Removing all three resources is permanent, so save any cohort findings you want to keep before continuing. These steps remove every local project resource.

• Copy the path shown after Dataset cache: in the earlier auditor output here: dataset cache path printed by the auditor.
• Remove the cached PushT data by running this command in the existing Miniforge Prompt:

rmdir /s /q "[[DATASET_CACHE_PATH="dataset cache path printed by the auditor"]]"

What does this command remove?

This command recursively removes the dataset cache path you recorded. It leaves the Conda environment and project workspace untouched.

• Confirm Miniforge Prompt returns without reporting a deletion problem.

Cache Folder Still Present?

Check that you copied the complete path printed by the auditor. Help me remove my local dataset cache on Windows.

• Close Visual Studio Code.
• Move out of the workspace and delete the entire lerobot-dataset-auditor folder by running these commands in Miniforge Prompt:

cd ..
rmdir /s /q lerobot-dataset-auditor

What do these commands remove?

The first line moves Miniforge Prompt to the folder above your workspace. The second line removes the workspace with its scripts, reports, comparison note, and disposable video cache.

• Confirm the workspace is gone by listing the folders in the current location:

dir

What should you see?

The folder list should no longer include lerobot-dataset-auditor.

Workspace Folder Still Listed?

Make sure Visual Studio Code is closed before trying the removal command again. Help me delete my Windows project folder safely.

• Close the current Miniforge Prompt.
• Press the Windows key to open the search bar.
• Type Miniforge Prompt and press Enter to open a fresh prompt.

The fresh prompt starts outside the environment you are about to remove. Conda requests confirmation before deleting its packages.

• Remove the lerobot-auditor environment by running this command:

conda remove -n lerobot-auditor --all

What does this command remove?

This removes every package installed inside the lerobot-auditor environment. It also removes the environment itself from Conda.

• Enter y when Conda asks for confirmation.

Conda finishes by returning you to the prompt after removing the environment. Your workspace, dataset caches, and isolated environment are now deleted.

Environment Removal Did Not Finish?

Check that the fresh prompt does not show lerobot-auditor as the active environment. Help me remove the Conda environment.

Nice Work!

Nice Work!

You did it! You built a Windows-ready quality gate for public PushT demonstrations with LeRobot.

You've learned how to:

• Inspect LeRobotDataset metadata to verify the PushT schema before auditing frame rows.
• Expose the decoded-access limitation. Use raw numeric rows to audit data without downloading videos.
• Build a reusable episode auditor that separates structural errors from behavioral warnings. Generate JSON reports for automation. Create Markdown reports for human review.
• Secret Mission: Compared two non-overlapping PushT cohorts. Classified shared warnings as collection-policy issues. Distinguished threshold-calibration issues from isolated episodes.

Ready to quiz yourself?