Build a Paper Reproduction Agent

Build the highest-fidelity paper reproduction its disclosed artifacts support.

Introduction

30 Second Summary

A result in a research paper can look convincing until you try to produce it yourself. Missing data or different test conditions can turn a close number into a misleading comparison.

In this project, you will build a terminal-based comparison agent that reads a public research-paper PDF through the Gemini Developer API. The agent compares one extracted result with the observed output from a small Python prototype that runs in a managed code environment.

What You'll Build

The finished agent moves from paper analysis to prototype execution to a qualified verdict in your terminal, leaving a complete evidence bundle in the output folder.

By the end of this project, you'll have:

  • A structured paper brief that shows the reported value beside its metric, units, input scale, and evaluation conditions.
  • An evidence trail that keeps the generated unverified_draft.md separate from the executed prototype.py and its observed execution.log.
  • A qualified comparison report that classifies the result as DIRECTLY COMPARABLE, DIRECTIONALLY CONSISTENT, or NOT DIRECTLY COMPARABLE based on the available evidence.
  • Secret Mission: Audit a second paper whose data or experimental conditions cannot be matched, then document why a numeric comparison would be invalid.

Are there any prerequisites?

You'll need a Mac with Python 3, Visual Studio Code, a Gemini API key, and a public research-paper PDF smaller than 50 MB. The project costs $0 within Gemini API Free Tier limits, where submitted content may be used to improve Google's products, so public papers are required.

Before We Start

This checkpoint commits you to one measurable result from a public research-paper PDF under 50 MB. The project's evidence starts with executed prototype output because an unexecuted generated draft is only a proposal.

Prepare the Gemini Environment

You already have a public paper. You also know which measurable result you want to evaluate.

A virtual environment gives your project an isolated Python setup. It keeps the pinned Google Gen AI Python SDK separate from packages used by other projects.

A session-only Gemini Developer API credential lets the SDK authenticate without placing your key in a source file. Visual Studio Code keeps the paper plus your agent file together.

In this step, get ready to:
  • Verify that Python 3.10 or newer is available.
  • Create an isolated environment with Google Gen AI Python SDK 2.26.0.
  • Prepare the session credential plus the project files.
Verify your Python version

The pinned SDK requires Python 3.10 or newer. Checking the interpreter first prevents package compatibility problems during installation.

  • Press Command-Space bar to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Return to open Terminal.
  • Type cd followed by a space in Terminal.
  • Drag the folder containing your selected public PDF from Finder into Terminal.

Terminal inserts the folder's absolute path after cd.

  • Press Return to move into that folder.
  • Check the installed Python version by running this command:
python3 --version

What does this command check?

  • The python3 command selects the Python 3 interpreter available in your current shell.
  • The --version flag prints the interpreter version without starting an interactive Python session.

Choose the tab that matches the version result in Terminal.

✔️ I see a supported version

Your output shows Python 3.10 or newer. Your interpreter supports the pinned SDK.

That compatibility check is complete. You can create the isolated environment next.

ⓧ I see an older version

Your current interpreter is below Python 3.10. Install the current stable macOS release before creating the environment.

  • Open the Python.org macOS downloads page.
  • Download the macOS installer for Python 3.14.8.
  • Complete the installer with its default options.
  • Quit Terminal after the installer finishes.
  • Reopen Terminal through Spotlight.
  • Return to the folder containing your selected PDF.
  • Check the updated version by running this command:
python3 --version

What confirms the upgrade?

The command reads the Python interpreter available to the reopened shell. Continue when the version is 3.10 or newer.

Still seeing the older version?

Close every Terminal window after installation. Reopen Terminal so the new shell can locate the updated interpreter.

If the older version remains, ask for help checking which Python installation your shell is using.

ⓧ Command not found

Terminal cannot find a Python 3 interpreter. Install the current stable macOS release before continuing.

  • Open the Python.org macOS downloads page.
  • Download the macOS installer for Python 3.14.8.
  • Complete the installer with its default options.
  • Quit Terminal after the installer finishes.
  • Reopen Terminal through Spotlight.
  • Return to the folder containing your selected PDF.
  • Confirm that Python is now available by running this command:
python3 --version

What confirms the installation?

The command should now print a Python version. Continue when that version is 3.10 or newer.

Still unable to find Python?

Make sure the Python.org installer completed before you reopened Terminal.

If Terminal still cannot find Python, ask for help checking your macOS Python installation.

Create and verify the virtual environment

The .venv folder stores a dedicated Python interpreter plus this project's packages. Activating it makes its interpreter the default for commands in the current Terminal session.

  • Create the virtual environment inside the current project folder by running this command:
python3 -m venv .venv

What does this command create?

  • The venv module creates an isolated Python environment.
  • The .venv value names the environment folder inside your project folder.
  • Confirm that Terminal returns to the prompt without reporting an error.

Could not create the environment?

Check that the version command from the previous substep reports Python 3.10 or newer. Confirm that Terminal is still inside the folder containing your selected PDF.

If environment creation still fails, ask for help diagnosing the venv command.

  • Activate the new environment in the current Terminal session by running this command:
source .venv/bin/activate

What does activation change?

  • The activation script updates the current shell to use the interpreter inside .venv.
  • The environment remains active only in this Terminal session.
  • Confirm that your Terminal prompt begins with (.venv).

Missing the environment name?

Confirm that the .venv folder was created in your current project folder. Run the activation command from that same folder.

If activation still fails, ask for help checking the environment path.

  • Install the pinned SDK into the active environment by running this command:
python -m pip install google-genai==2.26.0

What does this installation do?

  • The python -m pip portion uses the package installer belonging to the active environment.
  • The google-genai==2.26.0 pin installs the exact SDK version used by this project.
  • Confirm that the installation finishes without reporting an error.

Did the SDK installation fail?

Check that your prompt still begins with (.venv). A network interruption can also prevent the package download from completing.

If the installation keeps failing, ask for help diagnosing the package installation.

Good progress. The pinned SDK is now isolated inside the project environment.

Add the session key and project files

Your API key authorizes requests from the SDK. Keeping it in the current Terminal environment prevents it from being written into paper_agent.py.

This credential step is easy to undo. Closing the Terminal session removes the exported value from that environment.

  • Export your key for this Terminal session by replacing the plain placeholder in this command:
export GEMINI_API_KEY='your-api-key-here'

How does the SDK find the key?

  • The export command adds the value to the current Terminal environment.
  • The SDK automatically reads GEMINI_API_KEY when your script creates a client.
  • The key stays out of your Python source file.
  • Confirm that Terminal returns to the prompt without reporting an error.

Key unavailable in a later command?

Return to the same Terminal window where you exported the key. A new Terminal session needs its own export command.

If the SDK cannot read the key later, ask for help checking the session environment without sharing the key.

  • Leave this Terminal window open so the virtual environment remains active.
  • Press Command-Space bar to open Spotlight.
  • Type Visual Studio Code into Spotlight.
  • Press Return to open Visual Studio Code.
  • Click File in the top menu.
  • Select Open Folder....
  • Choose the folder containing your selected public PDF.
  • Click Open.

You should see the project folder in the Explorer view on the left.

  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

Why use a public paper?

The Gemini API free tier may use submitted content to improve Google's products. Keep this project limited to public research papers.

The PDF must remain smaller than 50 MB because the agent sends it as inline data.

  • Right-click your selected PDF in the Explorer view if it has a different filename.
  • Select Rename.
  • Enter paper.pdf as the filename.
  • Press Return to save the new filename.
  • Select the New File... button in the Explorer view.
  • Enter paper_agent.py as the new filename.
  • Press Return to create the empty agent file.

The Explorer should now list paper.pdf plus paper_agent.py in the same folder.

Before you run the final check, predict whether Python can import the pinned SDK from the active environment.

  • Return to the same Terminal window from earlier.
  • Verify the SDK import by running this command:
python -c "from google import genai; print('SDK ready')"

What does this check prove?

  • The active python interpreter attempts to import the SDK's genai module.
  • The print statement runs only after that import succeeds.

You should see SDK ready in Terminal. That confirms the active virtual environment can load the pinned SDK.

Import check did not succeed?

Confirm that the prompt begins with (.venv). Return to the SDK installation command if the package was installed outside the active environment.

If the import still fails, ask for help comparing the active Python interpreter with the SDK installation.

Your Python environment can now load the Gemini SDK. Next, you will turn paper_agent.py into a CLI that extracts the reported result from paper.pdf.

Extract the Paper Result

Your environment can now call Gemini from Python. A trustworthy comparison still needs an exact result from the paper.

This step uses a multimodal prompt to turn your public PDF into a structured Markdown brief. The brief keeps the paper's claims separate from interpretation.

In this step, get ready to:
  • Validate the local paper before analysis.
  • Convert the paper into inline document data.
  • Save one structured brief containing the reported result.
Build the PDF input layer

Gemini receives the paper as an inline document block. Local validation rejects an invalid file before the API request uses it.

  • Create paper_agent.py inside the project folder by using the file creation control in the VS Code sidebar.
  • Add the imports plus shared limits by pasting this code at the top of paper_agent.py:
import argparse
import base64
import os
import sys
from pathlib import Path

from google import genai
from google.genai import errors

MODEL_ID = "gemini-3.8-flash"
MAX_PDF_BYTES = 50_000_000

What does this code set up?

  • The imports provide command-line parsing, document encoding, environment access, error output, plus file-path handling.
  • The SDK imports provide the Gemini client plus its API error type.
  • The MODEL_ID constant keeps every interaction on gemini-3.8-flash.
  • The MAX_PDF_BYTES constant represents the 50 MB inline-data limit.
  • Save paper_agent.py.
  • Confirm the import block has no Python syntax warning in VS Code.

Seeing an import warning?

Confirm VS Code is using the Python interpreter inside .venv. The activated environment already contains the pinned SDK.

Check each import against the block above if the warning points to a misspelled module.

Help me resolve the import warning in paper_agent.py.

  • Add the command-line argument parser below MAX_PDF_BYTES by pasting this function:
def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Turn a research-paper PDF into a tested Python prototype."
    )
    parser.add_argument("pdf", type=Path, help="Path to a public PDF under 50 MB")
    return parser.parse_args()

What does the parser do?

The parse_args() function requires one PDF path when the CLI runs. Converting that argument to a Path gives the remaining helpers a consistent file object.

  • Save paper_agent.py.
  • Confirm VS Code recognizes parse_args() as a complete function.

Is the parser block underlined?

Make sure every line inside parse_args() uses four spaces of indentation. Check that both closing parentheses remain in place.

Help me check the indentation in parse_args().

  • Add the input checks below parse_args() by pasting this function:
def validate_pdf(pdf_path: Path) -> None:
    if not pdf_path.is_file():
        raise ValueError(f"PDF not found: {pdf_path}")
    if pdf_path.suffix.lower() != ".pdf":
        raise ValueError("The input file must have a .pdf extension.")
    size = pdf_path.stat().st_size
    if size == 0:
        raise ValueError("The PDF is empty.")
    if size > MAX_PDF_BYTES:
        raise ValueError("The PDF exceeds the 50 MB inline-data limit.")

How does validation protect the request?

  • The first check confirms that the supplied path points to a file.
  • The extension check limits this workflow to .pdf input.
  • The size checks reject empty files plus files above the inline-data limit.
  • Save paper_agent.py.
  • Confirm validate_pdf() contains four separate rejection checks.

Does the function look incomplete?

Confirm every raise ValueError line is nested beneath its matching condition. Check that size is assigned before either size comparison.

Help me check validate_pdf() for a missing validation branch.

  • Add the safe output-folder naming helper below validate_pdf() by pasting this function:
def safe_stem(pdf_path: Path) -> str:
    cleaned = "".join(
        character if character.isalnum() or character in "-_" else "_"
        for character in pdf_path.stem
    )
    return cleaned or "paper"

Why clean the paper name?

The safe_stem() helper turns the PDF filename into a predictable folder name. Unsupported characters become underscores.

The fallback value keeps the output path usable when the original stem contains no accepted characters.

  • Save paper_agent.py.
  • Confirm the generator expression remains inside the join() call.

Seeing a bracket warning?

Check that join( has one matching closing parenthesis after the generator expression. Keep the return line outside that expression.

Help me fix the brackets in safe_stem().

Inline document data keeps this one-paper workflow focused on the analysis request. It creates no persisted Files API resource.

  • Add the document conversion helper below safe_stem() by pasting this function:
def document_block(pdf_path: Path) -> dict[str, str]:
    encoded_pdf = base64.b64encode(pdf_path.read_bytes()).decode("utf-8")
    return {
        "type": "document",
        "data": encoded_pdf,
        "mime_type": "application/pdf",
    }

How does the document block work?

  • The helper reads the local paper as bytes.
  • Base64 encoding converts those bytes into text that fits inside the request structure.
  • The MIME type tells Gemini to interpret the encoded data as a PDF document.
  • Save paper_agent.py.
  • Confirm the returned dictionary contains type, data, plus mime_type.

Is the document block underlined?

Check that every dictionary entry ends with a comma. Confirm the closing brace aligns with the return line.

Help me fix the document_block() dictionary.

  • Add the response-text guard below document_block() by pasting this function:
def require_output_text(interaction, stage_name: str) -> str:
    text = interaction.output_text
    if not text or not text.strip():
        raise RuntimeError(f"{stage_name} returned no text output.")
    return text.strip()

Why check the returned text?

The SDK exposes the response text through interaction.output_text. This helper prevents an empty response from becoming a misleading report file.

  • Save paper_agent.py.
  • Confirm require_output_text() strips surrounding whitespace before returning the result.

Seeing a return-type warning?

Confirm text is assigned from interaction.output_text. Keep the final return line outside the conditional block.

Help me check require_output_text().

Your input layer now knows how to reject an invalid paper. It can also package a valid paper for analysis.

Add the paper analysis interaction

The analysis prompt defines the evidence standard for the whole project. It asks for one measurable result plus the conditions needed to judge a later comparison.

  • Insert the analysis prompt below MAX_PDF_BYTES by pasting this block before parse_args():
ANALYSIS_PROMPT = """
Read this research paper as a reproducibility analyst.

Return a concise Markdown brief with exactly these headings:

# Paper
## Core Method
## Reproduction Target
## Original Reported Result
## Metric and Units
## Original Evaluation Conditions
## Required Inputs and Dependencies
## Missing or Ambiguous Details
## Minimal Prototype Plan
## Comparison Risks

Requirements:
- Select one measurable headline result that a small Python prototype could test.
- Quote the reported value, metric, units, dataset or input, scale, baseline, and evaluation procedure when the paper states them.
- Distinguish statements found in the paper from your interpretation.
- Write "not stated" for information the paper does not provide.
- If a faithful reproduction requires unavailable data, specialized hardware, long training, or proprietary dependencies, say so explicitly.
- Do not invent a numeric result or experimental condition.
""".strip()

What does the prompt require?

  • The headings create a consistent evidence record for every paper.
  • The result requirements preserve the reported value plus its measurement context.
  • The interpretation rule exposes any reasoning that comes from Gemini instead of the source paper.
  • The final restrictions prevent missing details from becoming invented facts.
  • Save paper_agent.py.
  • Confirm the prompt starts below MAX_PDF_BYTES.
  • Confirm the closing strip() appears before parse_args().

Does the prompt swallow the next function?

Check for three double quotes at both ends of ANALYSIS_PROMPT. A missing quote can make Python treat the next function as prompt text.

Help me find an unclosed string in ANALYSIS_PROMPT.

  • Add the Gemini interaction below require_output_text() by pasting this function:
def analyze_paper(client, pdf_path: Path) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=[
            document_block(pdf_path),
            {"type": "text", "text": ANALYSIS_PROMPT},
        ],
    )
    return require_output_text(interaction, "Paper analysis")

How does the analysis request work?

  • The interaction uses the shared model identifier.
  • The document block appears before the text prompt in the input sequence.
  • The response guard returns usable analysis text or raises a clear error.
  • Save paper_agent.py.
  • Confirm the input list contains the document block first.
  • Confirm the analysis prompt is the second input item.

Seeing a request-structure warning?

Confirm the input value uses square brackets because it contains two items. Check that the text item remains inside curly braces.

Help me fix the analyze_paper() request structure.

  • Add the report-writing helper below analyze_paper() by pasting this function:
def write_text(path: Path, content: str) -> None:
    path.write_text(content.rstrip() + "\n", encoding="utf-8")

Why use one writing helper?

The write_text() helper saves every report with UTF-8 encoding. It also keeps exactly one newline at the end of the file.

  • Save paper_agent.py.
  • Confirm write_text() appears immediately below analyze_paper().

Is the newline displayed incorrectly?

Keep "\n" inside the Python string. It represents the final newline written to the Markdown file.

Help me fix the write_text() string.

Run the analysis checkpoint

The remaining code connects the helpers into one CLI run. It keeps the credential in the terminal environment instead of placing it in the source file.

  • Add the analysis-only main() function below write_text() by pasting this block:
def main() -> int:
    args = parse_args()
    pdf_path = args.pdf.expanduser().resolve()

    try:
        if not os.getenv("GEMINI_API_KEY"):
            raise ValueError("Set GEMINI_API_KEY in the current terminal session.")

        validate_pdf(pdf_path)
        output_dir = Path("outputs") / safe_stem(pdf_path)
        output_dir.mkdir(parents=True, exist_ok=True)

        with genai.Client() as client:
            print("1/3 Analyzing the paper and extracting its reported result...")
            brief = analyze_paper(client, pdf_path)
            write_text(output_dir / "paper_brief.md", brief)

        print(f"Complete. Open {output_dir} to inspect the evidence bundle.")
        return 0

    except errors.APIError as error:
        print(
            f"Gemini API error {error.code}: {error.message}",
            file=sys.stderr,
        )
        return 1
    except (OSError, RuntimeError, ValueError) as error:
        print(f"Error: {error}", file=sys.stderr)
        return 1

What does the checkpoint do?

  • The CLI resolves the supplied PDF path before validating it.
  • The environment check confirms that the current terminal session still has the credential.
  • The output directory uses the cleaned paper name.
  • The client sends the analysis request before saving paper_brief.md.
  • The exception handlers turn API, file, response, plus validation failures into readable terminal messages.
  • Save paper_agent.py.
  • Confirm the with genai.Client() as client block contains the analysis call.
  • Confirm write_text() targets paper_brief.md.

Does main() show indentation warnings?

Keep the credential check plus analysis workflow inside try. Align both except blocks with try.

Help me check the indentation in main().

  • Add the script entry point at the bottom of paper_agent.py by pasting these lines:
if __name__ == "__main__":
    raise SystemExit(main())

What does the entry point do?

The entry-point guard calls main() when you run the file as a script. SystemExit returns the function's status code to the terminal.

  • Save paper_agent.py.
  • Confirm the entry-point guard is the final block in the file.

Does the script stop before main()?

Confirm the entry-point guard appears after every function definition. Check that the raise SystemExit(main()) line uses four spaces of indentation.

Help me place the entry-point guard correctly.

The checkpoint script should now match the complete analysis-only file. Use the tabs below to compare your version before the API call.

✔️ Awesome, I've got everything!

  • Confirm paper_agent.py is saved.
  • Continue to the prediction prompt below.

ⓧ I'd like to double check the full code

import argparse
import base64
import os
import sys
from pathlib import Path

from google import genai
from google.genai import errors

MODEL_ID = "gemini-3.8-flash"
MAX_PDF_BYTES = 50_000_000

ANALYSIS_PROMPT = """
Read this research paper as a reproducibility analyst.

Return a concise Markdown brief with exactly these headings:

# Paper
## Core Method
## Reproduction Target
## Original Reported Result
## Metric and Units
## Original Evaluation Conditions
## Required Inputs and Dependencies
## Missing or Ambiguous Details
## Minimal Prototype Plan
## Comparison Risks

Requirements:
- Select one measurable headline result that a small Python prototype could test.
- Quote the reported value, metric, units, dataset or input, scale, baseline, and evaluation procedure when the paper states them.
- Distinguish statements found in the paper from your interpretation.
- Write "not stated" for information the paper does not provide.
- If a faithful reproduction requires unavailable data, specialized hardware, long training, or proprietary dependencies, say so explicitly.
- Do not invent a numeric result or experimental condition.
""".strip()


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Turn a research-paper PDF into a tested Python prototype."
    )
    parser.add_argument("pdf", type=Path, help="Path to a public PDF under 50 MB")
    return parser.parse_args()


def validate_pdf(pdf_path: Path) -> None:
    if not pdf_path.is_file():
        raise ValueError(f"PDF not found: {pdf_path}")
    if pdf_path.suffix.lower() != ".pdf":
        raise ValueError("The input file must have a .pdf extension.")
    size = pdf_path.stat().st_size
    if size == 0:
        raise ValueError("The PDF is empty.")
    if size > MAX_PDF_BYTES:
        raise ValueError("The PDF exceeds the 50 MB inline-data limit.")


def safe_stem(pdf_path: Path) -> str:
    cleaned = "".join(
        character if character.isalnum() or character in "-_" else "_"
        for character in pdf_path.stem
    )
    return cleaned or "paper"


def document_block(pdf_path: Path) -> dict[str, str]:
    encoded_pdf = base64.b64encode(pdf_path.read_bytes()).decode("utf-8")
    return {
        "type": "document",
        "data": encoded_pdf,
        "mime_type": "application/pdf",
    }


def require_output_text(interaction, stage_name: str) -> str:
    text = interaction.output_text
    if not text or not text.strip():
        raise RuntimeError(f"{stage_name} returned no text output.")
    return text.strip()


def analyze_paper(client, pdf_path: Path) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=[
            document_block(pdf_path),
            {"type": "text", "text": ANALYSIS_PROMPT},
        ],
    )
    return require_output_text(interaction, "Paper analysis")


def write_text(path: Path, content: str) -> None:
    path.write_text(content.rstrip() + "\n", encoding="utf-8")


def main() -> int:
    args = parse_args()
    pdf_path = args.pdf.expanduser().resolve()

    try:
        if not os.getenv("GEMINI_API_KEY"):
            raise ValueError("Set GEMINI_API_KEY in the current terminal session.")

        validate_pdf(pdf_path)
        output_dir = Path("outputs") / safe_stem(pdf_path)
        output_dir.mkdir(parents=True, exist_ok=True)

        with genai.Client() as client:
            print("1/3 Analyzing the paper and extracting its reported result...")
            brief = analyze_paper(client, pdf_path)
            write_text(output_dir / "paper_brief.md", brief)

        print(f"Complete. Open {output_dir} to inspect the evidence bundle.")
        return 0

    except errors.APIError as error:
        print(
            f"Gemini API error {error.code}: {error.message}",
            file=sys.stderr,
        )
        return 1
    except (OSError, RuntimeError, ValueError) as error:
        print(f"Error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

How to use this reference

Compare the reference against your saved file from top to bottom. Keep your session-only API key out of the source code.

Before you run the agent, do you expect the brief to report one isolated number or a number tied to specific test conditions?

  • Run the paper analysis from the activated Terminal session by using this command:
python paper_agent.py paper.pdf

What should the terminal show?

The terminal first reports that the paper analysis has started. The final line points you to outputs/paper when the input file is named paper.pdf.

The request can take a little while because Gemini reads the full paper. A quiet terminal during that request is normal.

  • Return to the VS Code window from earlier.
  • Expand the outputs folder in the left sidebar.
  • Expand the paper folder.
  • Open paper_brief.md.
  • Confirm the brief identifies one reported result.
  • Confirm the brief records the metric plus its units.
  • Confirm the brief records the original evaluation conditions.
  • Confirm unsupported details appear as missing information or interpretation.

What proves the extraction worked?

The brief should contain the required headings from Paper through Comparison Risks. Its reproduction target should connect the reported value to the source paper's input plus evaluation procedure.

That evidence boundary gives the later prototype a precise result to test.

Did the analysis fail?

If the script reports a missing PDF, confirm paper.pdf is inside the same project folder as paper_agent.py.

If the script reports a Gemini API problem, confirm the current Terminal session still has the exported credential. Re-export it in that session if the terminal was restarted.

Help me diagnose the paper analysis failure.

You now have a paper result backed by its measurement context. Next, you'll ask Gemini for a prototype while keeping that unexecuted draft separate from observed evidence.

Generate an Unverified Draft

Your paper brief now isolates a measurable result from the PDF. The next challenge is turning that evidence into a small experiment without confusing proposed code with observed evidence.

A Gemini interaction can produce a convincing Python prototype from the brief. The draft remains a proposal until its output has been observed.

In this step, get ready to:
  • Define a prompt that requests a complete but unverified prototype.
  • Generate the draft from the extracted paper brief.
  • Inspect the draft for assumptions that prevent a fair result comparison.
Define the unverified draft request

The draft prompt sets a strict evidence boundary. It asks for a complete demonstration while forbidding claims that the program has run or reproduced the paper.

  • Return to paper_agent.py in Visual Studio Code.
  • Locate the closing """.strip() line for ANALYSIS_PROMPT.
  • Add the draft prompt directly below that line by copying this code:
DRAFT_PROMPT = """
You are proposing an implementation from a research-paper brief.

Create a complete Python draft that demonstrates the paper's core method on a small input. Explain the planned metric and expected result. Do not claim that the code has been executed or that the paper has been reproduced.

Paper brief:

{brief}
""".strip()

What does this prompt control?

  • The {brief} placeholder gives the model the extracted result and its evaluation conditions.
  • The request asks for a complete program that demonstrates the paper's core method on a small input.
  • The evidence rule prevents the draft from claiming execution or reproduction.
  • Save paper_agent.py.
  • Confirm the saved prompt includes the planned metric requirement and the execution restriction.

Seeing a string or indentation problem?

Check that DRAFT_PROMPT begins and ends with three double quotes. Keep .strip() immediately after the closing quotes.

Ask for help with the prompt syntax.

Generate the draft from the brief

A separate interaction keeps document analysis distinct from prototype generation. The second interaction receives only the structured brief.

  • Locate analyze_paper in paper_agent.py.
  • Add generate_draft directly below the completed analyze_paper function by copying this code:
def generate_draft(client, brief: str) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=DRAFT_PROMPT.format(brief=brief),
    )
    return require_output_text(interaction, "Draft generation")

What does this function do?

  • The function formats DRAFT_PROMPT with the paper brief produced by analyze_paper.
  • The interaction uses the existing MODEL_ID for a text-only draft request.
  • The require_output_text helper rejects an empty response before the CLI saves a file.
  • Save paper_agent.py.
  • Confirm generate_draft sits between analyze_paper and write_text.

Does the function look incomplete?

Confirm every line inside generate_draft uses four spaces of indentation. Check that the final line returns the result from require_output_text.

Ask for help checking the function structure.

  • Scroll to the main function in paper_agent.py.
  • Find these existing lines inside the client context:
brief = analyze_paper(client, pdf_path)
write_text(output_dir / "paper_brief.md", brief)

Where does the draft begin?

These lines finish the analysis stage. The brief variable now holds the evidence that the draft interaction needs.

  • Keep the existing analysis lines in place.
  • Add the draft-generation lines directly below them so the block matches this code:
brief = analyze_paper(client, pdf_path)
write_text(output_dir / "paper_brief.md", brief)

print("2/3 Generating an explicitly unverified implementation draft...")
draft = generate_draft(client, brief)
write_text(output_dir / "unverified_draft.md", draft)

What changes in the workflow?

  • The terminal label makes the draft's unverified status visible during the run.
  • The brief value becomes the input to generate_draft.
  • The returned text is saved as unverified_draft.md beside paper_brief.md.
  • Save paper_agent.py.

✔️ Awesome, I've got everything!

Your paper_agent.py file now contains the analysis stage and the unverified draft stage.

ⓧ I'd like to double check the full code

import argparse
import base64
import os
import sys
from pathlib import Path

from google import genai
from google.genai import errors

MODEL_ID = "gemini-3.8-flash"
MAX_PDF_BYTES = 50_000_000

ANALYSIS_PROMPT = """
Read this research paper as a reproducibility analyst.

Return a concise Markdown brief with exactly these headings:

# Paper
## Core Method
## Reproduction Target
## Original Reported Result
## Metric and Units
## Original Evaluation Conditions
## Required Inputs and Dependencies
## Missing or Ambiguous Details
## Minimal Prototype Plan
## Comparison Risks

Requirements:
- Select one measurable headline result that a small Python prototype could test.
- Quote the reported value, metric, units, dataset or input, scale, baseline, and evaluation procedure when the paper states them.
- Distinguish statements found in the paper from your interpretation.
- Write "not stated" for information the paper does not provide.
- If a faithful reproduction requires unavailable data, specialized hardware, long training, or proprietary dependencies, say so explicitly.
- Do not invent a numeric result or experimental condition.
""".strip()

DRAFT_PROMPT = """
You are proposing an implementation from a research-paper brief.

Create a complete Python draft that demonstrates the paper's core method on a small input. Explain the planned metric and expected result. Do not claim that the code has been executed or that the paper has been reproduced.

Paper brief:

{brief}
""".strip()


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Turn a research-paper PDF into a tested Python prototype."
    )
    parser.add_argument("pdf", type=Path, help="Path to a public PDF under 50 MB")
    return parser.parse_args()


def validate_pdf(pdf_path: Path) -> None:
    if not pdf_path.is_file():
        raise ValueError(f"PDF not found: {pdf_path}")
    if pdf_path.suffix.lower() != ".pdf":
        raise ValueError("The input file must have a .pdf extension.")
    size = pdf_path.stat().st_size
    if size == 0:
        raise ValueError("The PDF is empty.")
    if size > MAX_PDF_BYTES:
        raise ValueError("The PDF exceeds the 50 MB inline-data limit.")


def safe_stem(pdf_path: Path) -> str:
    cleaned = "".join(
        character if character.isalnum() or character in "-_" else "_"
        for character in pdf_path.stem
    )
    return cleaned or "paper"


def document_block(pdf_path: Path) -> dict[str, str]:
    encoded_pdf = base64.b64encode(pdf_path.read_bytes()).decode("utf-8")
    return {
        "type": "document",
        "data": encoded_pdf,
        "mime_type": "application/pdf",
    }


def require_output_text(interaction, stage_name: str) -> str:
    text = interaction.output_text
    if not text or not text.strip():
        raise RuntimeError(f"{stage_name} returned no text output.")
    return text.strip()


def analyze_paper(client, pdf_path: Path) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=[
            document_block(pdf_path),
            {"type": "text", "text": ANALYSIS_PROMPT},
        ],
    )
    return require_output_text(interaction, "Paper analysis")


def generate_draft(client, brief: str) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=DRAFT_PROMPT.format(brief=brief),
    )
    return require_output_text(interaction, "Draft generation")


def write_text(path: Path, content: str) -> None:
    path.write_text(content.rstrip() + "\n", encoding="utf-8")


def main() -> int:
    args = parse_args()
    pdf_path = args.pdf.expanduser().resolve()

    try:
        if not os.getenv("GEMINI_API_KEY"):
            raise ValueError("Set GEMINI_API_KEY in the current terminal session.")

        validate_pdf(pdf_path)
        output_dir = Path("outputs") / safe_stem(pdf_path)
        output_dir.mkdir(parents=True, exist_ok=True)

        with genai.Client() as client:
            print("1/3 Analyzing the paper and extracting its reported result...")
            brief = analyze_paper(client, pdf_path)
            write_text(output_dir / "paper_brief.md", brief)

            print("2/3 Generating an explicitly unverified implementation draft...")
            draft = generate_draft(client, brief)
            write_text(output_dir / "unverified_draft.md", draft)

        print(f"Complete. Open {output_dir} to inspect the evidence bundle.")
        return 0

    except errors.APIError as error:
        print(
            f"Gemini API error {error.code}: {error.message}",
            file=sys.stderr,
        )
        return 1
    except (OSError, RuntimeError, ValueError) as error:
        print(f"Error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

How should I use this reference?

Compare this reference with your saved file from top to bottom. At this checkpoint, the workflow ends after writing unverified_draft.md.

Run and inspect the unverified draft

The new workflow produces source code and a planned metric. This run tests whether the draft looks plausible while preserving the difference between a proposal and observed output.

Before you run this, do you expect a generated program to count as observed evidence?

  • Run the updated agent against paper.pdf by using this command:
python paper_agent.py paper.pdf

What does this command do?

The command analyzes paper.pdf first. It then passes the saved brief into the separate draft-generation interaction.

You will see the terminal label 2/3 Generating an explicitly unverified implementation draft... before the completion message. That wording keeps the evidence status visible.

  • Return to the file list in Visual Studio Code.
  • Expand the outputs folder.
  • Select unverified_draft.md in the folder named after your paper.
  • Locate the planned metric and expected result in the draft.
  • Identify one assumption that the paper brief does not support.
  • Note any original data that the draft replaces or omits.
  • Compare the proposed test conditions with the paper's evaluation conditions.
  • Confirm execution.log, prototype.py, and comparison.md are absent.

You now have plausible code with no observed prototype output. That shortfall is intentional because the draft cannot support a result comparison on its own.

Is the draft missing?

Confirm the current Terminal session still has GEMINI_API_KEY available. Check that paper.pdf remains in the same project folder as paper_agent.py.

Ask for help diagnosing the missing draft.

Strong work. Your workflow now preserves generated code without presenting it as evidence. Next, you will run the prototype and capture the output needed for a fair comparison.

Execute and Compare the Prototype

You now have an extracted paper result plus an unverified draft. The missing evidence is an observed prototype result produced by executing the generated program.

In this step, Gemini uses sandboxed code execution to run the prototype. Your agent captures each attempt before judging whether the result can be compared fairly with the paper.

In this step, get ready to:
  • Require a complete executable prototype with measurable output.
  • Capture every code attempt plus its observed result.
  • Save an evidence bundle with a qualified comparison verdict.
Require an executed comparison

The unverified draft describes a possible implementation. The next prompt turns that proposal into a runnable experiment with explicit rules for evidence quality.

  • In paper_agent.py, add the execution prompt directly below DRAFT_PROMPT by copying this code:
PROTOTYPE_PROMPT = """
You are the execution and comparison stage of a research-paper reproduction agent.

Use the paper brief and unverified draft below to build and run a minimal Python prototype.

Execution rules:
- Use synthetic data if the original data is unavailable.
- Keep execution comfortably below 30 seconds.
- Use only Python and libraries already available in the code-execution environment.
- Every code execution attempt must contain a complete standalone program, not a patch or fragment.
- Print the prototype metric, value, units, test conditions, and any random seed.
- If execution fails, repair the complete program and run it again.

Comparison rules:
- Compare the paper's reported result with the final observed prototype result.
- Calculate an absolute difference or relative percentage difference only when the metric, units, and evaluation conditions are aligned.
- Label the comparison DIRECTLY COMPARABLE, DIRECTIONALLY CONSISTENT, or NOT DIRECTLY COMPARABLE.
- Never describe a toy or scaled experiment as a reproduction of the full paper.
- End with a concise Markdown report containing Original Result, Prototype Result, Comparison Verdict, Reproduction Gaps, and Next Experiment.

Paper brief:

{brief}

Unverified draft:

{draft}
""".strip()

How does the prompt protect the comparison?

  • The execution rules require a complete standalone program on every attempt.
  • The printed output must identify the metric plus its value.
  • The printed output must also identify the units plus the test conditions.
  • The comparison rules permit a numerical difference only when the evidence aligns.
  • Add test_and_compare directly below generate_draft by copying this code:
def test_and_compare(client, brief: str, draft: str):
    return client.interactions.create(
        model=MODEL_ID,
        input=PROTOTYPE_PROMPT.format(brief=brief, draft=draft),
        tools=[{"type": "code_execution"}],
    )

What does this function do?

  • A separate text-only interaction keeps PDF extraction apart from prototype execution.
  • The formatted prompt supplies the extracted brief plus the unverified draft.
  • The tools=[{"type": "code_execution"}] setting enables Python execution in Gemini's managed environment.
  • Save paper_agent.py.
  • Confirm that the editor shows no Python syntax errors for PROTOTYPE_PROMPT or test_and_compare.

Seeing a syntax error?

Check that the prompt starts plus ends with three double quotes. Confirm that test_and_compare begins at the left edge of the file.

Help me fix the new prompt or function

Capture every execution attempt

A final answer alone cannot prove that generated code ran. An execution trace preserves the program attempts plus the output returned from each attempt.

  • Add collect_execution_trace directly below test_and_compare by copying this code:
def collect_execution_trace(interaction) -> tuple[str, str]:
    code_attempts: list[str] = []
    trace_parts: list[str] = []

    for step in interaction.steps or []:
        if step.type == "code_execution_call":
            code = step.arguments.code or ""
            code_attempts.append(code)
            trace_parts.append(
                f"## Code attempt {len(code_attempts)}\n\n```python\n{code}\n```"
            )
        elif step.type == "code_execution_result":
            status = "ERROR" if step.is_error else "OK"
            trace_parts.append(
                f"## Execution result: {status}\n\n```text\n{step.result}\n```"
            )

    if not code_attempts:
        raise RuntimeError("Gemini returned no code-execution attempt.")
    if not any(step.type == "code_execution_result" for step in interaction.steps or []):
        raise RuntimeError("Gemini returned no code-execution result.")

    return code_attempts[-1], "\n\n".join(trace_parts)

How is the evidence collected?

  • Each code_execution_call contributes one complete Python attempt to the trace.
  • Each code_execution_result contributes observed output marked as OK or ERROR.
  • The final code attempt becomes the saved prototype.py program.
  • Missing execution evidence raises an error instead of producing an unsupported comparison.
  • Save paper_agent.py.
  • Confirm that the editor shows no Python syntax errors inside collect_execution_trace.

Is the trace function highlighted as invalid?

Check the indentation beneath the for statement. Confirm that both final checks remain outside the loop.

Help me repair the trace function

Complete and run the evidence workflow

The final workflow turns the trace into three new artifacts. These files separate executable code from observed output plus the model's interpretation.

  • In main, replace the existing genai.Client() block with this completed workflow:
        with genai.Client() as client:
            print("1/3 Analyzing the paper and extracting its reported result...")
            brief = analyze_paper(client, pdf_path)
            write_text(output_dir / "paper_brief.md", brief)

            print("2/3 Generating an explicitly unverified implementation draft...")
            draft = generate_draft(client, brief)
            write_text(output_dir / "unverified_draft.md", draft)

            print("3/3 Executing a prototype and comparing observed results...")
            interaction = test_and_compare(client, brief, draft)
            prototype_code, execution_log = collect_execution_trace(interaction)
            comparison = require_output_text(interaction, "Result comparison")

            write_text(output_dir / "prototype.py", prototype_code)
            write_text(output_dir / "execution.log", execution_log)
            write_text(output_dir / "comparison.md", comparison)

What does the completed workflow save?

  • The first stage refreshes paper_brief.md from the public PDF.
  • The second stage refreshes the explicitly unverified unverified_draft.md.
  • The third stage executes the prototype before collecting its evidence.
  • The three final writes separate code from execution history plus comparison reasoning.
  • Under the client block, make the successful completion path match this code:
        print(f"Complete. Open {output_dir} to inspect the evidence bundle.")
        return 0

Why finish with the output path?

The completion message identifies the exact folder containing the evidence bundle. A zero return value records that every stage completed successfully.

  • Save paper_agent.py.

✔️ Awesome, I've got everything!

Your saved CLI now includes execution plus comparison as its third stage.

ⓧ I'd like to double check the full code

import argparse
import base64
import os
import sys
from pathlib import Path

from google import genai
from google.genai import errors

MODEL_ID = "gemini-3.8-flash"
MAX_PDF_BYTES = 50_000_000

ANALYSIS_PROMPT = """
Read this research paper as a reproducibility analyst.

Return a concise Markdown brief with exactly these headings:

# Paper
## Core Method
## Reproduction Target
## Original Reported Result
## Metric and Units
## Original Evaluation Conditions
## Required Inputs and Dependencies
## Missing or Ambiguous Details
## Minimal Prototype Plan
## Comparison Risks

Requirements:
- Select one measurable headline result that a small Python prototype could test.
- Quote the reported value, metric, units, dataset or input, scale, baseline, and evaluation procedure when the paper states them.
- Distinguish statements found in the paper from your interpretation.
- Write "not stated" for information the paper does not provide.
- If a faithful reproduction requires unavailable data, specialized hardware, long training, or proprietary dependencies, say so explicitly.
- Do not invent a numeric result or experimental condition.
""".strip()

DRAFT_PROMPT = """
You are proposing an implementation from a research-paper brief.

Create a complete Python draft that demonstrates the paper's core method on a small input. Explain the planned metric and expected result. Do not claim that the code has been executed or that the paper has been reproduced.

Paper brief:

{brief}
""".strip()

PROTOTYPE_PROMPT = """
You are the execution and comparison stage of a research-paper reproduction agent.

Use the paper brief and unverified draft below to build and run a minimal Python prototype.

Execution rules:
- Use synthetic data if the original data is unavailable.
- Keep execution comfortably below 30 seconds.
- Use only Python and libraries already available in the code-execution environment.
- Every code execution attempt must contain a complete standalone program, not a patch or fragment.
- Print the prototype metric, value, units, test conditions, and any random seed.
- If execution fails, repair the complete program and run it again.

Comparison rules:
- Compare the paper's reported result with the final observed prototype result.
- Calculate an absolute difference or relative percentage difference only when the metric, units, and evaluation conditions are aligned.
- Label the comparison DIRECTLY COMPARABLE, DIRECTIONALLY CONSISTENT, or NOT DIRECTLY COMPARABLE.
- Never describe a toy or scaled experiment as a reproduction of the full paper.
- End with a concise Markdown report containing Original Result, Prototype Result, Comparison Verdict, Reproduction Gaps, and Next Experiment.

Paper brief:

{brief}

Unverified draft:

{draft}
""".strip()


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Turn a research-paper PDF into a tested Python prototype."
    )
    parser.add_argument("pdf", type=Path, help="Path to a public PDF under 50 MB")
    return parser.parse_args()


def validate_pdf(pdf_path: Path) -> None:
    if not pdf_path.is_file():
        raise ValueError(f"PDF not found: {pdf_path}")
    if pdf_path.suffix.lower() != ".pdf":
        raise ValueError("The input file must have a .pdf extension.")
    size = pdf_path.stat().st_size
    if size == 0:
        raise ValueError("The PDF is empty.")
    if size > MAX_PDF_BYTES:
        raise ValueError("The PDF exceeds the 50 MB inline-data limit.")


def safe_stem(pdf_path: Path) -> str:
    cleaned = "".join(
        character if character.isalnum() or character in "-_" else "_"
        for character in pdf_path.stem
    )
    return cleaned or "paper"


def document_block(pdf_path: Path) -> dict[str, str]:
    encoded_pdf = base64.b64encode(pdf_path.read_bytes()).decode("utf-8")
    return {
        "type": "document",
        "data": encoded_pdf,
        "mime_type": "application/pdf",
    }


def require_output_text(interaction, stage_name: str) -> str:
    text = interaction.output_text
    if not text or not text.strip():
        raise RuntimeError(f"{stage_name} returned no text output.")
    return text.strip()


def analyze_paper(client, pdf_path: Path) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=[
            document_block(pdf_path),
            {"type": "text", "text": ANALYSIS_PROMPT},
        ],
    )
    return require_output_text(interaction, "Paper analysis")


def generate_draft(client, brief: str) -> str:
    interaction = client.interactions.create(
        model=MODEL_ID,
        input=DRAFT_PROMPT.format(brief=brief),
    )
    return require_output_text(interaction, "Draft generation")


def test_and_compare(client, brief: str, draft: str):
    return client.interactions.create(
        model=MODEL_ID,
        input=PROTOTYPE_PROMPT.format(brief=brief, draft=draft),
        tools=[{"type": "code_execution"}],
    )


def collect_execution_trace(interaction) -> tuple[str, str]:
    code_attempts: list[str] = []
    trace_parts: list[str] = []

    for step in interaction.steps or []:
        if step.type == "code_execution_call":
            code = step.arguments.code or ""
            code_attempts.append(code)
            trace_parts.append(
                f"## Code attempt {len(code_attempts)}\n\n```python\n{code}\n```"
            )
        elif step.type == "code_execution_result":
            status = "ERROR" if step.is_error else "OK"
            trace_parts.append(
                f"## Execution result: {status}\n\n```text\n{step.result}\n```"
            )

    if not code_attempts:
        raise RuntimeError("Gemini returned no code-execution attempt.")
    if not any(step.type == "code_execution_result" for step in interaction.steps or []):
        raise RuntimeError("Gemini returned no code-execution result.")

    return code_attempts[-1], "\n\n".join(trace_parts)


def write_text(path: Path, content: str) -> None:
    path.write_text(content.rstrip() + "\n", encoding="utf-8")


def main() -> int:
    args = parse_args()
    pdf_path = args.pdf.expanduser().resolve()

    try:
        if not os.getenv("GEMINI_API_KEY"):
            raise ValueError("Set GEMINI_API_KEY in the current terminal session.")

        validate_pdf(pdf_path)
        output_dir = Path("outputs") / safe_stem(pdf_path)
        output_dir.mkdir(parents=True, exist_ok=True)

        with genai.Client() as client:
            print("1/3 Analyzing the paper and extracting its reported result...")
            brief = analyze_paper(client, pdf_path)
            write_text(output_dir / "paper_brief.md", brief)

            print("2/3 Generating an explicitly unverified implementation draft...")
            draft = generate_draft(client, brief)
            write_text(output_dir / "unverified_draft.md", draft)

            print("3/3 Executing a prototype and comparing observed results...")
            interaction = test_and_compare(client, brief, draft)
            prototype_code, execution_log = collect_execution_trace(interaction)
            comparison = require_output_text(interaction, "Result comparison")

            write_text(output_dir / "prototype.py", prototype_code)
            write_text(output_dir / "execution.log", execution_log)
            write_text(output_dir / "comparison.md", comparison)

        print(f"Complete. Open {output_dir} to inspect the evidence bundle.")
        return 0

    except errors.APIError as error:
        print(
            f"Gemini API error {error.code}: {error.message}",
            file=sys.stderr,
        )
        return 1
    except (OSError, RuntimeError, ValueError) as error:
        print(f"Error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

What should the complete file contain?

This reference contains the cumulative CLI through the execution stage. It preserves the earlier PDF analysis plus draft generation before adding evidence capture.

Before you run the completed agent, which comparison verdict do you expect from your paper plus the small prototype?

  • Run the completed workflow against paper.pdf with this command:
python paper_agent.py paper.pdf

What should I see?

The terminal shows three numbered stages. The final message points to the paper-specific folder inside outputs.

The exact prototype code plus numerical result can vary between runs. Successful evidence capture is the stable checkpoint.

That closes the evidence gap. Your comparison now rests on executed code plus captured output instead of an untested proposal.

Did the completed run fail?

Confirm that the virtual environment remains activated. Check that the session-only API key remains available in the same Terminal session.

If the agent reports missing execution evidence, rerun it once. Gemini may repair a failed program with another complete attempt.

Help me diagnose the completed agent

  • Expand outputs in the VS Code Explorer sidebar.
  • Expand the paper-specific folder inside outputs.
  • Open execution.log from that folder.
  • Open prototype.py from that folder.
  • Open comparison.md from that folder.

How do the artifacts prove the result?

  • The execution.log file contains every attempted program plus each observed result.
  • The prototype.py file contains the final complete program executed by Gemini.
  • The comparison.md file places the original result beside the observed prototype result.
  • The verdict is DIRECTLY COMPARABLE, DIRECTIONALLY CONSISTENT, or NOT DIRECTLY COMPARABLE.

A trustworthy report calculates a difference only when the metric plus units align. The data plus scale plus evaluation procedure must align as well.

Secret mission

Secret Mission: Audit an Invalid Comparison

A close number can still be invalid when the data, scale, hardware, or procedure differs. Stress-test your agent with a second paper. Then document the fair experiment the paper would require.

Clean Up Your Resources

Clean Up Your Resources

This project used the Gemini Developer API free tier. Every remaining artifact lives on your Mac without ongoing cost.

Resources you used:

  • The current Terminal session holding the activated .venv environment.
  • The session-only GEMINI_API_KEY credential.
  • The local .venv virtual environment.
  • The completed paper_agent.py CLI.
  • The public input file paper.pdf.
  • The second public input file second_paper.pdf.
  • The first evidence bundle at outputs/<safe-paper-stem>.
  • The audited evidence bundle at outputs/<second-safe-paper-stem>.

Keep everything running

No action is needed. Choose this option if you want to compare more public papers or revisit the evidence bundles.

  • Keep the project folder in its current location.
  • Retain each folder under outputs as the evidence bundle for its paper.
  • Set GEMINI_API_KEY again before using the CLI in a new Terminal session.

Pause - I'll come back to this later

This option ends the active shell session. Your code files remain available for the next comparison.

  • Close the Terminal session from earlier.
  • Leave the project folder in its current location.

Closing Terminal ends the current .venv activation. The session-only GEMINI_API_KEY value is no longer available.

Delete - I don't want to use this again

This option clears the session credential. It also removes every local project artifact.

Deletion Is Permanent

This is the irreversible part. Your files remain recoverable until you empty Trash.

Closing Terminal removes the credential from the current shell session.

  • Close the Terminal session from earlier.
  • Click the Finder icon in the macOS Dock.
  • Search for paper_agent.py.
  • Open the project folder that contains the matching file.
  • Confirm that the folder contains .venv.
  • Confirm that the folder contains outputs.
  • Move the entire project folder to Trash.

What Gets Removed?

  • The local .venv environment.
  • The completed paper_agent.py CLI.
  • Both public PDF files.
  • Every evidence bundle under outputs.

The CLI sent each PDF as inline data. It created no Gemini Files API resource to delete.

  • Open Trash.
  • Confirm that the project folder is listed.
  • Empty Trash.
  • Search your Mac for paper_agent.py again.

You should see no matching project file. The local environment and both evidence bundles are now removed.

Nice Work!

Nice Work!

Excellent work! Your terminal-based comparison agent now tests a paper's reported result against observed prototype output. Its evidence bundle keeps generated drafts separate from executed results.

What you learned:

  • Used multimodal PDF analysis to extract a measurable result from a research paper. Your brief preserves the metric. It also records the units plus evaluation conditions.
  • Protected the evidence boundary by labelling generated code as unverified. You used sandboxed code execution to capture the final prototype plus its observed output.
  • Produced an evidence-based comparison that checks whether the metric matches. It also checks whether the test conditions align. The final verdict explains every reproduction gap.
  • Secret Mission: Audited a second paper with incompatible reproduction conditions. Your reproducibility audit rejected an invalid numeric match. It also documented the experiment required for a fair comparison.

Ready to quiz yourself?