Audit Apify Scrapers for AI Agents

Audit Apify Actors to catch scraper failures your AI agent might miss.

Introduction

30 Second Summary

A scraper can report success while giving an AI assistant nothing useful to work with. That quiet failure can turn a confident answer into a guess.

In this project, you will audit public Apify Actors with a local probe. You will finish with evidence that separates useful results from loud errors and silent failures.

What You'll Build

Your terminal will show a classified table that exposes which public Actors return useful data, reject weak input, or quietly report success with nothing worthwhile behind it.

By the end of this project, you'll have:

A single-Actor check that prints a clear classification for one scraper result in your terminal.
A multi-Actor sweep that sorts each result into REJECTED, LOUD-FAIL, SILENT-EMPTY, SILENT-STATS, or OK.
A saved audit report with one-line verdicts for every Actor and proof from the Apify Console that a successful run can still hold no useful data.
Secret Mission: Create a rerun mode that retests one silent failure with realistic input and compares both classifications.

Are there any prerequisites?

This project requires a free Apify account and an API token stored as APIFY_TOKEN. You also need Python 3.9 or newer on your laptop.

Before We Start

The probe can only classify an Actor when your laptop can run the downloaded script. A missing runtime would stop the audit before it reaches Apify.

You will confirm that your terminal can reach the internet. You will also prepare Python 3.9 or newer for the probe.

In this step, get ready to:

Open your laptop terminal.
Confirm that your internet connection can reach the Python website.
Confirm that Python 3.9 or newer is available.

Prepare your command line

The terminal gives you a place to run the probe in later steps. Your operating system includes one by default.

Press Cmd+Space on macOS or the Windows key on Windows.
Type Terminal on macOS or PowerShell on Windows.
Press Enter.

You should see a command prompt where you can type commands. This is the workspace that runs the probe.

Confirm your internet connection by opening the official Python downloads page in your browser.

You should see the Python downloads page load with options for your operating system. That page confirms your laptop can reach the resources used in this project.

Check your Python runtime

A runtime reads the downloaded probe script before executing its instructions. The probe requires Python 3.9 or newer.

macOS

Before you run the check, do you expect your Mac to have a supported Python version already?

Check your installed Python version by running this command:

python3 --version

What does this command check?

The --version option prints the version used by the python3 command. The first two numbers tell you whether the runtime meets the probe's minimum.

Windows

Before you run the check, do you expect Windows to find a supported Python runtime?

Check your installed Python version by running this command:

python --version

What does this command check?

The --version option asks the current python command to print its version. The first two numbers show whether the runtime is recent enough for the probe.

✔️ I see Python 3.9 or higher

Your runtime meets the probe's requirement. Keep this terminal available for the next step.

ⓧ I see an older version

A version below 3.9 cannot run the probe reliably. Install a current Python release for your operating system.

macOS

Open the official Python releases for macOS page.
Select Download macOS installer beside the latest stable release.
Double-click the downloaded .pkg file in your Downloads folder.
Click Continue on each installer screen until you reach the installation screen.
Click Install.
Enter your macOS administrator password when prompted.

Installation may take a minute while macOS copies the runtime. You should see a successful installation message when it finishes.

Complete the certificate setup

The installer opens a Python folder after it finishes. The Install Certificates.command file prepares Python to make secure web requests.

Double-click Install Certificates.command in the Python folder.
Quit Terminal after the certificate command finishes.
Press Cmd+Space.
Type Terminal.
Press Enter.

Before you check again, do you expect the new runtime to satisfy the minimum version?

Verify the new Python version by running this command:

python3 --version

What does this verification prove?

The command prints the Python version selected by your refreshed terminal. A version of 3.9 or higher can run the probe.

You should now see Python 3.9 or higher.

Still seeing the older version?

Quit every Terminal window before opening a fresh one. This gives the updated shell path a chance to take effect.

Use the full version printed by the installer if your Mac keeps selecting its older system runtime. Help me select the newly installed Python runtime.

Windows

Open the official Python releases for Windows page.
Select the latest Python install manager.
Open the downloaded installer file.
Select Install.

The install manager adds Python management commands to Windows. Use it to install a supported runtime next.

Install Python 3.15 by running this command in PowerShell:

py install 3.15

What does this command do?

The py install command asks the Python install manager to add the 3.15 runtime. That release is newer than the probe's 3.9 minimum.

Before you verify the installation, do you expect the new runtime to appear as your current Python version?

List your installed Python runtimes by running this command:

py list

What does this verification prove?

The py list command shows the Python runtimes managed on your computer. You should see a 3.15 runtime in the output.

You should now see Python 3.15 listed as an installed runtime.

Install command not working?

Close PowerShell after installing the Python install manager. Open a fresh PowerShell window before trying the command again.

Existing Python path changes can also hide the new manager. Help me fix the Python install manager on Windows.

ⓧ Command not found

Your terminal cannot find a Python runtime yet. Install a current release before continuing.

macOS

Open the official Python releases for macOS page.
Select Download macOS installer beside the latest stable release.
Double-click the downloaded .pkg file in your Downloads folder.
Click Continue on each installer screen until you reach the installation screen.
Click Install.
Enter your macOS administrator password when prompted.

Installation may take a minute while macOS copies the runtime. You should see a successful installation message when it finishes.

Complete the certificate setup

The installer opens a Python folder after it finishes. The Install Certificates.command file prepares Python to make secure web requests.

Double-click Install Certificates.command in the Python folder.
Quit Terminal after the certificate command finishes.
Press Cmd+Space.
Type Terminal.
Press Enter.

Before you check the installation, do you expect Terminal to find the new runtime?

Verify the Python installation by running this command:

python3 --version

What does this verification prove?

The command prints the Python version selected by your refreshed terminal. A version of 3.9 or higher can run the probe.

You should now see Python 3.9 or higher.

Python still not found?

Quit every Terminal window before opening a fresh one. This allows the installer updates to reach your new shell session.

Confirm that the installer reported a successful installation. Help me make Python available in Terminal.

Windows

Open the official Python releases for Windows page.
Select the latest Python install manager.
Open the downloaded installer file.
Select Install.

The install manager adds Python management commands to Windows. Use it to install a supported runtime next.

Install Python 3.15 by running this command in PowerShell:

py install 3.15

What does this command do?

The py install command asks the Python install manager to add the 3.15 runtime. That release is newer than the probe's 3.9 minimum.

Before you verify the installation, do you expect the new runtime to appear in the installed versions?

List your installed Python runtimes by running this command:

py list

What does this verification prove?

The py list command shows the Python runtimes managed on your computer. You should see a 3.15 runtime in the output.

You should now see Python 3.15 listed as an installed runtime.

Python still not available?

Close PowerShell after installing the Python install manager. Open a fresh PowerShell window before trying the commands again.

Existing Python path changes can also hide the new manager. Help me finish the Python installation on Windows.

That foundation is ready. Your terminal can reach the internet through your browser verification. It can also run the Python probe.

Your laptop is ready for the audit. Next, you will set up Apify access so the probe can call its first Actor.

Create Your Apify Token

Your probe cannot call Apify Actors until Apify can identify the request. An API token gives the probe authenticated access to your account.

This step puts the token into APIFY_TOKEN in your active terminal session. The probe can then read the credential without storing it in a project file.

In this step, get ready to:

Create or access a free Apify account.
Generate an API token in Apify Console settings.
Export the token as APIFY_TOKEN in your active terminal session.

Create or access your Apify account

Apify Console is the web application where you manage your account credentials. You need to reach your account home screen before you can create the token.

Visit the Apify Console page.
Create a free account if you do not already have one.
Sign in to Apify Console with your account.

You will see your account home screen after authentication succeeds. This confirms your account is ready for token management.

Generate your API token

The token lets programs act through your Apify account. A dedicated token description helps you identify this credential when you review it later.

Creating a credential deserves extra care. You will keep this token in the active terminal instead of a project file.

Select Settings in the left-side panel.
Select the API & Integrations tab.
Create a new token from the API token management area.
Give the token a description that identifies this scraper probe.
Generate the token using the creation control on the page.
Copy the newly generated token to your clipboard.

Why Use a Dedicated Token?

Apify recommends using a separate token for each service. A clear description makes this probe's token easier to identify if you need to rotate or delete it later.

Export your token

An environment variable gives the probe access to your token through the terminal. This export lasts for the current terminal session.

Shell syntax can be fiddly here. Use the tab that matches the active terminal from earlier.

macOS

Return to the active terminal from earlier.
Replace your_apify_api_token_here in the command below with the token on your clipboard.
Export the token in the current session by running the edited command:

export APIFY_TOKEN=your_apify_api_token_here

What Does This Command Do?

The command creates APIFY_TOKEN in the current shell. It keeps the value in the active terminal environment instead of writing it to a project file.

The next check briefly reveals your token in the terminal. Keep the screen private while the credential is visible.

Before you run it, predict what the terminal prints:

Confirm the exported value by running:

echo $APIFY_TOKEN

What Should You See?

You should see the token you copied from Apify Console. This confirms APIFY_TOKEN is available in the active terminal session.

Windows

Return to the active PowerShell terminal from earlier.
Replace YOUR_TOKEN in the command below with the token on your clipboard.
Export the token in the current session by running the edited command:

$Env:APIFY_TOKEN = "YOUR_TOKEN"

What Does This Command Do?

The command creates APIFY_TOKEN in the current PowerShell session. It keeps the value in the active terminal environment instead of writing it to a project file.

The next check briefly reveals your token in PowerShell. Keep the screen private while the credential is visible.

Before you run it, predict what PowerShell prints:

Confirm the exported value by running:

$Env:APIFY_TOKEN

What Should You See?

You should see the token you copied from Apify Console. This confirms APIFY_TOKEN is available in the active PowerShell session.

Is the Output Blank?

Repeat the export command in the same terminal tab. A new terminal starts a separate session without the temporary value.

Check that APIFY_TOKEN uses the same capitalization shown above.

Help me troubleshoot why APIFY_TOKEN is empty in my active terminal session.

✔️ Awesome, I've got everything!

That is your authentication bridge ready. The active terminal can now provide your Apify token to the probe.

ⓧ I'd like to double check the full code

Your environment should now match this state.

APIFY_TOKEN.

It is exported in the active terminal environment with your newly generated Apify API token. The token value remains secret. It is not saved in project files.

Your account and active terminal are authenticated. Next, you will download the probe and use this token to classify your first Actor result.

Run the Probe Once

Your APIFY_TOKEN is ready for authenticated calls. A successful call still cannot tell an agent whether an Apify Actor returned useful data.

In this step, you will download the published probe. You will send minimal input through the MCP path to reveal how one public Actor reports its result.

In this step, get ready to:

Download the two probe files to your Desktop.
Run the probe against one public Apify Actor.
Classify the terminal result.

Download the probe files

The published probe uses one primary script plus a small helper. The helper reads APIFY_TOKEN from your active terminal environment.

Switch back to the active terminal from the previous step.
Move to your Desktop by running this command:

cd ~/Desktop

What does this command do?

The cd command changes the terminal location to your Desktop. This gives the downloaded files a predictable home.

Save the unchanged source files on your Desktop by running these commands:

curl -L https://raw.githubusercontent.com/rara-cyber/apify-agent-call-probe/main/agent_call_probe.py -o apify-probe.py
curl -L https://raw.githubusercontent.com/rara-cyber/apify-agent-call-probe/main/mcp_client.py -o mcp_client.py

What do these commands do?

The first command downloads the published probe logic unchanged. It saves the local file as apify-probe.py.
The second command downloads mcp_client.py. This helper makes the authenticated MCP request.
Neither command writes your APIFY_TOKEN value to a project file.

Confirm both files exist by running:

ls apify-probe.py mcp_client.py

What should I see?

You should see apify-probe.py plus mcp_client.py listed in the terminal. Their presence confirms the probe has everything it needs to run.

That download is sorted. The unchanged probe now sits on your Desktop with its required helper.

Files missing from the list?

Check that your laptop still has internet access. Repeat the download commands from your Desktop if either file is missing.

Help me diagnose a missing probe file.

Run one public Actor

Minimal input means an empty JSON object written as {}. The probe sends that input internally to reproduce what an agent may try when no extra context is available.

This is one real Actor run tied to your account. The terminal may stay quiet for a few minutes while the Actor finishes.

Predict whether the Actor rejects empty input or reports a completed run.
Run the probe against the public Actor by running:

python3 apify-probe.py sian.agency/apartments-com-property-scraper

What does this command do?

The python3 command executes your downloaded probe.
The Actor ID selects sian.agency/apartments-com-property-scraper as the target.
The probe reads APIFY_TOKEN from the active terminal environment.
The probe calls the Actor with {} before inspecting its status plus returned dataset.

You should see a terminal summary containing the Actor ID plus its status. The output also includes an item count followed by one classification.

How to read your verdict

REJECTED means MCP refused the minimal input before an Actor run started.
LOUD-FAIL means the Actor run failed visibly.
SILENT-EMPTY means the run succeeded with zero returned rows.
SILENT-STATS means the run succeeded with summary rows instead of useful data.
OK means dataset rows came back.

A classification other than REJECTED creates an Actor run that remains available in Apify Console. You now have your first classified result from the same minimal-input path an agent can use.

Probe did not return a classification?

Confirm you are using the active terminal where APIFY_TOKEN remains exported. Check that apify-probe.py plus mcp_client.py are together on your Desktop.

Help me troubleshoot the missing probe classification.

You have one classified baseline. Next, you will expand the same check across a list of public Actors.

Sweep Public Actors

Your first probe turned one Apify Actor run into a verdict an AI agent could use. That proves the classifier can separate a useful result from a misleading success.

One result cannot reveal a pattern across public scrapers. This step applies the same minimal input to a wider sample so quiet failures stand out.

In this step, get ready to:

Build an actors.txt list containing 8 to 10 public Actor URLs.
Run the unchanged probe against every Actor with minimal input.
Identify a SILENT-EMPTY result for the next inspection when the sweep produces one.

Build your Actor list

The Apify Store in Apify Console contains public Actors for different scraping jobs. A varied list gives the probe more chances to reveal different outcomes.

Your actors.txt file keeps one candidate on each line. Saving it beside apify-probe.py keeps the audit files together.

macOS

Press Cmd+Space to open Spotlight.
Type TextEdit into the search field.
Press Enter to open TextEdit.
Choose Format from the menu bar.
Select Make Plain Text.
Save the blank file as actors.txt in the same local folder as apify-probe.py.

You should see actors.txt as the document name in TextEdit.

Windows

Press the Windows key to open Search.
Type Notepad into the search field.
Press Enter to open Notepad.
Save the blank file as actors.txt in the same local folder as apify-probe.py.

You should see actors.txt as the document name in Notepad.

Switch back to the browser with Apify Console from earlier.
Open the public Apify Store.
Search for one scraping job that an AI agent might request.
Open one public Actor from the search results.
Select the Pricing tab on the Actor page.

Keep the Sample Comparable

A varied sample is more useful than several Actors built for the same job. Each candidate still receives the same minimal input of {}.

The Pricing details show whether an Actor adds a separate charge beyond ordinary platform usage. Excluding those Actors keeps this audit focused on behavior.

Exclude the candidate when its Pricing details require an extra paid plan or separate result charge.
Copy the candidate's Actor URL from the browser address bar.
Return to actors.txt in your text editor.
Paste the Actor URL onto a new line.
Repeat the selection process until the file contains 8 to 10 public Actor URLs.
Save actors.txt.
Count the non-empty lines in the editor.

Your candidate queue is ready. You should see 8 to 10 Actor URLs with one URL on each line.

Sweep every Actor

The unchanged probe sends the same minimal input to every candidate. Changing only the Actor URL makes each result comparable.

Each cloud run uses some of your Apify account allowance, so this is a sensible point to be deliberate. Your list contains only 8 to 10 candidates whose Pricing details you reviewed.

How the Verdicts Work

REJECTED means the Actor clearly refused the minimal input.
LOUD-FAIL means the run failed visibly.
SILENT-EMPTY means the run reported success without returning useful rows.
SILENT-STATS means the run reported success with statistics or error metadata instead of useful data.
OK means the Actor returned useful data that an agent could use.

Before you start the sweep, which classification do you expect to appear most often?

Return to the active terminal from your first probe run.
Press the Up Arrow key to restore the probe command from the previous step.
Replace the previous Actor URL with the first URL from actors.txt.
Press Enter to run the probe.
Wait for the probe to print a settled result row.

You should see the Actor identifier plus a run link or run ID. The row should also contain its classification, run status, and returned-data summary.

Press the Up Arrow key to restore the same probe command.
Replace the current Actor URL with the next URL from actors.txt.
Press Enter to start the next run.
Wait for the next settled result row.
Repeat this process for every remaining URL in actors.txt.

That is the biggest part of the audit complete. Your terminal should now contain 8 to 10 comparable rows from the unchanged probe.

Missing a Result Row?

Check that the Actor URL matches its line in actors.txt.
Keep using the active terminal session where APIFY_TOKEN remains exported.
Rerun only the Actor whose result row is missing.

Ask for help with a missing probe result.

Find a silent failure

A succeeded run status only confirms that execution ended normally. The classification reveals whether the run returned useful data.

Before you scan the table, do you expect every succeeded run to have an OK classification?

Scan the classification column for every SILENT-EMPTY row.
Compare each matching row with its run status.
Record one matching run link or run ID here when your sweep contains one: your SILENT-EMPTY run link or run ID.
Retain the complete table when your sample contains no SILENT-EMPTY result.

If SILENT-EMPTY appears beside a succeeded status, you have found the quiet failure this audit is designed to catch. A sweep without that verdict is still a valid record of your tested Actors.

The result table does not need your API token. Your screenshot can prove the sweep without exposing the secret.

Resize the terminal until all 8 to 10 result rows are visible.
Scroll any earlier token export command outside the visible terminal area.
Confirm that every row includes an Actor identifier.
Confirm that every row includes a run link or run ID.
Confirm that every row includes a classification.
Confirm that every row includes a run status.
Confirm that every row includes a concise returned-data summary.

Your terminal now shows the outcome of every Actor in actors.txt. Any SILENT-EMPTY row has enough run information for the next inspection.

Your sweep has turned a list of public scrapers into comparable evidence. Next, you will inspect one quiet failure in Apify Console to see what its succeeded status hides.

Inspect a Silent Empty Run

Your sweep has already flagged at least one Apify Actor as SILENT-EMPTY. The terminal classification is a clue that deserves a manual check.

An AI agent may treat a successful run status as proof that the scraper worked. The default dataset reveals whether the run produced useful results.

In this step, get ready to:

Open the run details for a result classified as SILENT-EMPTY.
Verify that the selected run has a successful status.
Inspect the default dataset for useful result data.

Open the selected run

The run ID connects a terminal row to the exact execution record in Apify Console. Matching the record prevents another run from muddying your evidence.

Long run IDs are fiddly to compare. A character-by-character match keeps this inspection tied to the result your probe classified.

Switch back to the sweep table in your terminal.
Locate one row classified as SILENT-EMPTY.
Keep the Actor identifier visible in the terminal.
Keep the run ID visible in the terminal.
Use the Apify run link in that row to open the exact run in your browser.
Switch back to the Apify Console from earlier if your table only shows a run ID.
Select Runs in the left-side panel if you are locating the run by its ID.
Select the run whose ID matches the terminal row.

You should see run details for the same Actor identifier shown in your terminal. The run ID should also match the selected row.

Cannot find the matching run?

Compare every character of the run ID with the value in your terminal table.
Use the Apify run link from the terminal when one is available.
Check the Actor identifier to rule out a similarly timed run from another Actor.

Ask for help with finding the exact run from my sweep table.

Verify the successful status

A run status describes where the execution finished in its lifecycle. The dataset provides separate evidence about the usefulness of its output.

Find the status displayed in the selected run's details.
Confirm that the status reads Succeeded.
Compare the displayed status with the status recorded in your terminal table.

You should see Succeeded in the run details. Your terminal row should report the same successful status.

What Does Succeeded Prove?

A successful status confirms that the Actor reached a successful terminal state. Output quality requires a separate check of the stored results.

Inspect the default dataset

Structured scraper results usually live in the run's default dataset. Opening that storage reveals what an agent actually receives after the successful run.

Before you inspect the stored results, do you expect the successful status to match a useful dataset?

Select the Storage tab in the run details.
Open the default Dataset.
Check the Table view for useful result rows.
Select the JSON view.
Check the JSON view for useful result objects.

You should see that the default dataset contains no useful result data. The run status still reads Succeeded.

You caught the hidden failure. The scraper completed successfully while giving an agent nothing useful to continue with.

Why This Is a Silent Failure

The Actor reported success at the lifecycle level. Its dataset gave the agent no useful result to work with.

That combination earns the SILENT-EMPTY classification. An agent that checks only the run status carries on with empty evidence.

Seeing Useful Result Rows?

Useful result rows suggest that you opened a different run or selected the wrong sweep result.

Compare the Console run ID with the run ID in the SILENT-EMPTY row.
Return to the sweep table if the IDs differ.
Open another result classified as SILENT-EMPTY.

Ask for help with checking whether this dataset contains useful scraper output.

Your manual check now proves why an agent needs more than a successful status. Next, you will compare this run with an OK run and a LOUD-FAIL run to see how each outcome looks in the Console.

Compare Failure Signals

Your SILENT-EMPTY run already proved that a successful status can hide a useless result.

An OK run pairs success with useful records. A LOUD-FAIL run exposes a problem through its status or error details.

In this step, you will compare both against the SILENT-EMPTY run in Apify Console. This reveals the signals an AI agent can miss when it trusts a scraper response.

In this step, get ready to:

Establish a healthy baseline from an OK run.
Identify visible failure evidence in a LOUD-FAIL run.
Compare both runs with your confirmed SILENT-EMPTY run.

Inspect an OK run

An OK run is your control case. Its status should agree with its useful dataset.

Switch back to the terminal table from your Actor sweep.
Find a row classified OK.
Open the Apify run link from that row in a new browser tab.
Confirm that the Actor identifier matches the selected OK row.
Inspect the run status in Apify Console.
Record the run status here: your OK run status.
Inspect the default dataset from the run page.
Summarize the useful result data here: summary of useful OK data.

You should see useful result records that match the Actor's purpose.

Good, you now have a healthy run that proves what usable success looks like.

OK Run Looks Empty?

An OK classification should have useful result data. A mismatch often means the browser is showing a different run.

Return to the sweep table in your terminal.
Reopen the link from the row classified OK.

Ask for help if the linked run still looks empty: help me check why my OK Apify run has no useful dataset records.

Inspect a LOUD-FAIL run

A LOUD-FAIL run makes its problem visible. Its Console page gives you a failure signal that an agent can detect directly.

Switch back to the terminal table from your Actor sweep.
Find a row classified LOUD-FAIL.
Open the Apify run link from that row in a new browser tab.
Confirm that the Actor identifier matches the selected LOUD-FAIL row.
Inspect the run status in Apify Console.
Record the run status here: your LOUD-FAIL run status.
Inspect the visible error details on the run page.
Summarize the failure evidence here: summary of loud failure details.

You should see a failed outcome supported by visible error information.

Can't Find the Loud Failure?

The Console page must belong to the LOUD-FAIL row from your sweep. A different run can show a different outcome.

Return to the sweep table in your terminal.
Reopen the link from the row classified LOUD-FAIL.

Ask for help if the failure signal is still unclear: help me interpret the visible failure signals in my LOUD-FAIL Apify run.

Compare all three signals

The three runs form a small test matrix. A trustworthy result needs every visible signal to agree.

Return to the SILENT-EMPTY run from your earlier inspection.
Inspect its run status again.
Record the status here: your SILENT-EMPTY run status.
Inspect its default dataset again.
Summarize the missing useful data here: summary of missing useful data.
Arrange the three run tabs side by side in your browser.

Before you make the final comparison, which run do you expect an agent to misread if it trusts the run status alone?

Compare your OK run status with your SILENT-EMPTY run status.
Compare summary of useful OK data with summary of missing useful data.
Contrast your LOUD-FAIL run status with both successful run statuses.
Match summary of loud failure details to the failed outcome.

What Does the Comparison Prove?

The OK run shows a successful status. Its dataset contains useful data.

The SILENT-EMPTY run reports Succeeded. Its dataset contains no useful result data.

The LOUD-FAIL run surfaces its problem through the Console status or error details.

An AI agent that trusts only a scraper's success signal can confuse OK with SILENT-EMPTY.

Strong work. You can now distinguish a healthy success from a silent failure even when both runs look successful at first glance.

You now know which Console signals separate useful output from silent failure. Next, you will turn the sweep results into a markdown report with a verdict for each Actor.

Write Your Actor Report

Your sweep has exposed how Apify Actors behave under minimal input. Your Apify Console comparison also proved that a successful status can hide useless data.

Terminal output is temporary. This step turns your evidence into a saved Markdown report that another person can audit before connecting an Actor to an AI agent.

In this step, get ready to:

Create the structure for actor-probe-report.md.
Add one result entry for every tested Actor.
Document the Console comparison with traceable run references.

Create the report structure

The report separates the sweep results from the evidence you confirmed by hand. Keeping it beside actors.txt makes the probe easy to rerun later.

Return to the plain-text editor you used for actors.txt.
Use the editor's new-file control to create a plain-text file in the same local folder as actors.txt.
Save the new file as actor-probe-report.md.

You should see actor-probe-report.md as the filename in your editor. This confirms the report exists in the folder that already holds your probe files.

Build the report outline with # Actor Probe Report as the title.
Add ## Sweep Results below the title.
Add ## Console Comparison below the sweep section.
Save actor-probe-report.md.

You should now see a title followed by two empty report sections. The first section holds every automated result.

What belongs in each section?

The sweep section records what the probe returned for each Actor. Every entry needs the Actor identifier plus its classification.

The comparison section records what you confirmed in the Console. Its run references connect your conclusions to the underlying evidence.

Can't find the report file?

Check the folder shown in your editor's save window. Choose the same local folder that contains actors.txt plus apify-probe.py.

Ask for help with locating the saved report.

Write one verdict for every Actor

A classification describes what happened during a run. A verdict explains whether that behavior is safe for an agent to trust.

Turn outcomes into verdicts

A REJECTED result clearly exposes missing input requirements. An agent can detect that rejection.
A LOUD-FAIL result exposes a visible failure. An agent can handle or report that error.
A SILENT-EMPTY result reports success without useful records. It requires dataset validation before an agent can trust it.
A SILENT-STATS result returns operational statistics without useful records. Treat it as a silent failure.
An OK result returns useful records under minimal input. Its output schema still needs review before production use.

What does each result entry need?

The Actor identifier exactly as it appears in the sweep table.
The classification assigned by the probe.
The Apify run link or run ID.
A one-sentence verdict that explains whether an agent can detect the outcome.

Switch back to the active terminal from your sweep.
Return to actor-probe-report.md after reading the first result row.
Write one four-part result entry for the first Actor using the fields above.
Save actor-probe-report.md.

You should see the first Actor under ## Sweep Results with a classification plus a traceable run reference.

Repeat the four-part result entry for every remaining row in the sweep table.
Save actor-probe-report.md.
Count the result entries under ## Sweep Results.
Compare that count with the number of Actors in actors.txt.

You should see 8 to 10 report entries. Every Actor from actors.txt should appear once.

Add the Console evidence and verify the report

The automated table shows what the probe classified. The Console evidence shows why those classifications matter.

Return to the browser tabs from your earlier Console comparison.
Return to actor-probe-report.md after reviewing the three runs.
Add a ### SILENT-EMPTY entry under ## Console Comparison using the manually confirmed run.
Record its Succeeded status plus the absence of useful dataset records.
Include the relevant run link or run ID in that entry.
Add a ### OK entry using the useful dataset you inspected.
Include the relevant run link or run ID in the OK entry.

You should now see two comparison entries with different evidence. One succeeded without useful data while the other produced useful records.

Add a ### LOUD-FAIL entry using the visible error details you inspected.
Include the relevant run link or run ID in the LOUD-FAIL entry.
Finish the comparison with one sentence explaining which signals an agent must inspect before trusting an Actor.

Your comparison should now cover the silent failure plus the useful run plus the loud failure. Each entry should point back to the run that supports it.

Save actor-probe-report.md.

Before you perform the final check, could another person trace every verdict to a specific Actor run without asking you for more context?

Use actors.txt from earlier as your final Actor checklist.
Confirm every tested Actor appears under ## Sweep Results.
Confirm every result entry contains one of the five classifications.
Confirm every result entry contains an Apify run link or run ID.
Confirm every result entry ends with a one-line verdict.
Confirm ## Console Comparison includes SILENT-EMPTY plus OK plus LOUD-FAIL evidence.

You should see 8 to 10 sweep entries with classifications plus run references plus one-line verdicts. The comparison should include the manually confirmed silent failure.

The editor should show the saved filename actor-probe-report.md. Your report now preserves the evidence that a success flag alone cannot provide.

✔️ My report has every result

Your report matches the completed audit. Keep it beside the probe so you can reassess these Actors before using them in an agent.

ⓧ I'd like to double check my report

Use this checklist to compare your saved report with the required result.

Confirm the saved filename is actor-probe-report.md.
Confirm every tested Actor has a classification plus a run reference.
Confirm every tested Actor has a one-line verdict.
Confirm the manually inspected SILENT-EMPTY run is identified as Succeeded without useful data.
Confirm the comparison includes notes from the OK run plus the LOUD-FAIL run.

That is your audit trail complete. You now have a reusable probe report that exposes loud failures plus the silent failures an agent could miss.

Secret mission

Build an Actor Admission Gate

Turn your sweep evidence into a reusable admission gate for AI agents. Test the gate against an OK run, a loud failure, and the silent failure you confirmed by hand.

Clean Up Your Resources

Keeping the completed probe files costs nothing. Use the tabs below to choose your cleanup level.

Resources you used:

An API token in Apify Console.
An active terminal session with APIFY_TOKEN exported.
The local probe script apify-probe.py.
The local Actor list actors.txt.
The completed report actor-probe-report.md.

Keep everything running

No action needed. Choose this if you plan to test more Actors or update the report.

Your Apify API token stays valid for future checks.
The APIFY_TOKEN value stays available while the active terminal session remains open.
Your local probe files stay ready for another sweep.
Your report keeps the classifications plus verdicts from this audit.

Pause - I'll come back to this later

Pausing ends the active terminal session. Your account token plus local files remain available.

Close the active terminal window to end its current session.

The APIFY_TOKEN export disappears with that session. A future terminal session needs a fresh export before the probe can call Apify Actors.

Your completed Console runs remain available for later comparison.

Delete - I don't want to use this again

Remove the local audit files plus the credential used by the probe. This gives you a clean starting point.

Deleting the token disables future probe calls that rely on it. Your local report stays unchanged until you remove the file yourself.

Delete the API token:

Return to Apify Console from earlier.
Return to the Console settings area where you generated the token.
Locate the API token you created for this project.
Use the deletion control beside that token.
Confirm the deletion.

The token no longer appears in your account's token list. Future probe calls need a new token.

Delete the local probe files:

macOS

Press Cmd+Space to open the macOS search bar.
Type Finder into the search bar.
Press Enter to open Finder.
Return to the folder containing apify-probe.py.
Select apify-probe.py, actors.txt, plus actor-probe-report.md.
Press Cmd+Delete to move the selected files to the Trash.

The three project files no longer appear in their original folder.

Windows

Press the Windows key to open the Windows search bar.
Type File Explorer into the search bar.
Press Enter to open File Explorer.
Return to the folder containing apify-probe.py.
Select apify-probe.py, actors.txt, plus actor-probe-report.md.
Press Delete to move the selected files to the Recycle Bin.

The three project files no longer appear in their original folder.

Close the active terminal window to discard its exported APIFY_TOKEN value.

That terminal session no longer exists. Its exported token value is gone.

Nice Work!

You caught the quiet failures! Your local probe now shows which Apify Actors an AI agent can trust.

You've learned how to:

Run a repeatable Actor sweep using the public Actors in actors.txt. Classify each response across five clear outcomes.
Expose silent scraper failures through manual inspection in Apify Console. Confirm that a run marked Succeeded can hold no useful data.
Preserve your findings in a reusable audit report named actor-probe-report.md. Record each run reference. Give every tested Actor a one-line verdict.
Extend the audit through the optional Secret Mission to push your scraper-testing skills further.

Ready to quiz yourself?