Scrape a Page with AI

Use AI and Apify to scrape a public page into a reusable CSV.

Introduction

30 Second Summary

Useful information often sits on web pages designed for reading. Copying each item into a spreadsheet takes time.

In this project, you will turn a public list page into a structured CSV by describing the fields you want. ChatGPT converts your description into an Apify scraper configuration that you test on two pages from the same site.

What You'll Build

You'll open a spreadsheet filled with structured rows from two matching pages on a real website.

By the end of this project, you'll have:

A successful single-page scrape where you can inspect the raw dataset before deciding which columns matter.
A reusable scraper configuration generated from your plain-language field description. You can check every requested field in the dataset.
A CSV export that opens with populated columns. The same fields line up when you run the configuration on a second page.
Secret Mission: An optional challenge to push your skills further.

Are there any prerequisites?

You'll need Apify and ChatGPT accounts plus an Apify token. Your target must be a public page with a list from a site whose terms permit automated access.

Before We Start

This workflow moves through three places in your browser. A public page supplies the source data.

ChatGPT turns your field description into configuration. Apify Console runs the scraper, so preparing both accounts keeps the rest of the project focused.

In this step, get ready to:

Confirm your browser can load a public web page.
Sign in to ChatGPT.
Sign in to Apify Console.

Check your browser access

Every part of this project runs through your browser. A successful public page load confirms that your browser can reach the services you need.

Press Cmd+Space on macOS or the Windows key on Windows to open system search.
Type the name of an installed web browser into system search.
Press Enter to launch the browser.
Open the public Apify web scraping guide in a new tab.

You should see an Apify article about using ChatGPT for web scraping. This confirms that your browser can access a public page.

Page Not Loading?

Check whether another public website loads in the same browser.
Temporarily disable any browser extension that blocks page content.
Help me troubleshoot public pages that will not load in my browser.

Sign in to ChatGPT

ChatGPT prepares the scraper configuration from the fields you describe. Your signed-in account keeps that conversation available while you inspect the scraped data.

Both account sign-ins happen on the services' official pages. Your credentials stay inside those pages.

Open the official ChatGPT account page in a new browser tab.
Choose your account method under Log in or sign up.
Complete the account prompts in your browser.
Open the profile menu in ChatGPT.
Confirm that the menu shows your active account.

That account is ready. ChatGPT can now hold the field description you create later in the project.

Having Trouble Signing In?

Use the same authentication method that you used when creating your ChatGPT account.
Try the official sign-in page in a private browser window if the page keeps returning to the login screen.
Help me troubleshoot my ChatGPT sign-in flow.

Sign in to Apify Console

Apify Console is where the scraper runs against a public page. It also holds the structured dataset produced by each run.

Open the official Apify Console sign-in page in a new browser tab.
Use the account creation link if this is your first visit.
Choose the account method tied to your Apify account.
Complete any account verification prompt shown in your browser.
Confirm that Apify Console takes you to the Dashboard.

Before you refresh both services, do you expect either one to ask you to sign in again?

Refresh the ChatGPT tab.

You should remain inside your signed-in ChatGPT workspace.

Refresh the Apify Console tab.

You should return to the Dashboard without another sign-in prompt. Both sides of your scraping workflow are now ready.

Console Returning to Sign-In?

Complete any pending email verification before refreshing Apify Console.
Return to the official sign-in page if your browser session has expired.
Help me troubleshoot my Apify Console sign-in flow.

Your browser sessions are ready. Next up, you'll choose a public list page. Then you'll run one controlled scrape.

Run a Raw Scrape

Before you ask ChatGPT to define useful fields, you need proof that the source can be reached. A raw run separates access problems from extraction problems.

An Apify Actor is a ready-made scraping program. Its dataset stores returned records in a table.

In this step, you'll run Web Scraper against the public Hacker News front page. The unrefined dataset becomes your baseline for the next configuration.

In this step, get ready to:

Confirm the public Hacker News front page is your source.
Prepare Apify Web Scraper for a one-page run.
Run the Actor to inspect its raw dataset.

Confirm the public source

A source page needs a repeated list that produces several comparable records. Hacker News provides a public list of ranked stories on one page.

Select the address bar in your browser.
Enter https://news.ycombinator.com/.
Press Enter.
Confirm the page shows a ranked list of story links.

You should see the Hacker News title near the top. Below it, you should see numbered story links with details such as points or comment counts.

Use Public Data Responsibly

This test sends a small request to a public page. A larger collection needs a review of the site's terms before it starts.

Confirm your token and choose Web Scraper

Apify uses an API token to authenticate integrations that access your account. The Console run in this step uses your existing signed-in session.

Keep Your Token Private

An API token can act on your Apify account. This step only confirms that a token exists.

Its value stays hidden throughout this project.

Switch back to the Apify Console from earlier.
Select Settings in the left-side panel.
Select API & Integrations.
Confirm a personal API token appears on the page.
Return to Apify Store.
Enter apify/web-scraper in the search bar.
Select Web Scraper published by Apify.
Use the pricing information on the Actor detail page to confirm that a one-page test fits your available balance.

That credential check is safely complete. Your signed-in Apify session is ready to run the Actor.

Can't Find Web Scraper?

Search for the exact Actor ID apify/web-scraper.
Confirm the result identifies Apify as the publisher.
Ask for help with the Store search:

Configure one raw run

The Start URLs field defines the first pages an Actor visits. An empty Link selector prevents Web Scraper from adding links from those pages to its queue.

The prefilled Page function determines the default fields returned by the Actor. It stays unchanged for this baseline.

Select the Input tab on the Web Scraper page.
Remove every prefilled entry from Start URLs.
Add https://news.ycombinator.com/ to Start URLs.

You should now see one entry under Start URLs. It should contain the Hacker News front-page address.

Clear the Link selector field.
Keep the prefilled Page function unchanged.

Your input now targets one page without following its links. The remaining input options stay at their prefilled values.

Why Keep the Raw Settings?

The default Page function gives you an unrefined result before you define your own fields. This baseline proves that the Actor can fetch the source and write records to a dataset.

Starting the Actor consumes account usage. This one-page input keeps the run tightly scoped.

The first launch can sit at RUNNING for a few seconds while the Actor starts. A short pause here is normal.

Before you click Start, do you expect the dataset to contain polished columns or the scraper's default fields?

Start the raw scrape by clicking Start.
Wait until the run status changes to SUCCEEDED.

The completed status confirms that Web Scraper finished processing its queue. The queue contained only the Hacker News front page.

Select the Output tab.
Open the linked Dataset from the run output.
Inspect at least one item in the dataset table.

You should see at least one raw item from Hacker News. Its columns come from the default Page function.

Some values may be broad or sparse because you have not defined custom fields. Your first live scrape now has a raw Hacker News dataset behind it.

What Should Be Open Now?

The Web Scraper input contains only https://news.ycombinator.com/ under Start URLs.
The Link selector field is empty.
The completed run shows a SUCCEEDED status.
The raw dataset is open in Apify Console for inspection.

Run Failed or Dataset Empty?

Return to the Input tab for the failed run.
Confirm Start URLs contains only https://news.ycombinator.com/.
Confirm the Link selector field is empty.
Confirm the Page function still contains the Actor's prefilled extraction code.
Help me diagnose my Apify Web Scraper run.

Your source is reachable through Apify. Next, you'll describe the fields you want and turn that description into scraper input.

Describe Your Fields

The raw dataset from your first Apify run proves that the Hacker News page is reachable. It also contains more page detail than your final spreadsheet needs.

Now you'll give ChatGPT a precise description of one story record. Its response becomes the extraction configuration you use in the next run.

In this step, get ready to:

Identify one repeating Hacker News story row.
Describe the five fields you want to extract.
Review ChatGPT's extraction configuration for every requirement.

Map one story row

A repeating row gives the scraper one record per story. Each requested field must belong to that same story.

Switch back to the Hacker News front page from earlier.
Choose one visible story as your example row.
Locate the story headline.
Identify the destination behind the headline link.
Locate the story score shown as points.
Locate the submitting username.
Locate the displayed age.

Why define a repeating row?

The row boundary groups one story's visible details into one dataset record. That keeps every extracted value tied to the same story.

Ask ChatGPT for the configuration

Your Apify API token stays private during this handoff. ChatGPT only needs the scraper's non-secret input structure.

Fields to Request

The title field contains the story headline.
The url field contains the story destination link.
The points field contains the story score.
The author field contains the submitting username.
The age field contains the displayed time since submission.

Switch back to the Apify scraper run from earlier.
Copy its current basic/raw input configuration.
Switch back to ChatGPT from earlier.
Paste the copied configuration into the message box.
Ask ChatGPT to preserve the existing start URL of https://news.ycombinator.com/.
Ask ChatGPT to define one repeating story row.
Ask ChatGPT to return a complete extraction configuration in the same input format with the five fields listed above.
Submit the message.

You'll see a structured configuration based on the input format you supplied. The response should preserve the single-page scope.

Missing the Configuration?

Ask for the full configuration only if the response contains an explanation without usable configuration.
Tell ChatGPT which field is missing if its response contains fewer than five fields.
Repeat that the configuration needs one repeating story row if it treats the full page as a single record.
Help me improve the ChatGPT response for my Hacker News extraction configuration.

Check every requested field

AI-generated configuration can look complete while silently dropping a field. Before you inspect the response, do you expect every field name to match your request exactly?

Review the configuration for one repeating story row.
Confirm that it preserves https://news.ycombinator.com/ as the single start URL.
Check that title represents the story headline.
Check that url represents the story destination link.
Check that points represents the story score.
Check that author represents the submitting username.
Check that age represents the displayed time since submission.
Keep the complete response in ChatGPT for the next step.

You should now see one extraction configuration containing all five requested fields. Each field should belong to the same repeating Hacker News story row.

✔️ My configuration is complete

Your configuration is ready for the next Apify run. Keep the ChatGPT conversation available so you can copy the complete response.

ⓧ I'd like to double check the configuration

Compare the response with this complete checklist:

The start URL is https://news.ycombinator.com/.
The configuration defines one repeating Hacker News story row.
The title field extracts the story headline.
The url field extracts the story destination link.
The points field extracts the story score.
The author field extracts the submitting username.
The age field extracts the displayed time since submission.

Your field description is now a scraper-ready configuration. Next up, Apify turns it into rows you can inspect.

Extract Structured Stories

Your request in ChatGPT now describes how Apify should turn the Hacker News front page into repeating story rows. Your raw dataset shows what the scraper collected before that structure was applied.

A configuration proves useful when the scraper produces consistent records from it. In this step, you will run the configured scraper against https://news.ycombinator.com/ and check every requested field.

In this step, get ready to:

Load the extraction configuration into the scraper input.
Run the scraper against the Hacker News front page.
Check the structured rows for all five requested fields.

Load the extraction configuration

The extraction configuration tells the scraper where one story begins. It also defines the values that belong in each resulting row.

Switch back to the ChatGPT conversation from earlier.
Copy only the Apify extraction configuration that ChatGPT produced.

The copied configuration should describe one repeating story row. It should also define the fields title, url, points, author, and age.

Switch back to the scraper's Input tab in the Apify Console.
Replace the basic or raw input with the extraction configuration you copied.

You should now see the structured configuration in the input editor. The start URL should still point to https://news.ycombinator.com/.

What Changes With This Input?

The earlier input asked the scraper to fetch one page with basic settings. This input tells it to identify every repeating story row and collect the same five values from each one.

Run the configured scraper

Each scraper run creates output from the input active at that moment. This run should produce multiple story records from the single source page.

Before you start the run, do you expect one page-level result or multiple story rows?

Click Start to run the scraper with the extraction configuration.
Stay on the run detail page until the run finishes.

You should see the run complete with structured results available in its output. Each result represents one Hacker News story.

Did the configured run fail?

Return to the Input tab for this run.
Compare the pasted configuration with the original response in ChatGPT.
Confirm that the start URL still matches https://news.ycombinator.com/.
Ask for help with the failed Apify run.

Check every story field

A structured dataset displays each object as a row. Its attributes become columns that you can inspect across several stories.

Before you inspect the results, can you name the five column keys you expect to find?

Select the Output tab for the completed run.
Select the Table view for the dataset.

You should see multiple rows. Each row should contain a story headline, a destination link, a score, a submitting username, and a displayed age.

Inspect the column headers for title, url, points, author, and age.
Scan the first few rows for populated values under every column.

The first few rows should line up consistently across all five columns. This proves the scraper repeated the same extraction pattern for separate stories.

Select the JSON view for the same dataset.
Inspect the first item for the five lowercase field names.

You should see one item with the keys title, url, points, author, and age.

Return to the Table view to leave the structured dataset open.

That is the key conversion done: your plain-language request now produces structured Hacker News story rows.

Are any fields empty or missing?

Compare the missing field with the matching field definition in the ChatGPT configuration.
Check that the configuration still defines one repeating story row.
Run the scraper again after correcting the configuration.
Ask for help with an empty dataset field.

Your structured Hacker News dataset is ready. Next up, you will export it to CSV and fix any field that came back empty or incorrect.

Export and Check the CSV

Your configured Apify run now returns one structured row for each Hacker News story.

A CSV makes those rows portable enough to inspect in a spreadsheet. This step checks that every column survived the export with the intended values.

In this step, get ready to:

Export the page-1 structured dataset as a CSV.
Compare the spreadsheet with the source page.
Correct any mismatched field before keeping the final file.

Export the structured dataset

A CSV stores each dataset item as a spreadsheet row. Your five extracted fields become columns.

Return to the page-1 structured dataset already open in the Apify Console.
Select Export for the dataset.
Choose CSV as the format.
Use Select fields to place title, url, points, author, and age in that order.
Select Preview.

You should see five columns in the requested order. Each preview row should represent one Hacker News story.

Select Download.

You should see a CSV file appear in your browser's downloads list.

Locate the downloaded file in your browser's downloads list.
Rename the file to hacker-news-page-1.csv.

The filename now identifies the source page represented by this export.

Open hacker-news-page-1.csv with your spreadsheet application.

That is the export shape locked in. You should see the five field names across the first row with story data underneath.

Missing rows or columns?

Confirm you exported the structured dataset from the configured run.
Check that Select fields contains title, url, points, author, and age.
Make sure the spreadsheet opened hacker-news-page-1.csv instead of an earlier download.
Help me find out why my Apify CSV is missing rows or columns.

Audit the spreadsheet rows

A tidy spreadsheet can still contain silent extraction mistakes. Comparing it with the source page reveals empty fields or values taken from the wrong story row.

Before you compare the rows, which of the five columns do you expect to reveal a mismatch first?

Arrange the Hacker News front page beside your spreadsheet.
Compare the first five spreadsheet rows with their matching source stories.
Confirm each title cell matches the corresponding story headline.
Confirm each url cell points to the destination linked by the corresponding story headline.
Confirm each points cell matches the corresponding displayed score.
Confirm each author cell matches the corresponding submitting username.
Confirm each age cell matches the corresponding displayed submission age.
Scan the remaining spreadsheet rows for empty or visibly shifted cells.

A trustworthy export shows five populated cells for every checked story. Each cell belongs to the same source row.

You have completed the check that catches silent extraction errors. You now have either a clean pass or a precise field to repair.

Correct any mismatched field

Your audit determines the final path. Choose the tab that matches what you found.

✔️ All five columns are correct

Your extraction configuration already matches the source page. The open spreadsheet is your validated page-1 export.

Confirm the final file is named hacker-news-page-1.csv.
Leave the validated extraction configuration unchanged.

ⓧ A column is empty or wrong

A blank or incorrect value usually points to one field in the extraction configuration. Keep the repeating story row intact while you repair that field.

Return to the ChatGPT conversation from earlier.
Send ChatGPT a correction request that includes the current extraction configuration plus one source example for the failed field.
Copy the corrected extraction configuration from the response.
Return to the scraper's Input tab from earlier.
Replace the current extraction configuration with the corrected version.
Select Start.

The completed rerun should contain the repaired field alongside the four preserved fields.

Return to the completed run's Output tab.
Compare the corrected field with the matching values on the source page.

You should see the corrected value populated for each checked story row.

Select Export for the corrected dataset.
Choose CSV as the format.

The export panel should now show CSV as the selected format.

Use Select fields to place title, url, points, author, and age in that order.
Select Preview.

You should see the repaired field in the preview with the other four columns preserved.

Select Download.

You should see the corrected CSV appear in your browser's downloads list.

Rename the corrected file to hacker-news-page-1.csv.
Open the corrected file with your spreadsheet application.

The replacement spreadsheet should contain the repaired field across the page-1 story rows.

Before your final pass, do you expect any cell in the five extracted columns to be blank?

Return to the final hacker-news-page-1.csv already open in your spreadsheet.
Scan every story row across the five extracted columns.
Compare the first row with its matching story on the Hacker News front page.
Compare the last row with its matching story on the Hacker News front page.

You should see the title, url, points, author, and age columns in that order. Every page-1 story row should contain the intended values.

Still seeing mismatched rows?

Check that your spreadsheet shows the corrected download instead of an older file with the same name.
Confirm the repeating story row still groups each headline with its own metadata.
Rerun the configured scraper if the live source page changed after your previous run.
Help me debug mismatched rows in my Hacker News CSV.

That is the page-1 export finished: your spreadsheet contains five validated columns from the structured dataset. Next up, you will test the same extraction configuration against a second page.

Test a Second Page

Your validated Apify extraction has already produced hacker-news-page-1.csv with five populated columns. That proves the configuration works on the Hacker News front page.

A single page can hide rules that only fit one layout. Testing page 2 with the same extraction configuration shows whether the pattern generalises.

In this step, get ready to:

Point the validated extraction configuration at the second Hacker News page.
Run the scraper without changing the story or field rules.
Confirm the page 2 values line up under the same five columns.

Run the configuration on page 2

The start URL chooses which page the scraper reads. The extraction configuration chooses how each repeated story becomes a row.

Return to the scraper input in the Apify Console from earlier.
Find the start URL value that currently contains https://news.ycombinator.com/.
Replace the start URL with https://news.ycombinator.com/news?p=2.
Keep the repeating story rule unchanged.
Keep the title, url, points, author, and age extraction rules unchanged.

Before you start the run, do you think the same five columns will stay populated on page 2?

Start a new run using the same control you used for the previous runs.
Wait for the run to complete.
Open the dataset created by the new run.

You should see structured rows from the second Hacker News page. Each row should contain the same five fields as your page 1 dataset.

You have the first proof now because page 2 is producing structured rows with the same extraction rules.

No rows in the new dataset?

Confirm the start URL is exactly https://news.ycombinator.com/news?p=2.
Check that the repeating story rule still matches the validated page 1 configuration.
Restore any extraction field that changed after the page 1 run.
Run the scraper again after correcting the input.
Help me diagnose why my page 2 dataset is empty.

Check that every field still lines up

What does field alignment mean?

The title field contains the story headline.
The url field contains the story destination link.
The points field contains the story score.
The author field contains the submitting username.
The age field contains the displayed time since submission.

Before you compare a row, do you expect the page 2 layout to preserve every field?

Switch to the Hacker News page at https://news.ycombinator.com/news?p=2.
Choose the first visible story row.
Return to the page 2 dataset in the Apify Console.
Find the dataset row whose title matches the chosen story.
Compare the url value with the story destination link.
Compare the points value with the displayed score.
Compare the author value with the displayed username.
Compare the age value with the displayed submission age.

You should find that every value describes the same selected story. This proves the fields stayed aligned when the source page changed.

Confirm the pattern across the dataset

One matching row is useful evidence. A wider scan catches extraction rules that only happened to work near the top of the page.

Scan several additional rows across the page 2 dataset.
Confirm each title value is a story headline.
Confirm each url value belongs to its story.
Confirm each points value stays in the score column.
Confirm each author value stays in the username column.
Confirm each age value stays in the submission age column.

You should see the same five columns preserve their meaning throughout the second-page dataset. This confirms the validated configuration works across both Hacker News pages.

Seeing an empty or shifted field?

Compare the affected dataset row with the same story on page 2.
Treat a blank as an extraction problem when the source page displays a value for that row.
Restore the validated page 1 extraction rule for the affected field.
Run page 2 again after restoring the rule.
Help me fix a field that is empty or shifted in my page 2 dataset.

You have proved this configuration works beyond one page. That makes it a reusable extraction setup instead of a one-page result.

Secret mission

Audit the Dataset for Silent Failures

Two successful runs show that the configuration works across two pages. Audit a 20-row sample from both Hacker News datasets to separate extraction failures from normal live-page drift.

Clean Up Your Resources

All three scraper runs are complete, so no process is actively fetching pages. Decide how much project data to preserve before a future Apify run uses more of your credit balance.

Resources you used:

Three completed Apify scraper run records covering the basic page 1 scrape, the structured page 1 scrape, and the structured page 2 scrape.
Three Apify datasets containing the raw page 1 results, the structured page 1 results, and the structured page 2 results.
The validated extraction configuration retained in the two structured run inputs.
The exported CSV file named hacker-news-page-1.csv.

Keep everything running

No action needed. Choose this if you want the validated configuration and project outputs ready for another scrape.

Leave the three completed runs in the Apify Console.
Leave the three datasets in the Apify Console.
Keep hacker-news-page-1.csv in its current saved location.
Reuse the validated extraction configuration from either structured run when you return.
Check your available Apify credits before starting another run.

Pause - I'll come back to this later

No scraper process needs to be stopped because all three runs have finished. Choose this option to close your workspace while preserving every result.

Leave the completed run records unchanged in the Apify Console.
Leave the raw dataset and both structured datasets unchanged.
Close the spreadsheet containing hacker-news-page-1.csv.
Close the Apify Console browser tabs.
Resume later from either structured run to recover the validated extraction configuration.

Delete - I don't want to use this again

Deleting these records is permanent. Your accounts stay intact.

Remove the Apify Datasets

Remove the datasets before the run records. This keeps each result easy to identify during cleanup.

Return to the raw Hacker News dataset from earlier.
Use the deletion option on its dataset detail page.
Return to the structured page 1 dataset from earlier.
Use the deletion option on its dataset detail page.
Return to the structured page 2 dataset from earlier.
Use the deletion option on its dataset detail page.
Return to the Apify datasets list.

You should no longer see any of the three Hacker News datasets.

Remove the Completed Runs

The two structured run records retain the validated extraction configuration. Removing the three run records clears the remaining project history from Apify.

Return to the completed runs list in the Apify Console from earlier.
Select the basic page 1 run that used the scraper's raw settings.
Use the deletion option on its run detail page.
Select the structured page 1 run.
Use the deletion option on its run detail page.
Select the structured page 2 run.
Use the deletion option on its run detail page.
Refresh the completed runs list.

You should no longer see any of the three project run records.

Remove the CSV

The CSV is stored on your computer. Use the instructions for your operating system to find the file without assuming where your browser saved it.

macOS

Press Cmd+Space to open the search bar.
Type hacker-news-page-1.csv into the search field.
Select the matching file in the Finder results.
Move the selected file to Trash.
Empty Trash to remove the file permanently.
Search for hacker-news-page-1.csv again.

You should see no matching project CSV in the results.

Windows

Press the Windows key to open the search bar.
Type hacker-news-page-1.csv into the search field.
Show the matching file in File Explorer.
Move the selected file to the Recycle Bin.
Empty the Recycle Bin to remove the file permanently.
Search for hacker-news-page-1.csv again.

You should see no matching project CSV in the results.

Nice Work!

You made it! Your plain-language field description now drives an Apify scraper that produces validated Hacker News story rows.

You exported those rows to a CSV that opens in a spreadsheet. You also proved that the extraction configuration works across a second page.

You've learned how to:

Translate a plain-language field request into a reusable extraction configuration with ChatGPT.
Perform dataset checks for empty or incorrect values across the title, url, points, author, and age fields.
Create a CSV export as hacker-news-page-1.csv. Test the same extraction configuration on the second Hacker News page.
Extend your scraping workflow through an optional Secret Mission that pushes your skills further.

Ready to quiz yourself?