Migrate Google Custom Search to Apify

Port a Python search script to Apify and fetch the top result as Markdown.

Introduction

30 Second Summary

Software can work reliably for years, then stop overnight when one of its online services closes. A Google service used by search tools reaches that deadline on January 1, 2027.

In this project, you will migrate a Python search script from Google's Custom Search JSON API to Apify Web Search before the shutdown. The migrated script keeps its familiar printed results while adding a Markdown preview of the top page through Apify Web Fetch.

What You'll Build

Run your finished script to see each live result printed with its title, link, and domain before the first paragraph from the top page appears as Markdown.

By the end of this project, you'll have:

  • Live search results returned to your terminal as JSON. The same query appears as a dataset table in the Apify Console.
  • A migrated Python script that prints each result's title, link, and domain. Your existing output format stays intact.
  • A Markdown page preview that prints the top result's first paragraph. A usage check that shows how much of your free monthly usage the tests consumed.
  • Secret Mission: An optional challenge to push your skills further.

Are there any prerequisites?

You need Python or curl available on your computer. You create a free Apify account during the project.

Before We Start

The search migration needs a local place to run. You also need a browser to work with Apify.

This step makes your terminal ready. It also confirms that Python 3 is installed.

In this step, get ready to:
  • Keep a web browser available for the account setup later.
  • Open a terminal for running local commands.
  • Confirm that Python 3 is installed.
Open your working tools

Your browser handles the Apify pages. Your terminal runs the local checks in this project.

  • Keep this project page open in your web browser.

The terminal app has a different name on each operating system. Use the tab for your computer.

macOS

  • Press Cmd+Space to open Spotlight.
  • Enter Terminal in the search field.
  • Press Return to open Terminal.

Windows

  • Press the Windows key to open Windows Search.
  • Enter Windows PowerShell in the search field.
  • Select Windows PowerShell from the search results.

Good start. You now have the two workspaces needed for the rest of the project.

Check your Python installation

Python runs the migrated search script on your computer. The version output confirms whether the required Python 3 runtime is available.

Before you run the check, do you expect the terminal to find Python on your computer?

macOS

  • Check your Python version by running this command:
python3 --version

What does this command do?

The python3 command selects Python 3. The --version option prints its version number before exiting.

Windows

  • Check your Python version by running this command:
python --version

What does this command do?

The python command selects your default Python runtime. The --version option prints its version number before exiting.

✔️ I see Python 3

You should see output that starts with Python 3.

That check passed. Your local Python runtime is ready to run the migrated search script.

ⓧ Python 3 is unavailable

Install Python from its official download page. Choose the tab for your computer.

macOS

  • Open the official Python downloads page for macOS in your browser.
  • Select the newest stable Python release.
  • Click Download macOS installer.
  • Open the downloaded .pkg file.
  • Click Continue on the installer screens.
  • Click Agree when the license prompt appears.
  • Click Install to start the installation.
  • Approve the macOS permission prompt with your computer password.
  • Return to Terminal after the installer finishes.
  • Confirm the installation by running this command:
python3 --version
What does this command confirm?

The command asks the newly installed Python 3 runtime for its version. Output beginning with Python 3 confirms the installation succeeded.

You should now see output that starts with Python 3.

Windows

  • Open the official Python releases page for Windows in your browser.
  • Select the newest stable Python release.
  • Click Download Windows installer (64-bit).
  • Open the downloaded installer.
  • Complete the installer with its default selections.
  • Approve the Windows permission prompt when it appears.
  • Return to Windows PowerShell after the installer finishes.
  • Confirm the installation by running this command:
python --version
What does this command confirm?

The command asks the installed Python runtime for its version. Output beginning with Python 3 confirms the installation succeeded.

You should now see output that starts with Python 3.

Still Unable to Run Python?

  • Close your terminal after installation. Reopen it so the app can load the updated command path.
  • Use python3 --version on macOS. Use python --version on Windows.
  • Check that the installer reached its success screen before closing it.
  • Help me diagnose why my terminal cannot find Python after installation.

Your browser is available. Your terminal can now run Python 3. Next, you'll create your Apify account and get the API token that authorizes your searches.

Create Your Apify Account

Your migrated search script needs permission to call Apify services. A free Apify account provides that identity.

In this step, you'll enter Apify Console. You'll copy the API token that authenticates later Web Search requests and Web Fetch requests.

In this step, get ready to:
  • Create a free Apify account.
  • Confirm you can access the Apify Console Dashboard.
  • Copy your personal API token from the API & Integrations page.
Create your free Apify account

Apify Console is the web interface for managing your account. Your free account gives you access to the search tools used throughout this project.

Email and password

  • Enter your email address in the email field.
  • Create a password in the password field.
  • Select Sign up.
  • Open the verification email from Apify.
  • Follow the verification link to enter Apify Console.

Google

  • Click Continue with Google.
  • Sign in to your Google account.
  • Authorize Apify when Google asks for permission.

You return to Apify Console after authorization.

GitHub

  • Click Continue with GitHub.
  • Sign in to your GitHub account.
  • Authorize Apify when GitHub asks for permission.
  • Complete email verification if Apify requests it.

You return to Apify Console after authorization.

  • Select Dashboard in the left-side panel.
  • Check the Dashboard for suggestions linked to your new account.

You're in. The Dashboard confirms your free account is ready for the search migration.

Can't reach the Dashboard?

  • Return to the Apify sign-in page if your verification email does not arrive.
  • Select Resend verification email on the verification page.
  • Complete the reCAPTCHA if Apify displays one.

Help me get into my Apify account. You can also ask the Apify & Crawlee community for account setup help.

Copy your personal API token

A personal API token proves that requests came from your account. The terminal commands and Python script use it to access Apify services.

Keep your token private

Copying a credential can feel risky. Keep this token on devices you trust.

Never paste it into a public post. Never commit it to version control.

  • Select Settings in the Apify Console left-side panel.
  • Select API & Integrations.
  • Locate the Personal API tokens section.

Before you copy the token, what confirmation do you expect Apify Console to show?

  • Click the Copy icon next to your personal API token.

You should see a brief confirmation from Apify Console. That confirms the token is now on your clipboard.

  • Keep the token on your clipboard for the next step.

Can't copy your token?

  • Confirm that you are viewing the Personal API tokens section.
  • Refresh the API & Integrations page if the token controls do not load.
  • Sign back in if Apify Console reports that your session has expired.
  • Help me copy my Apify API token safely.

That's the access hurdle cleared. Your personal token is ready to authenticate your first Web Search request in the next step.

Query Web Search with Curl

Your API token is ready for its first authenticated call to Apify Web Search. The next check proves the new endpoint accepts the request fields your migration depends on.

The curl command sends an HTTP request directly from your terminal. You can inspect the raw JSON response before any Python code transforms it.

In this step, get ready to:
  • Send an authenticated curl request to the Web Search endpoint.
  • Shape the search with request parameters.
  • Confirm the response contains the expected search result fields.
Send the authenticated request

The endpoint decides where the request goes. The Authorization header proves who is calling.

The request body carries the search controls as JSON. Each control changes a specific part of the live search.

The JSON response is safe to capture once the command containing your token is outside the frame.

  • Keep the command line outside every screenshot because it contains your API token.

Before you run this, do you think the endpoint accepts the request or rejects its authentication?

A live search can take a couple of seconds. A short pause means the search engine is collecting current results.

  • Run the command below with your-apify-api-token-here replaced by the token you copied earlier:
curl -X POST 'https://web-search.apify.actor/' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer your-apify-api-token-here' \
  -d '{
    "query": "Apify web scraping",
    "countryCode": "US",
    "maxResults": 3
  }'

What does this request do?

  • The https://web-search.apify.actor/ endpoint runs Web Search immediately.
  • The Authorization header carries your API token.
  • The query field holds the text you want to search for.
  • The countryCode field runs the search as if it came from the selected country.
  • The maxResults field caps the number of returned results.

That is your first authenticated search working. The terminal now holds live JSON from Web Search.

Still waiting or seeing a failed request?

  • Confirm that you replaced your-apify-api-token-here with the complete token.
  • Check that the command's quotation marks match the code block.
  • Retry the request after confirming your internet connection is active.
  • Help me diagnose my Web Search curl request.
Inspect the JSON response

The response groups individual result objects inside results. Each object describes one page returned by the search.

Before you count the objects, do you expect maxResults to act as a ceiling or a guaranteed count?

  • Scroll through the terminal output to find the top-level results array.
  • Count the objects inside the results array.
  • Locate the first result object.
  • Confirm the first object contains a url field.
  • Confirm the first object contains a title field.
  • Confirm the first object contains a domain field.

You'll see no more than three objects because maxResults is a ceiling. The first object includes url.

The same object includes title. It also includes domain.

Your terminal test proves the migration endpoint accepts the new request shape. Next, you'll run the same query in Apify Console to see its dataset table.

Run Web Search in the Console

Your terminal request proved that Apify Web Search accepts your search input. It also showed the structured JSON response.

Now you will run the same search through the Apify Console. A regular Actor run saves its output in a dataset that you can inspect as a table.

In this step, get ready to:
  • Match the Web Search Actor input to your earlier API request.
  • Complete a Console run with the same search input.
  • Inspect the saved search output in a dataset table.
Match the previous search input

The Input tab exposes the same fields that you used in the HTTP request. Matching those values gives you a fair comparison between the API and Console paths.

  • Open the official Web Search Actor page in the browser from earlier.
  • Select the Input tab at the top of the Actor page.
  • Switch back to the terminal from earlier.
  • Find the curl request that you ran in the previous step.
  • Return to the Web Search Actor in your browser.
  • Match the query field to the search text in your earlier request.
  • Match the countryCode field to the country code in your earlier request.

The query field now matches your earlier search text. The countryCode field now matches your earlier country filter.

  • Match the maxResults field to the result limit in your earlier request.

The Actor input now represents the same search that produced JSON in your terminal.

Why match the same input?

Using identical values isolates the way you run the search. Any difference in presentation comes from the Console storing the response as a dataset.

This comparison confirms that the API endpoint and regular Actor run use the same search fields.

Run the Actor and inspect the dataset

A regular Actor run performs one search. It saves the response as one dataset item.

The live search usually takes a couple of seconds. A brief running state is expected while the Actor waits for results.

  • Predict whether the Console will preserve the same query and response structure as your terminal request.
  • Select Start to begin the Actor run.

You will see the run status update while Web Search processes the query.

  • Wait for the Actor run to finish.

Your browser-based search is complete. The response is now stored in the run's dataset.

  • Select the Output tab on the run page.
  • Select Table to view the dataset item as a row.
  • Compare the query cell with the search text from your terminal request.
  • Inspect the results cell for the result data.

You should see one dataset row for the search run. Its result data includes the url, title, and domain fields that appeared in the terminal response.

What does the table prove?

The table confirms that a regular Actor run stores structured search output in a dataset. Each top-level response field becomes part of the saved item.

You can now inspect a search visually while your scripts continue to consume the same JSON structure.

No row in the table?

  • Return to the Input tab.
  • Confirm that query contains your earlier search text.
  • Confirm that the Actor run has finished before selecting Output.
  • Refresh the browser if the finished run still shows an empty output area.
  • Help me diagnose why my Web Search Actor run has no dataset row.

You have now proved that the same search works through both the API and the Console. Next, you will move this working search into the migration script.

Migrate the Search Script

Your terminal test proved that Apify Web Search returns live JSON results. The Apify Console dataset confirmed the same result fields in a table.

The starter script still uses Google Custom Search's request names. In this step, you will swap that API contract while keeping the printed title, link, and domain labels stable.

In this step, get ready to:
  • Translate the old request parameters into Apify Web Search parameters.
  • Remap the response fields used by the result loop.
  • Run the migrated script to verify its printed output.
Map the old search contract

The migration changes the names used on each side of the request. Your script can preserve its visible output because the new response still contains equivalent values.

  • Map q to query.
  • Map gl to countryCode.
  • Map num to maxResults.
  • Replace key with an Authorization header containing your Apify API token.
  • Read the results array instead of items.
  • Read url instead of link.
  • Read domain instead of displayLink.

Why can the output stay the same?

The provider-facing field names change inside the script. The labels printed for anyone using the script remain Title, Link, and Domain.

Create the migrated script

The migrated file keeps the starter's search values in legacy_params. It translates those values immediately before sending the request.

  • Create search.py in your Python editor by pasting this code:
import json
from getpass import getpass
from urllib.parse import urlencode
from urllib.request import Request, urlopen

# Keep the starter script's original search values
legacy_params = {"q": "best coffee shops", "gl": "fr", "num": 10}

# Map the Google request names to the Apify request names
params = {
    "query": legacy_params["q"],
    "countryCode": legacy_params["gl"].upper(),
    "maxResults": legacy_params["num"],
}

# Replace the key parameter with an Authorization header
api_token = getpass("Paste your Apify API token: ")
request = Request(f"https://web-search.apify.actor/?{urlencode(params)}", headers={"Authorization": f"Bearer {api_token}"})

# Send the search request and read its JSON response
with urlopen(request, timeout=30) as response:
    data = json.load(response)

# Read results instead of items while preserving the old output labels
for result in data["results"]:
    print(f"Title: {result['title']}")
    print(f"Link: {result['url']}")
    print(f"Domain: {result['domain']}\n")

What does this code do?

  • The legacy_params dictionary represents the values previously sent as q, gl, and num.
  • The params dictionary translates those values into the Apify Web Search request shape.
  • The getpass() prompt keeps your token hidden while you paste it into the terminal.
  • The result loop reads results, url, and domain while preserving the original printed labels.
  • Save search.py.
Run the migrated search

The script asks for the token you copied earlier. Your terminal does not display the token while you paste it.

The live search can take a couple of seconds to return results. A short pause means the search is running.

Before you run the script, which three labels do you expect to repeat for every result?

  • Run the migrated search from the folder containing search.py by using this command:
python search.py
  • Paste your copied Apify API token when the script prompts for it.
  • Press Enter to send the search request.

You should see repeating Title, Link, and Domain lines. The link values come from url while the domain values come from domain.

That migration is working. Your Python script now calls Apify Web Search without changing the output format its users rely on.

Script not printing results?

  • Confirm that you saved the file as search.py in the folder used by your terminal.
  • Rerun the script if the copied token contained an extra space.
  • Check that your internet connection can reach the Web Search endpoint.
  • Ask for help with the request if the script still stops before printing results.

✔️ Awesome, I've got everything!

Great. Double-check that you saved search.py after the successful run.

ⓧ I'd like to double check the full code

Compare your complete search.py file with this version.

import json
from getpass import getpass
from urllib.parse import urlencode
from urllib.request import Request, urlopen

# Keep the starter script's original search values
legacy_params = {"q": "best coffee shops", "gl": "fr", "num": 10}

# Map the Google request names to the Apify request names
params = {
    "query": legacy_params["q"],
    "countryCode": legacy_params["gl"].upper(),
    "maxResults": legacy_params["num"],
}

# Replace the key parameter with an Authorization header
api_token = getpass("Paste your Apify API token: ")
request = Request(f"https://web-search.apify.actor/?{urlencode(params)}", headers={"Authorization": f"Bearer {api_token}"})

# Send the search request and read its JSON response
with urlopen(request, timeout=30) as response:
    data = json.load(response)

# Read results instead of items while preserving the old output labels
for result in data["results"]:
    print(f"Title: {result['title']}")
    print(f"Link: {result['url']}")
    print(f"Domain: {result['domain']}\n")

Your search script has moved off Google Custom Search without losing its familiar output. Next, you will use the top result as the starting point for reading a full page as Markdown.

Fetch the Top Result as Markdown

Your migrated search script already returns the same title, link, and domain fields as before. Each link still points to a page that your script has not read.

In this step, you will pass the first result to Apify Web Fetch. It returns the page as Markdown, which gives your script clean text it can process.

In this step, get ready to:
  • Send the first search result URL to Web Fetch.
  • Extract the first paragraph from the returned Markdown.
  • Print the paragraph after the familiar search results.
Add page reading to the script

The search response already contains the URL that Web Fetch needs. Your script can take the first URL from results, request its Markdown, and select the first block that looks like a paragraph.

  • Return to search.py in the editor from the previous step.
  • Place your cursor after the loop that prints each result.
  • Add the Web Fetch request and paragraph extraction by pasting this code:
# Fetch the first search result as Markdown
first_result_url = resp.json()["results"][0]["url"]
fetch_resp = requests.get(
    "https://web-fetch.apify.actor/",
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    params={"url": first_result_url, "formats": "markdown"},
    timeout=30,
)
markdown = fetch_resp.json()["markdown"]

# Select the first content block that looks like a paragraph
markdown_blocks = markdown.split("\n\n")
first_paragraph = next(
    block.strip()
    for block in markdown_blocks
    if block.strip() and not block.lstrip().startswith(("#", "!", "-", "*"))
)

# Print the page excerpt after the search results
print("\nFirst paragraph:")
print(first_paragraph)

What does this code do?

  • The first_result_url variable reads the url field from the first item in results.
  • The second request sends that URL to https://web-fetch.apify.actor/ with the same authenticated header used by the search request.
  • The formats parameter asks Web Fetch to include markdown in its JSON response.
  • The paragraph filter skips empty blocks. It also skips common Markdown headings, images, and list items.
  • The final two lines label the excerpt and print it after the search results.
Run the search and inspect the excerpt
  • Save search.py.

Before you run it, ask yourself: what should appear after the familiar title, link, and domain output.

The live search and page fetch can take a few seconds. A short pause means both requests are collecting fresh content.

  • Rerun search.py with the same terminal command from the previous step.

You should see the search results first. Below them, you should see First paragraph: followed by text from the top result.

You have turned the top search result into readable page content that your Python script can use.

No paragraph in the output?

  • Confirm that the search output contains at least one result. The fetch uses the first item in the results array.
  • Check that the Web Fetch request uses the same APIFY_TOKEN environment variable as your working search request.
  • Try the script again if the top page briefly fails to load. Search results are live, so the first URL can change between runs.
  • Help me diagnose why Web Fetch is not returning a Markdown paragraph.

✔️ Awesome, I've got everything!

Great. Double check that you saved search.py after seeing the first paragraph in your terminal.

ⓧ I'd like to double check the full code

At the end of search.py, confirm that the Web Fetch section appears after the existing loop that prints the search results.

# Fetch the first search result as Markdown
first_result_url = resp.json()["results"][0]["url"]
fetch_resp = requests.get(
    "https://web-fetch.apify.actor/",
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    params={"url": first_result_url, "formats": "markdown"},
    timeout=30,
)
markdown = fetch_resp.json()["markdown"]

# Select the first content block that looks like a paragraph
markdown_blocks = markdown.split("\n\n")
first_paragraph = next(
    block.strip()
    for block in markdown_blocks
    if block.strip() and not block.lstrip().startswith(("#", "!", "-", "*"))
)

# Print the page excerpt after the search results
print("\nFirst paragraph:")
print(first_paragraph)

Your script now searches the web and reads the top page in one run. Next up, you will check how much of your free monthly usage these requests consumed.

Review Your Apify Usage

Your migrated script now gets results from Apify Web Search. It also reads the top page through Apify Web Fetch.

Those live requests consume part of your included $5 monthly usage. This final step uses the Apify Console to check what your tests spent.

In this step, get ready to:
  • Open the current billing period in the Apify Console.
  • Compare your test activity with the included monthly usage.
Open the current usage view

Apify collects usage for the active billing cycle under Current period. Its plan consumption graph shows how much of your free plan you have used.

This billing check is safe because it only reads your existing usage. It does not change your plan.

  • Switch back to the Apify Console browser tab from earlier.
  • Select Billing from the left navigation panel.
  • Select the Current period tab.
  • Locate the Plan Consumption Graph near the top of the page.
  • Compare the used portion with the included $5 monthly usage.

You should see your total usage for the active billing cycle. You should also see the billing cycle dates.

How to read this view

  • The Current period tab covers your active billing cycle.
  • The Plan Consumption Graph compares your usage with the amount included in your plan.
  • The service breakdown separates usage into categories such as Actors.
Connect the usage to your tests

Your current total includes the requests made while testing the migrated script. The service breakdown helps you trace where that usage came from.

Before you inspect the breakdown, do you expect a few test requests to use a small share or a large share of your monthly usage?

  • Scroll below the plan consumption graph to the service breakdown.
  • Review the Actors section for usage associated with your Web Search tests.
  • Look for usage associated with your Web Fetch tests.
  • Compare the test activity with the total shown in the plan consumption graph.

You should see the test activity counted against your monthly usage. The amount should occupy only a small part of the included $5 monthly usage.

Recent usage not showing?

  • Wait briefly for the account totals to update.
  • Refresh the Current period page.
  • Confirm that the signed-in account is the one whose token powered your tests.
  • Help me understand why my recent Apify usage is missing.

That closes the loop. You can now connect every live test with its effect on your monthly allowance.

Secret mission

Test a Domain-Restricted Search

Your migration preserves the printed output. Now prove it can also restrict results to one approved domain by mapping the old siteSearch behavior to allowedDomains and checking every dataset row.

Clean Up Your Resources

Clean Up Your Resources

Your migrated script uses no Apify monthly usage while idle. Choose whether to keep it ready, pause your work, or remove the local file plus revoke its credential.

Resources you used:

  • The search.py file containing your migrated Python search plus page-reading code.
  • The personal Apify API token that authenticates requests to the Web Search API plus Web Fetch.

Keep everything running

No action is needed. This option fits if you plan to keep using the migrated search script.

  • Keep search.py in its existing folder.
  • Keep the personal API token active only while you plan to use the script.
  • Review the Apify usage page after future test searches.

Future Web Search plus Web Fetch requests consume your monthly usage when they run.

Pause - I'll come back to this later

Leave the script idle to pause your work. No server or background process needs to stay running.

  • Stop any request that is still running by pressing Ctrl+C in the terminal.
  • Close the terminal window.
  • Keep search.py in its existing folder for your next session.
  • Keep any command or file containing the personal API token private.

The idle script consumes no monthly usage. Usage resumes when you send another request.

Delete - I don't want to use this again

Remove the local script plus its project credential. Your Apify account remains available.

Revoking a credential is permanent for that token. You can create a new token through your account later.

  • Return to the Apify Console from earlier.
  • Select Settings.
  • Select API & Integrations.
  • Find the personal API token you copied for this project.
  • Revoke the personal API token.
  • Confirm the token is no longer available for active use.

Your revoked token can no longer authenticate the migrated script. Your Apify account remains ready for a new token later.

Deleting search.py removes only your local copy of the migrated script.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type search.py into system search.
  • Use the project copy in the results to open its containing folder.
  • Select search.py in that folder.
  • Move the selected file to Trash on macOS or Recycle Bin on Windows.
  • Permanently delete search.py from the recovery location if you do not want to keep a recoverable copy.
  • Return to the folder that contained the script.
  • Confirm search.py is no longer listed there.

Cleanup is complete. The local script is gone from its folder.

Nice Work!

Nice Work!

You did it! Your Python search script now runs on Apify. It preserves the familiar search output before printing the top page's first paragraph.

You've learned how to:

  • Call the Apify Web Search API from the terminal. You inspected its JSON results before confirming the same search in an Apify Console dataset table.
  • Migrate a Python script away from Google Custom Search. You preserved the script's original printed result format through request parameter mappings and response field mappings.
  • Use Apify Web Fetch to retrieve the top result. Your script now extracts its first paragraph from Markdown before displaying it alongside the search results.
  • Complete an optional Secret Mission to extend the migrated search workflow.

Ready to quiz yourself?