Find Buttons with Chrome CDP

Build a Python tool that finds and clicks page controls through Chrome CDP.

Introduction

30 Second Summary

A busy web page can hide dozens of clickable choices behind text, icons, and menus. Searching through its pixels adds work when the browser already tracks each choice.

In this project, you will build a Python script that finds every clickable element in a Chrome tab without using a screenshot. The script turns Chrome's accessibility tree into a numbered list before controlling your chosen element through its coordinates.

What You'll Build

A terminal tool that maps the clickable elements on Hacker News before highlighting your selection in the live browser.

By the end of this project, you'll have:

  • A numbered clickable-element list showing each element's role, name, and node ID.
  • A JSON export you can compare with a full-page screenshot to see the size difference.
  • A coordinate-based click tool that highlights your chosen element before opening its link in Chrome.
  • Secret Mission: An optional challenge to push your browser automation skills further.

Are there any prerequisites?

You need Chrome plus Python 3 installed on your computer. No account or API key is required.

Before We Start

Your script can only read a page after Google Chrome exposes a control channel. The Chrome DevTools Protocol provides that channel through remote debugging.

This step prepares Python in an open terminal. You finish with Hacker News open in a debuggable Chrome session.

In this step, get ready to:
  • Verify Python from an open terminal.
  • Start an isolated Chrome session on localhost:9222.
  • Expose Hacker News as a remote debugging target.
Check Python in your terminal

Your later script uses Python to send commands to Chrome. A version check confirms that the interpreter is available from your terminal.

  • Press Cmd+Space on macOS or the Windows key on Windows to open system search.
  • Type Terminal on macOS or Command Prompt on Windows.
  • Press Enter to open the terminal.
  • Check your Python version by running this command:
python3 --version

What does this command do?

  • The python3 command selects the Python 3 interpreter.
  • The --version option prints the installed version.

✔️ I see a Python 3 version

Your terminal can run Python scripts. That local interpreter is ready to become the control layer for Chrome.

ⓧ Command not found

Python needs to be installed before the project can continue. Choose your operating system below.

macOS

  • Download the current macOS installer from the official Python downloads page.
  • Double-click the downloaded .pkg file to start the installer.
  • Follow the installer prompts using the standard options.
  • Return to your terminal after the installation finishes.
  • Rerun the version check shown above.

Windows

  • Download the Python install manager from the official Python downloads page.
  • Double-click the downloaded installer.
  • Select Install when the installer opens.
  • Return to Command Prompt after the installation finishes.
  • Rerun the version check shown above.

Linux

  • Install Python 3 using the package manager documented by your Linux distribution.
  • Return to your terminal after the installation finishes.
  • Rerun the version check shown above.

You should now see a Python version whose major number is 3. Your terminal is ready for the script you build next.

Launch Chrome with remote debugging

Remote debugging lets a local program inspect pages through Chrome. Port 9222 gives your future script a local address for that connection.

Use the isolated profile

Giving software browser control can feel risky. The dedicated profile keeps this project separate from the cookies in your everyday Chrome profile.

Only trusted local programs should connect to this session. You turn remote debugging off when the project is complete.

macOS

  • Launch Chrome with an isolated debugging profile by running this command:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-profile-stable

What does this command do?

  • The --remote-debugging-port=9222 option exposes Chrome's debugging connection on a local port.
  • The --user-data-dir=/tmp/chrome-profile-stable option stores this session in a separate profile.
  • Wait for the new Chrome window to open.
  • Keep the terminal running while you complete the project.

Windows

  • Launch Chrome with an isolated debugging profile by running this command:
start chrome --remote-debugging-port=9222 --user-data-dir=%TEMP%\chrome-profile-stable

What does this command do?

  • The --remote-debugging-port=9222 option exposes Chrome's debugging connection on a local port.
  • The --user-data-dir=%TEMP%\chrome-profile-stable option stores this session in a separate profile.
  • Wait for the new Chrome window to open.
  • Keep Command Prompt open while you complete the project.

Linux

  • Launch Chrome with an isolated debugging profile by running this command:
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-profile-stable

What does this command do?

  • The --remote-debugging-port=9222 option exposes Chrome's debugging connection on a local port.
  • The --user-data-dir=/tmp/chrome-profile-stable option stores this session in a separate profile.
  • Wait for the new Chrome window to open.
  • Keep the terminal running while you complete the project.

You should see a new Chrome window using a fresh profile. Your regular browsing profile remains separate.

Chrome did not open?

Confirm that Google Chrome is installed in its standard location. On Linux, confirm that the Chrome executable is available from your terminal.

Close any Chrome window using the same temporary profile before trying the command again.

Help me launch Chrome with remote debugging.

Enable debugging and open Hacker News

Chrome's inspect page has a per-session control for incoming debugging connections. Enabling it makes the running browser available to the script you build in the next step.

  • Select the address bar in the isolated Chrome window.
  • Enter chrome://inspect/#remote-debugging.
  • Turn on Allow remote debugging for this browser instance.
  • Click Allow if Chrome asks for permission to accept debugging connections.

The remote debugging control should now appear enabled. That sensitive setup is complete for this browser session.

Before you check the connection, do you expect Chrome to list the Hacker News tab as an inspectable target?

  • Open another Chrome tab.
  • Enter http://localhost:9222 in the address bar.

You should see the remote-debugging targets page with Hacker News listed. This proves that Chrome is exposing the tab on the local debugging port.

Targets page not loading?

Keep the terminal that launched Chrome running. Confirm that you entered the local address with port 9222.

If another program already uses that port, close it before relaunching the isolated Chrome session.

Help me find my Chrome debugging target.

✔️ Awesome, I've got everything!

Great. Keep the isolated Chrome window open with Hacker News available as a debugging target.

ⓧ I'd like to double check the full code

Browser tab

Hacker News is open in a Chrome tab.

Chrome remote debugging

Enabled for the running Chrome instance on localhost:9222.

Chrome is ready to accept local automation commands. Next, you connect your Python script to the Hacker News tab.

Read Chrome's Accessibility Tree

Your Chrome session can now accept local debugging connections. The missing piece is a program that turns that connection into page data.

In this step, you will write a Python script that uses the Chrome DevTools Protocol to discover the Hacker News tab.

The script requests Chrome's full accessibility tree. Your terminal prints the total number of nodes that Chrome returns.

In this step, get ready to:
  • Install the WebSocket client used to exchange messages with Chrome.
  • Create a script that discovers the Hacker News target.
  • Request the full accessibility tree and print its node count.
Install the WebSocket client

A WebSocket provides the two-way connection that carries DevTools Protocol messages. The websocket-client package gives your Python script that connection.

  • Install the WebSocket client by running:
pip install websocket-client

What Did This Install?

The package provides create_connection() for opening a short-lived WebSocket connection. Your script uses that connection to send one request to Chrome and receive one response.

The installation is complete when the command finishes without reporting an installation failure.

Installation Not Completing?

Check that your terminal still has access to the internet. Confirm that the Python installation from the previous step is available in this terminal.

If your terminal cannot find the package installer, ask for help diagnosing your Python package setup.

Build the CDP request

Chrome exposes a target descriptor for every inspectable page. Each descriptor includes the WebSocket address that a client uses to control that page.

  • Use a plain-text editor to create find_buttons.py in the folder shown by your terminal prompt.
  • Paste the following code into find_buttons.py:
import json
import urllib.request
from websocket import create_connection

TARGETS_URL = "http://localhost:9222/json"
HACKER_NEWS_TITLE = "Hacker News"

# Ask Chrome for every inspectable target on the debugging port
with urllib.request.urlopen(TARGETS_URL) as response:
    targets = json.load(response)

# Select the page target that belongs to the open Hacker News tab
hacker_news_target = next(
    target for target in targets
    if target["type"] == "page" and target["title"] == HACKER_NEWS_TITLE
)

# Connect directly to the selected page without a browser origin header
socket = create_connection(hacker_news_target["webSocketDebuggerUrl"], suppress_origin=True)

# Request the complete accessibility tree and keep its nodes in memory
socket.send(json.dumps({"id": 1, "method": "Accessibility.getFullAXTree"}))
message = json.loads(socket.recv())
ax_nodes = message["result"]["nodes"]
socket.close()

print(f"Total accessibility nodes: {len(ax_nodes)}")

What Does This Code Do?

  • The TARGETS_URL value points to Chrome's list of inspectable targets.
  • The next() expression selects the page target titled Hacker News.
  • The webSocketDebuggerUrl field supplies the address for that specific page.
  • The Accessibility.getFullAXTree request asks Chrome for every accessibility node in the page.
  • The ax_nodes variable stores the returned node list before len() counts it.
  • Save find_buttons.py.
Run the tree request

Chrome may ask whether to allow the first local debugging connection from your script.

  • Approve the local debugging request if Chrome displays a permission prompt.

Before you run the script, make a quick prediction about whether Chrome returns one node or a full set representing the page.

  • Run the accessibility-tree request by entering:
python3 find_buttons.py

What Should I See?

You'll see Total accessibility nodes: followed by a number. The exact count can change as Hacker News changes.

You have the connection working. Chrome is now handing your script live page structure.

No Node Count?

Keep the Hacker News tab open while the script runs. Confirm that remote debugging remains enabled on localhost:9222.

If the script cannot find the target, check that the tab title still reads Hacker News.

If the connection or response still fails, ask for help tracing the CDP request.

✔️ Awesome, I've got everything!

Great. Double check that find_buttons.py is saved and that your terminal prints the total accessibility-tree node count.

ⓧ I'd like to double check the full code

Compare your saved find_buttons.py with this complete version.

import json
import urllib.request
from websocket import create_connection

TARGETS_URL = "http://localhost:9222/json"
HACKER_NEWS_TITLE = "Hacker News"

# Ask Chrome for every inspectable target on the debugging port
with urllib.request.urlopen(TARGETS_URL) as response:
    targets = json.load(response)

# Select the page target that belongs to the open Hacker News tab
hacker_news_target = next(
    target for target in targets
    if target["type"] == "page" and target["title"] == HACKER_NEWS_TITLE
)

# Connect directly to the selected page without a browser origin header
socket = create_connection(hacker_news_target["webSocketDebuggerUrl"], suppress_origin=True)

# Request the complete accessibility tree and keep its nodes in memory
socket.send(json.dumps({"id": 1, "method": "Accessibility.getFullAXTree"}))
message = json.loads(socket.recv())
ax_nodes = message["result"]["nodes"]
socket.close()

print(f"Total accessibility nodes: {len(ax_nodes)}")

Chrome is returning the complete accessibility tree to your script. Next up, you will filter that large tree down to the elements a user can click.

Filter Clickable Elements

Your script now pulls the full accessibility tree from Chrome. That tree contains useful controls alongside many nodes that an agent cannot click.

In this step, you will filter the tree by role. You will turn the result into a numbered list that exposes each control's role, accessible name, and node ID.

In this step, get ready to:
  • Keep nodes with link, button, or textbox roles.
  • Build a numbered list of clickable elements.
  • Compare the clickable count with the full accessibility-tree count.
Filter nodes by role

An AX role describes what an accessibility node does. Filtering on that semantic value keeps the controls that matter without depending on their HTML tags.

  • Return to find_buttons.py in the editor from the previous step.
  • Find the existing line that prints the total accessibility-tree node count.
  • Insert the filtering logic immediately below that line by copying this code:
# Keep only controls that an agent can choose
clickable_roles = {"link", "button", "textbox"}
clickable_elements = [
    node
    for node in nodes
    if node.get("role", {}).get("value") in clickable_roles
]
print(f"Clickable elements: {len(clickable_elements)}")

What does this code do?

  • The clickable_roles set defines the three kinds of control that the script keeps.
  • The list comprehension reads the role value from every node in nodes.
  • The membership check stores matching nodes in clickable_elements.
  • The final line prints the size of the filtered list.
  • Save find_buttons.py.

Before you rerun the script, predict whether the clickable count will be larger or smaller than the total node count.

  • Rerun the script with the same terminal command you used in the previous step.

You should see the total AX node count followed by a smaller clickable-element count. The exact numbers depend on the current Hacker News page.

Good progress: your script now separates page controls from the surrounding accessibility data.

Clickable count missing or zero?

  • Confirm that Hacker News is still open in the remotely controlled Chrome window.
  • Check that clickable_elements is defined after the full node list is stored in nodes.
  • Check that the role strings are lowercase link, button, and textbox.

Help me debug an empty clickable-element list.

Print the numbered element list

The accessible name tells you how a user identifies a control. The backend DOM node ID gives later protocol commands a handle for locating that control on the page.

  • Find the new print(f"Clickable elements: {len(clickable_elements)}") line in find_buttons.py.
  • Insert the numbered output loop immediately below that line by copying this code:
# Print the details needed to identify each retained control
for number, node in enumerate(clickable_elements, start=1):
    role = node.get("role", {}).get("value", "")
    name = node.get("name", {}).get("value", "(unnamed)")
    node_id = node.get("backendDOMNodeId", "n/a")
    print(f"{number}. {role} | {name} | node {node_id}")

What does this output show?

  • The enumerate() call assigns each retained control a number beginning at 1.
  • The role describes the control's function.
  • The accessible name provides the label that a user encounters.
  • The backendDOMNodeId value connects the accessibility node to its browser element.
  • Save find_buttons.py.

Before you rerun the script, predict what information the first numbered row will contain.

  • Rerun the script with the same terminal command you used earlier.

You should see a numbered list below the two counts. Each row should contain a role, an accessible name, and a node ID.

List missing names or node IDs?

  • Check that the name comes from the node's name value.
  • Check that the node ID comes from backendDOMNodeId.
  • Keep the fallback values so an incomplete node prints as (unnamed) or n/a.

Help me debug the numbered accessibility list.

✔️ Awesome, I've got everything!

Your saved script now prints the full AX node count, the filtered count, and a numbered list of clickable elements.

ⓧ I'd like to double check the full code

Compare your find_buttons.py file with this cumulative version.

import json
import urllib.request

import websocket

# Discover the pages exposed by Chrome remote debugging
with urllib.request.urlopen("http://localhost:9222/json") as response:
    targets = json.load(response)

# Select the Hacker News page and connect to its debugger socket
target = next(
    target
    for target in targets
    if "news.ycombinator.com" in target["url"]
)
ws = websocket.create_connection(target["webSocketDebuggerUrl"])

# Request the complete accessibility tree from the selected page
request = {"id": 1, "method": "Accessibility.getFullAXTree"}
ws.send(json.dumps(request))
message = json.loads(ws.recv())
nodes = message["result"]["nodes"]
print(f"Total AX nodes: {len(nodes)}")

# Keep only controls that an agent can choose
clickable_roles = {"link", "button", "textbox"}
clickable_elements = [
    node
    for node in nodes
    if node.get("role", {}).get("value") in clickable_roles
]
print(f"Clickable elements: {len(clickable_elements)}")

# Print the details needed to identify each retained control
for number, node in enumerate(clickable_elements, start=1):
    role = node.get("role", {}).get("value", "")
    name = node.get("name", {}).get("value", "(unnamed)")
    node_id = node.get("backendDOMNodeId", "n/a")
    print(f"{number}. {role} | {name} | node {node_id}")

Your script can now turn a noisy accessibility tree into a compact menu of page controls. Next up, you will save that menu as JSON and compare its size with a screenshot.

Save and Compare the Element List

Your script now turns the full accessibility tree into a numbered set of elements an agent can use. That is a useful milestone because the page has become a compact menu of actions.

A terminal list disappears when the session ends. Saving the same data as JSON gives you a reusable artifact that you can compare with a full-page screenshot.

In this step, get ready to:
  • Save the filtered elements in a reusable JSON file.
  • Capture the Hacker News page as a full-page PNG file.
  • Print both file sizes for a direct comparison.
Save the clickable elements as JSON

JSON preserves each element as structured text. The saved objects keep the number plus the role plus the accessible name plus the node identifiers needed for selection.

  • In find_buttons.py, place the JSON-saving block immediately after the loop that prints the numbered elements by adding this code:
# Save the agent-ready element list as readable JSON.
with open("clickable_elements.json", "w", encoding="utf-8") as json_file:
    json.dump(clickable_elements, json_file, indent=2)

print("Saved clickable_elements.json")

What does this code do?

  • The file-writing block serializes every dictionary in clickable_elements into clickable_elements.json.
  • The indent=2 setting formats the data across readable lines.
  • Each saved object retains the information your later selection step needs.
  • Save find_buttons.py.
  • Switch back to the terminal from earlier.
  • Rerun find_buttons.py with the same Python command from the previous step.

You'll see Saved clickable_elements.json after the numbered element list. Good work. Your filtered page data now survives beyond the terminal session.

JSON file missing?

Check that the new block sits after clickable_elements has been populated. Confirm that you saved find_buttons.py before rerunning it.

Make sure the Hacker News tab remains open in the remote-debugging Chrome window.

Help me fix the JSON export.

Capture a full-page screenshot

The Chrome DevTools Protocol can capture pixels beyond the visible viewport. This gives you a full-page image of the same Hacker News tab represented by the JSON file.

  • In find_buttons.py, place the screenshot block after the JSON-saving block by adding this code:
# Capture the entire page and decode Chrome's base64 image data.
import base64

screenshot_result = send_cdp_command(
    socket,
    2,
    "Page.captureScreenshot",
    {"format": "png", "captureBeyondViewport": True},
)
screenshot_bytes = base64.b64decode(screenshot_result["data"])

with open("hacker_news_full_page.png", "wb") as screenshot_file:
    screenshot_file.write(screenshot_bytes)

print("Saved hacker_news_full_page.png")

How does the screenshot become a file?

  • The Page.captureScreenshot command requests a png image from the connected page.
  • The captureBeyondViewport parameter includes page content outside the current browser view.
  • The response carries base64 image data. Python decodes that data before writing hacker_news_full_page.png.
  • Save find_buttons.py.
  • Switch back to the terminal from earlier.
  • Rerun find_buttons.py with the same Python command from the previous step.

You'll see Saved hacker_news_full_page.png after the JSON confirmation. You now have pixel data for the complete Hacker News page.

Screenshot not created?

Confirm that Chrome remote debugging is still enabled. Keep the Hacker News tab open while the script runs.

Check that socket still refers to the active Hacker News connection when the screenshot command runs.

Help me fix the screenshot capture.

Compare both file sizes

Byte counts give both page representations a shared measurement. You can now compare structured element data with a rendered image of the same page.

  • In find_buttons.py, place the size comparison after the screenshot block by adding this code:
# Measure both artifacts in bytes for a direct comparison.
import os

json_size = os.path.getsize("clickable_elements.json")
screenshot_size = os.path.getsize("hacker_news_full_page.png")
print(
    f"clickable_elements.json: {json_size:,} bytes | "
    f"hacker_news_full_page.png: {screenshot_size:,} bytes"
)

What does the comparison measure?

  • The first measurement reads the exact byte size of clickable_elements.json.
  • The second measurement reads the exact byte size of hacker_news_full_page.png.
  • The final line places both measurements together so the storage difference is easy to inspect.
  • Save find_buttons.py.

Before you rerun the script, which file do you expect to use more bytes?

  • Rerun find_buttons.py with the same Python command from the previous step.

You'll see one comparison line containing the byte size of clickable_elements.json plus the byte size of hacker_news_full_page.png. The PNG will usually be larger because it stores every rendered pixel.

That's the comparison complete. Your terminal now shows the storage cost of structured interaction data beside a screenshot of the same page.

Only one file size shown?

Confirm that both output files exist beside find_buttons.py. Check each filename for matching underscores plus matching file extensions.

Rerun the completed script so it recreates both artifacts before measuring them.

Help me fix the file comparison.

✔️ Awesome, I've got everything!

Great. Double check that clickable_elements.json plus hacker_news_full_page.png are saved beside find_buttons.py.

ⓧ I'd like to double check the full code

Compare your cumulative find_buttons.py file with this version:

import json
import requests
import websocket

DEBUG_URL = "http://localhost:9222/json"
CLICKABLE_ROLES = {"link", "button", "textbox"}


def send_cdp_command(socket, command_id, method, params=None):
    # Match each response to the command that requested it.
    message = {"id": command_id, "method": method}
    if params is not None:
        message["params"] = params
    socket.send(json.dumps(message))

    while True:
        response = json.loads(socket.recv())
        if response.get("id") == command_id:
            return response["result"]


# Find the open Hacker News page target.
targets = requests.get(DEBUG_URL, timeout=5).json()
target = next(
    item
    for item in targets
    if item.get("type") == "page"
    and "news.ycombinator.com" in item.get("url", "")
)

socket = websocket.create_connection(target["webSocketDebuggerUrl"])

# Read Chrome's full accessibility tree.
ax_result = send_cdp_command(socket, 1, "Accessibility.getFullAXTree")
ax_nodes = ax_result["nodes"]
print(f"Total accessibility nodes: {len(ax_nodes)}")

# Keep roles that represent elements a user can activate or type into.
clickable_elements = []
for node in ax_nodes:
    role = node.get("role", {}).get("value", "")
    if role not in CLICKABLE_ROLES:
        continue

    name = node.get("name", {}).get("value", "")
    element = {
        "number": len(clickable_elements) + 1,
        "role": role,
        "name": name,
        "nodeId": node["nodeId"],
        "backendDOMNodeId": node.get("backendDOMNodeId"),
    }
    clickable_elements.append(element)
    print(
        f'{element["number"]}. {element["role"]}: '
        f'{element["name"]} ({element["nodeId"]})'
    )

# Save the agent-ready element list as readable JSON.
with open("clickable_elements.json", "w", encoding="utf-8") as json_file:
    json.dump(clickable_elements, json_file, indent=2)

print("Saved clickable_elements.json")

# Capture the entire page and decode Chrome's base64 image data.
import base64

screenshot_result = send_cdp_command(
    socket,
    2,
    "Page.captureScreenshot",
    {"format": "png", "captureBeyondViewport": True},
)
screenshot_bytes = base64.b64decode(screenshot_result["data"])

with open("hacker_news_full_page.png", "wb") as screenshot_file:
    screenshot_file.write(screenshot_bytes)

print("Saved hacker_news_full_page.png")

# Measure both artifacts in bytes for a direct comparison.
import os

json_size = os.path.getsize("clickable_elements.json")
screenshot_size = os.path.getsize("hacker_news_full_page.png")
print(
    f"clickable_elements.json: {json_size:,} bytes | "
    f"hacker_news_full_page.png: {screenshot_size:,} bytes"
)

socket.close()

Your compact element list now has a screenshot benchmark beside it. Next, you'll choose one saved element plus reveal its exact position on the live page.

Highlight a Selected Element

The filtered accessibility tree has already reduced the page to targets your agent can use. Each number still points to accessibility data, so your script needs a path back to the live page.

The Chrome DevTools Protocol can resolve the selected node to a DOM node. Chrome can then return its box coordinates.

In this step, you will choose one result from the terminal. You will use the DevTools Overlay domain to outline that exact element on Hacker News.

In this step, get ready to:
  • Accept a number from the existing clickable-element list.
  • Resolve the selected accessibility node to its page coordinates.
  • Draw an outline around the selected element on Hacker News.
Accept a numbered selection

The displayed number gives you a stable way to retrieve one record from the existing filtered list. That record contains the node ID required for the browser lookup.

How does the selection flow work?

  • The prompt asks for one number from the printed clickable-element list.
  • The entered number maps to the matching record in that list.
  • The matching record supplies the accessibility node ID for the next browser request.
  • In find_buttons.py, add one selection block directly below the loop that prints the numbered clickable-element list.
  • Save find_buttons.py.
  • Rerun the script in the terminal with the same command you used in the previous step.
  • Enter one number currently shown in the clickable-element list.

You will see the numbered results followed by a new prompt for an element number. Good progress. Your terminal choice now points to one accessibility record.

Selection prompt missing?

If the list prints without a prompt, check that the selection block comes after the existing print loop. If the prompt appears once per element, move the block outside that loop.

If the selection fails, enter one of the numbers printed during the current run.

Help me debug the numbered selection flow.

Resolve and highlight the element

The selected accessibility node needs page geometry before your script can target it. The box coordinates also provide the centre point that the next step uses for a mouse click.

How does the highlight flow work?

  • The selected record supplies its existing accessibility node ID.
  • Chrome resolves that accessibility node to a DOM node.
  • The DOM.getBoxModel call returns the selected element's box coordinates.
  • The box coordinates provide the horizontal centre.
  • The box coordinates provide the vertical centre.
  • The Overlay domain draws a visible outline around the resolved DOM node.
  • In find_buttons.py, add one geometry and overlay block directly after the selection block.
  • Save find_buttons.py.

Before you rerun the script, picture which visible Hacker News link should gain the outline.

  • Rerun the script in the terminal with the same command you used earlier.
  • Choose the number for a link currently visible in the Hacker News tab.
  • Enter that number at the selection prompt.
  • Switch back to the Hacker News tab from earlier.

You will see an outline around the link that matches your terminal selection. That visible outline proves your script connected one accessibility record to the correct page element.

No outline on the page?

Choose a number for a link currently visible in the Hacker News viewport. An element outside the viewport can place its box beyond the visible page.

Confirm that the resolved node ID belongs to the selected record. Confirm that the Overlay domain is enabled before the highlight request.

Help me debug the missing overlay highlight.

✔️ Awesome, I've got everything!

Great. Your selected Hacker News element now has a visible outline from the DevTools overlay.

ⓧ I'd like to double check the full code

Check that your cumulative find_buttons.py follows this complete flow:

  • The existing script still discovers the Hacker News target through its WebSocket debugger URL.
  • The existing script still requests the full accessibility tree.
  • The existing script still filters the tree into a numbered clickable-element list.
  • The existing script still saves clickable_elements.json.
  • The existing script still saves hacker_news_full_page.png.
  • The new selection flow accepts one displayed element number.
  • The selected record supplies its accessibility node ID.
  • The new geometry flow resolves that ID to a DOM node.
  • The DOM.getBoxModel call retrieves the selected element's box coordinates.
  • The script calculates the centre of the returned box.
  • The Overlay domain outlines the selected element on the live Hacker News page.

Your script can now prove that its terminal selection matches a real element on the page. Next, you will send a mouse click to the centre of that highlighted box.

Click the Selected Element

The outlined Hacker News element proves your script can connect accessibility data to a real place on the page. The final gap is turning that location into the press and release sequence of a mouse click.

In this step, you will use the Chrome DevTools Protocol Input domain to activate the selected element. The open tab will navigate to the link chosen through your terminal.

In this step, get ready to:
  • Dispatch a mouse press at the selected element's centre.
  • Dispatch a mouse release at the same coordinates.
  • Verify that the selected Hacker News link opens.
Dispatch the mouse press and release

A browser click combines a press with a release at the same coordinates. CDP receives each part through Input.dispatchMouseEvent.

  • In find_buttons.py, scroll to the block that highlights the selected element after calculating its centre coordinates.
  • Add the click events directly after the highlight block by copying this code:
# Complete one click with a press and release at the same centre point.
send_cdp_command(
    ws,
    "Input.dispatchMouseEvent",
    {
        "type": "mousePressed",
        "x": center_x,
        "y": center_y,
        "button": "left",
        "clickCount": 1,
    },
)

send_cdp_command(
    ws,
    "Input.dispatchMouseEvent",
    {
        "type": "mouseReleased",
        "x": center_x,
        "y": center_y,
        "button": "left",
        "clickCount": 1,
    },
)

print("Clicked the selected element.")

What does this code do?

  • The mousePressed event begins the click at the calculated centre.
  • The mouseReleased event completes the click at the same point.
  • The left value selects the primary mouse button.
  • The clickCount value of 1 describes one click.
  • Save find_buttons.py.
  • Confirm that the two Input.dispatchMouseEvent calls appear directly below the highlight block.
Run the completed click flow

Before you rerun the script, do you expect the outline to remain in place or the selected link to open?

  • Return to the terminal from the previous step.
  • Press the Up Arrow key to recall the command that ran find_buttons.py.
  • Press Enter to rerun the script.
  • Wait for the numbered clickable-element list to appear.
  • Enter the number of the Hacker News link you want to activate.
  • Switch back to the Chrome tab after the script prints its click confirmation.

You'll see the selected link open in the Chrome tab. Your terminal will show Clicked the selected element. after dispatching both mouse events.

You have closed the full automation loop. Your script now discovers an element through accessibility data and activates it through calculated coordinates.

Link stayed on the same page?

Check that both event calls use center_x with center_y. Different coordinates can turn the press and release into separate interactions.

Confirm that mousePressed appears before mouseReleased.

If the element is outside the visible browser area, choose a link that you can currently see before rerunning the script.

Help me debug why my selected element stays highlighted without opening.

✔️ Awesome, I've got everything!

Great work. Double check that find_buttons.py is saved and that the selected link opened in Chrome.

ⓧ I'd like to double check the full code

Compare the final click section of find_buttons.py with this reference:

# Complete one click with a press and release at the same centre point.
send_cdp_command(
    ws,
    "Input.dispatchMouseEvent",
    {
        "type": "mousePressed",
        "x": center_x,
        "y": center_y,
        "button": "left",
        "clickCount": 1,
    },
)

send_cdp_command(
    ws,
    "Input.dispatchMouseEvent",
    {
        "type": "mouseReleased",
        "x": center_x,
        "y": center_y,
        "button": "left",
        "clickCount": 1,
    },
)

print("Clicked the selected element.")

Secret mission

Prove Text Input Focus

Your script can open a link. Now use the same coordinate click to focus a Hacker News text input. A typed phrase will prove that the field received keyboard focus.

Clean Up Your Resources

Clean Up Your Resources

Your script and comparison files stay on your own computer. This project has no ongoing cost.

Choose whether to keep the reusable files, pause browser control, or delete the files entirely.

Resources you used:

  • Remote-debugging access for the running Chrome session at no cost.
  • Local find_buttons.py script at no cost.
  • Local clickable_elements.json file containing the filtered accessibility data at no cost.
  • Local hacker_news_full_page.png screenshot at no cost.

Keep everything running

No file cleanup is needed. Choose this if you are still testing the script during your current browser session.

  • Keep Chrome open while you continue testing the script.
  • Return to Chrome's inspect page when you finish testing.
  • Switch off the remote-debugging toggle after your final test.
  • Keep find_buttons.py, clickable_elements.json, and hacker_news_full_page.png for future browser-agent experiments.

Pause - I'll come back to this later

Shut down browser control while keeping your reusable script and comparison files. You can enable remote debugging again when you return.

  • Return to Chrome's inspect page.
  • Switch off the remote-debugging toggle.
  • Close Chrome.
  • Leave find_buttons.py, clickable_elements.json, and hacker_news_full_page.png in their current folder.

Delete - I don't want to use this again

Remove the reusable files if you want a clean start. Deleting these files is permanent.

Your regular browser settings stay in place after remote debugging is switched off.

  • Return to Chrome's inspect page.
  • Switch off the remote-debugging toggle.
  • Close Chrome.

macOS

  • Press Cmd+Space to open macOS search.
  • Type Finder.
  • Press Enter to open Finder.
  • Search for find_buttons.py.
  • Open the folder containing find_buttons.py.
  • Select find_buttons.py, clickable_elements.json, and hacker_news_full_page.png.
  • Move the selected files to the Trash.
  • Empty the Trash to complete permanent removal.

Windows

  • Press the Windows key to open Windows search.
  • Type File Explorer.
  • Press Enter to open File Explorer.
  • Search for find_buttons.py.
  • Open the folder containing find_buttons.py.
  • Select find_buttons.py, clickable_elements.json, and hacker_news_full_page.png.
  • Move the selected files to the Recycle Bin.
  • Empty the Recycle Bin to complete permanent removal.

Nice Work!

You did it! Your reusable Python tool now turns Chrome's accessibility tree into a numbered map of clickable elements.

You've learned how to:

  • Connected Python to a live Chrome tab through the Chrome DevTools Protocol. Used Accessibility.getFullAXTree to retrieve the complete node list.
  • Built a numbered inventory of clickable elements by filtering for links, buttons, and text inputs. Saved that inventory to clickable_elements.json for reuse.
  • Resolved an element's box with DOM.getBoxModel. Used coordinate-based interaction to highlight the element before clicking its centre through Input.dispatchMouseEvent.
  • Completed the optional Secret Mission to extend the browser-control workflow beyond the main project.

Ready to quiz yourself?