Script a Desktop App with Accessibility

Write a script that operates a desktop app through its accessibility tree.

Introduction

30 Second Summary

Desktop apps often keep useful work behind buttons that only a person can click. That leaves automation stuck when an app offers no direct connection for code.

In this project, you will use agent-desktop to read a real app through its accessibility tree. The finished script uses fresh element references to perform each action before confirming the result.

What You'll Build

You will run one command and watch Finder reach the intended end state while your pointer stays exactly where it is.

By the end of this project, you'll have:

  • A live accessibility snapshot that lists the app's actionable elements as JSON with a reference for each one.
  • A reference-driven desktop action that changes Finder without moving your pointer. You will also see how the app reports a stale reference.
  • A one-command script that takes fresh snapshots before acting. It also confirms that the app reaches the intended end state.
  • An optional Secret Mission to push your desktop automation skills further.

Are there any prerequisites?

You will need a laptop running macOS, Windows, or Linux. You will also need an npm-compatible package manager.

Before We Start

Your future automation script needs a real desktop app it can inspect through an accessibility tree. This project uses Finder on macOS as its test app.

The agent-desktop CLI installs through npm. You'll use Terminal to confirm that Node.js is ready before the installation step.

In this step, get ready to:
  • Confirm Finder is available as the desktop app for this project.
  • Start Terminal with macOS search.
  • Verify the Node.js toolchain prints its version numbers.
Confirm your Mac and Finder

Finder is the app you'll operate later in the project. A visible Finder window proves that your test app is available on this Mac.

  • Press Command-Space bar to open Spotlight.
  • Type Finder into Spotlight.
  • Press the Return key to open the selected result.

Your test app is ready. You should see a Finder window displaying files on your Mac.

Open Terminal and check Node.js

Node.js runs the command-line tools used in this project. The npm package manager comes with Node.js.

  • Press Command-Space bar to open Spotlight.
  • Type Terminal into Spotlight.
  • Press the Return key to open the selected result.

You should see a Terminal window with a command prompt. This is where you'll run every agent-desktop command in the project.

Before you check, do you expect both tools to print a version number?

  • Check the Node.js toolchain by running these commands:
node --version
npm --version

What Do These Checks Prove?

  • The node command prints the installed Node.js version.
  • The npm command prints the installed package manager version.
  • Two version numbers confirm that Terminal can reach both tools.

✔️ I see two version numbers

That's your command-line foundation sorted. Both results confirm that Node.js and npm are available in this Terminal window.

  • Keep Terminal open for the next step.

ⓧ Node.js or npm is unavailable

Installing Node.js also provides npm. macOS may ask you to approve the installer during this process.

  • Visit the official Node.js download page.
  • Download the LTS installer for macOS.
  • Run the installer after the download finishes.
  • Approve the installation if macOS asks for permission.
  • Complete the installer using its default options.
  • Return to the Terminal window from earlier.
  • Check both tools again by running:
node --version
npm --version

What Happens After Installation?

The installer adds Node.js plus npm to your Mac. These checks confirm that Terminal can find both commands.

You should now see one Node.js version number followed by one npm version number.

Still Missing a Version Number?

  • Confirm that the Node.js installer finished successfully.
  • Press Command-N in Terminal to start a fresh window.
  • Run the two version checks again in the fresh window.

Still stuck? Help me get Node.js and npm working in Terminal on macOS.

Your Mac now has the desktop app and toolchain this project needs. Next, you'll install the agent-desktop CLI.

Install agent-desktop

Your automation script needs a reliable way to read Finder through its accessibility tree. The agent-desktop CLI provides that connection.

This step puts the command on your macOS computer. A version check proves Terminal can find the executable.

In this step, get ready to:
  • Install the agent-desktop CLI globally with npm.
  • Confirm Terminal can run the installed CLI.
Set up the desktop command

A global npm package makes its command available across your Terminal sessions. This gives each later step access to the same CLI.

  • Add agent-desktop to your global npm packages by running this command:
npm install -g agent-desktop

What does this command do?

  • The npm install command downloads the package through npm.
  • The -g option installs the package globally.
  • The agent-desktop package provides the Terminal command used throughout this project.

Your Terminal shows npm's progress while the package installs. The prompt returns when the installation finishes.

Installation not finishing?

  • Check your internet connection if npm cannot download the package.
  • Review your npm global package permissions if npm reports a permissions problem.
  • Retry the installation command after npm can write to its global package location.

Still stuck? Help me troubleshoot my agent-desktop npm installation.

Prove the installation works

The version command starts the installed executable. Its output confirms that Terminal can resolve the command.

Before you test it, what do you expect Terminal to print when the CLI is available?

  • Check the installed CLI by running this command:
agent-desktop version

What should I see?

You'll see the installed version printed in Terminal. This output proves Terminal resolved the agent-desktop executable.

That's your first desktop-agent checkpoint complete. The command is ready for the rest of the project.

No version printed?

  • Run the npm installation command above again if Terminal cannot find agent-desktop.
  • Confirm the installation finishes before repeating the version check.

Need another pair of eyes? Help me work out why agent-desktop will not print its version.

Your CLI is installed. Next, you'll grant the macOS permission it needs to read Finder.

Grant Desktop Permissions

Your agent-desktop CLI passed its version check, so the installation is ready. macOS still blocks desktop control until you approve access.

The permission gives the CLI access to the accessibility tree that exposes an app's controls. You'll use the permission guide before confirming that macOS granted access.

In this step, get ready to:
  • Start the built-in permission guide.
  • Grant accessibility access to the terminal or process running the CLI.
  • Confirm the permission is enabled in System Settings.
Run the permission guide

macOS manages accessibility access at the operating system level. The CLI includes a guide that points you to the required permission.

  • Start the macOS permission guide from the Terminal session you used earlier by running:
agent-desktop permissions --guide

What does this command do?

  • The command asks the installed CLI to display its permission guidance for macOS.
  • The guide identifies the accessibility access required by the process that operates desktop apps.
  • macOS records the final permission after you approve that process in System Settings.

You should see guidance for granting macOS accessibility access. This confirms the permission workflow has started.

Don't see the permission guide?

  • Return to the Terminal session where the earlier version check succeeded.
  • Run the permission guide command above again.
  • Follow the location printed by the guide if you have trouble finding the permission page.

Still stuck? Help me troubleshoot the agent-desktop permission guide on macOS.

Approve accessibility access

Accessibility access is powerful because it lets software interact with controls inside desktop apps. Grant it only to the terminal or process that runs agent-desktop.

  • Follow the location displayed by the guide to reach accessibility permissions in System Settings.
  • Locate the terminal or process that runs agent-desktop.
  • Enable accessibility permission beside that entry.

Before you verify the result, which terminal or process entry do you expect macOS to show as allowed?

  • Return to the accessibility permission list shown by the guide.
  • Confirm the control beside your terminal or agent-desktop process is enabled.

You'll see accessibility permission enabled for the process running the CLI. That is the security gate cleared: agent-desktop can now request access to desktop app controls.

Permission still not enabled?

  • Check that you enabled the terminal or process that actually launched the CLI.
  • Use the entry identified by the permission guide if several terminal applications appear.
  • Run the guide again from the same Terminal session if the required process is missing.

Need another pair of eyes? Help me identify the correct macOS accessibility permission entry.

Your CLI is installed with the macOS permission it needs. Next, you'll use it to turn Finder's accessible controls into actionable references.

Inspect Finder Elements

Your Terminal can now read Finder through macOS's accessibility system. The next challenge is turning that access into a map the CLI can use.

An accessibility tree represents an app as structured elements. You'll capture Finder's tree as JSON.

Every actionable element in that snapshot carries a full ref. That ref is the handle a later command can target.

In this step, get ready to:
  • Bring Finder into view with a non-sensitive window state.
  • Produce an interactive JSON snapshot from Terminal.
  • Identify an actionable element by its full ref.
Bring Finder into view

A snapshot reflects the app state available when the command runs. Starting from a visible Finder window makes the output easier to recognize.

Finder filenames can appear in snapshot data. Choose a location that contains no private filenames before you capture it.

  • Select the Finder icon in the Dock.
  • Navigate to a location that does not show private filenames.

You should see a Finder window containing only information you are comfortable including in terminal output.

Capture the interactive snapshot

Interactive mode focuses the snapshot on elements the agent can operate. The resulting JSON gives each actionable element a full ref.

Before you run this, predict which visible Finder control will appear in the JSON.

  • Capture Finder's interactive accessibility snapshot by running this command in Terminal:
agent-desktop snapshot --app Finder -i

What does this command do?

  • The agent-desktop snapshot command captures the app's current accessibility tree.
  • The --app Finder option sets Finder as the target app.
  • The -i option focuses the output on interactive elements.
  • The resulting JSON attaches a full ref to every actionable element. Each ref identifies an element within this snapshot.

You should see JSON listing Finder's actionable elements. Each listed element should include its full ref.

You have turned Finder's live interface into structured data that a script can target.

Don't see Finder JSON?

  • Make a Finder window visible before retrying the snapshot.
  • Confirm the macOS accessibility permission from the previous step remains enabled.
  • Rerun the command from the same Terminal session that has permission.

Still stuck? Help me troubleshoot why the Finder snapshot is not returning JSON.

Read an actionable element's ref

The next action needs a full ref from this exact snapshot. Recording one now keeps the ref tied to an element you actually saw.

Before you scan the output, predict which visible Finder control will be easiest to recognize.

  • Scan the JSON for an actionable Finder element you recognize.
  • Record that element's full ref here: your Finder element full ref.
  • Crop or redact any private filenames before capturing your evidence.

Your recorded value should match the full ref in the JSON entry for the recognized element. That connection proves you can map a visible Finder control to its accessibility-tree entry.

Your Finder snapshot now shows actionable elements with full refs. Next up, you'll use one ref to change the app while your pointer stays put.

Act and Handle Stale Refs

Your Finder snapshot is a map of one moment in the app. The next test proves that agent-desktop can turn that map into a real action.

In this step, you will click a Finder element through its full ref without moving your physical pointer. A second snapshot lets you test the earlier ref against the changed accessibility tree.

In this step, get ready to:
  • Use a full ref to change Finder without moving your physical pointer.
  • Capture a second snapshot to inspect the changed accessibility tree.
  • Reuse the earlier ref to reveal the safety boundary around stale UI state.
Choose a state-changing Finder ref

A full ref contains the snapshot identifier before the colon. This ties your action to the Finder state that produced the ref.

  • Return to the interactive Finder snapshot output in Terminal.
  • Find an element whose available_actions list includes Click.
  • Choose an element that visibly changes Finder without revealing private content.
  • Copy its full ref here: your full Finder ref.
  • Place your physical pointer over a blank part of the desktop.

Before you run the action, what do you expect Finder to do while your pointer stays in place?

  • Activate the selected Finder element by running this command:
agent-desktop click [[FINDER_REF="your full Finder ref"]]

What does this command do?

  • The click command resolves your selected ref against the live Finder element before activating it.
  • The default ref action uses the accessibility API. Your physical pointer remains where you placed it.
  • A successful command prints a JSON envelope whose ok value confirms the action.
  • Check Finder to confirm the selected location or control changed.
  • Check your physical pointer to confirm it stayed in place.

That worked. Finder changed through its accessibility interface while your pointer stayed still.

Finder did not change?

  • Check that the copied value includes the snapshot prefix before the colon.
  • Choose a different element whose available_actions list includes Click.
  • Repeat the click with the new element's full ref.

Still stuck? Help me diagnose why my Finder ref did not produce a visible action.

Refresh the Finder snapshot

Finder now has a different UI state from the one your first snapshot described. Reading it again gives you a current accessibility tree for the next decision.

  • Capture the changed Finder state by running this command:
agent-desktop snapshot --app Finder -i

What does this command show?

  • The snapshot command reads the current Finder accessibility tree.
  • The -i option limits the result to interactive elements.
  • The new output contains the current snapshot_id plus full refs for the current Finder state.
  • Inspect the new JSON to find the changed Finder state.

You should see the current Finder tree with its own snapshot_id. Its full refs describe the UI after your click.

Snapshot not showing the changed state?

  • Confirm Finder still shows the view created by your click.
  • Run the snapshot command again after Finder finishes updating.

Need another pair of eyes? Help me compare my two Finder snapshots.

Reuse the earlier ref

The saved value your full Finder ref came from the first snapshot. Agent-desktop now compares that earlier observation with the live element before acting.

Before you reuse the earlier ref, do you think the action will succeed or fail?

  • Test the earlier Finder ref by running this command again:
agent-desktop click [[FINDER_REF="your full Finder ref"]]

Why did the command stop?

Your command should return an error envelope with the code STALE_REF. The earlier ref no longer matches a live element strongly enough for a safe action.

Strict ref resolution fails closed when the stored identity is stale. Agent-desktop stops before it can click the wrong Finder element.

You have now seen stale-ref protection work. The fresh snapshot showed Finder's current state while the old ref produced STALE_REF.

Did the earlier ref act again?

  • Confirm the first click visibly changed Finder.
  • Confirm you captured a fresh snapshot after the change.
  • Use the saved earlier ref from your full Finder ref for the final test.

Still missing the stale-ref result? Help me reproduce STALE_REF with my two Finder snapshots.

Your action loop now has a safety rule: observe, act, observe again. Next, you will put that loop into a script that refreshes refs before every action.

Automate the Finder Workflow

Your manual Finder test exposed the weak point: refs expire after the interface changes. Copying an old ref into another action can trigger STALE_REF.

Now you'll put the reliable loop into an executable shell script. The agent-desktop CLI takes a fresh snapshot before the script clicks anything.

A final snapshot reads Finder's window title. That gives your automation a clear pass or fail result.

In this step, get ready to:
  • Create operate-finder.sh with a reusable snapshot helper.
  • Select the full Downloads ref from the current JSON snapshot.
  • Run the Finder interaction from one command.
Create the snapshot helper

The script needs one consistent way to observe Finder. A helper function keeps the snapshot command identical wherever the workflow needs fresh refs.

  • Create operate-finder.sh on your Desktop by running this command in the Terminal from earlier:
cat > ~/Desktop/operate-finder.sh <<'SCRIPT'
#!/bin/sh

# Use Downloads as the Finder destination.
target_name="Downloads"

# Take a new interactive snapshot before an action.
snapshot_finder() {
  agent-desktop snapshot --app Finder -i
}
SCRIPT

What does this code do?

  • The target_name value defines the Finder destination that the script expects to reach.
  • The snapshot_finder() helper requests interactive elements from Finder. Each actionable element includes a full ref from that snapshot.
  • The quoted heredoc writes the script exactly as shown. Shell variables inside it remain untouched until the finished script runs.
  • Confirm the opening section exists by running this command:
cat ~/Desktop/operate-finder.sh

What should you see?

You should see the shell header followed by target_name="Downloads" and the snapshot_finder() function. This confirms that operate-finder.sh now exists on your Desktop.

Can't read the new script?

  • Check that the path contains Desktop with a capital D.
  • Repeat the creation command if Terminal reports that operate-finder.sh does not exist.

Still stuck? Help me create operate-finder.sh on my macOS Desktop.

Extract the current full ref

An accessibility snapshot is nested JSON. Your script needs to search that tree for the actionable element named Downloads.

The parser uses Node.js because it is already available on your computer. It returns the element's current ref_id to the shell.

  • Add the recursive ref lookup to operate-finder.sh by running this command:
cat >> ~/Desktop/operate-finder.sh <<'SCRIPT'
# Find the destination's full ref in the current snapshot.
find_ref() {
  SNAPSHOT_JSON="$1" TARGET_NAME="$target_name" node -e '
const input = JSON.parse(process.env.SNAPSHOT_JSON)
const target = process.env.TARGET_NAME

function findRef(value) {
  if (!value || typeof value !== "object") return null
  if (value.name === target && value.ref_id) return value.ref_id
  for (const child of Object.values(value)) {
    const items = Array.isArray(child) ? child : [child]
    for (const item of items) {
      const match = findRef(item)
      if (match) return match
    }
  }
  return null
}

const ref = findRef(input.data.tree)
if (!ref) {
  console.error(`No actionable Finder element named "${target}" was found.`)
  process.exit(1)
}
console.error(`Using fresh ref ${ref} for "${target}".`)
process.stdout.write(ref)
'
}
SCRIPT

How does the ref lookup work?

  • The shell passes the latest snapshot through SNAPSHOT_JSON. The target name enters Node.js through TARGET_NAME.
  • The findRef() function walks each nested object. It stops when the element name matches Downloads.
  • The parser returns the matching full ref. A missing match ends the run before any Finder action occurs.
  • Confirm the ref lookup was appended by running this command:
cat ~/Desktop/operate-finder.sh

What should you see now?

You should see find_ref() below snapshot_finder(). The file should end with the closing brace for the new shell function.

Is the ref lookup incomplete?

  • Repeat the append command if the file ends before the final closing brace.
  • Check that the line containing the single quote appears immediately before the final closing brace.

Need another pair of eyes? Help me check the find_ref function in my Finder automation script.

Run and verify the workflow

The ref lookup becomes useful when each action consumes its output immediately. The script takes another snapshot after the click so the verification reads Finder's resulting state.

  • Add the click sequence and final state check to operate-finder.sh by running this command:
cat >> ~/Desktop/operate-finder.sh <<'SCRIPT'

# Take a snapshot before the click.
snapshot_json="$(snapshot_finder)"
ref="$(find_ref "$snapshot_json")" || exit 1
agent-desktop click "$ref"

# Take a final snapshot.
# Verify Finder's window title.
final_snapshot="$(snapshot_finder)"
FINAL_SNAPSHOT="$final_snapshot" TARGET_NAME="$target_name" node -e '
const input = JSON.parse(process.env.FINAL_SNAPSHOT)
const target = process.env.TARGET_NAME
const title = input.data.window.title

if (title !== target) {
  console.error(`Expected "${target}", but Finder reached "${title}".`)
  process.exitCode = 1
} else {
  console.log(`Confirmed Finder reached "${target}".`)
}
'
SCRIPT

What completes the automation loop?

  • The first snapshot_json value contains the refs that are valid for the upcoming click.
  • The ref value comes directly from that snapshot. The click never reuses a ref from an earlier run.
  • The final_snapshot value captures Finder after the action. Its window title supplies the end-state check.
  • Give your script execute permission by running this command:
chmod u+x ~/Desktop/operate-finder.sh

What does this command do?

The chmod command adds execute permission for the file's owner. That lets macOS run operate-finder.sh as one command.

Before you run the script, do you think it can reach Downloads without reusing the ref from your manual test?

  • Run the complete Finder workflow with this command:
~/Desktop/operate-finder.sh

What should you see?

Finder should switch to Downloads while your physical pointer stays in place. Terminal first prints a fresh qualified ref for that destination.

The final line should say Confirmed Finder reached "Downloads". That message comes from the second snapshot's window title.

That's the full automation loop working: your script observes Finder, acts with a current ref, then verifies the result from fresh state.

Can't find Downloads?

  • Open Finder's Settings if the script reports that no actionable Downloads element was found.
  • Select Sidebar at the top of Finder settings.
  • Enable Downloads in the sidebar list.

If the final title still does not match, help me debug the snapshot, ref lookup, and Finder title check.

✔️ Awesome, I've got everything!

Your executable operate-finder.sh now runs the complete observe, act, observe loop from one command.

ⓧ I'd like to double check the full code

  • Compare your Desktop copy of operate-finder.sh with this complete script:
#!/bin/sh

# Use Downloads as the Finder destination.
target_name="Downloads"

# Take a new interactive snapshot before an action.
snapshot_finder() {
  agent-desktop snapshot --app Finder -i
}
# Find the destination's full ref in the current snapshot.
find_ref() {
  SNAPSHOT_JSON="$1" TARGET_NAME="$target_name" node -e '
const input = JSON.parse(process.env.SNAPSHOT_JSON)
const target = process.env.TARGET_NAME

function findRef(value) {
  if (!value || typeof value !== "object") return null
  if (value.name === target && value.ref_id) return value.ref_id
  for (const child of Object.values(value)) {
    const items = Array.isArray(child) ? child : [child]
    for (const item of items) {
      const match = findRef(item)
      if (match) return match
    }
  }
  return null
}

const ref = findRef(input.data.tree)
if (!ref) {
  console.error(`No actionable Finder element named "${target}" was found.`)
  process.exit(1)
}
console.error(`Using fresh ref ${ref} for "${target}".`)
process.stdout.write(ref)
'
}

# Take a snapshot before the click.
snapshot_json="$(snapshot_finder)"
ref="$(find_ref "$snapshot_json")" || exit 1
agent-desktop click "$ref"

# Take a final snapshot.
# Verify Finder's window title.
final_snapshot="$(snapshot_finder)"
FINAL_SNAPSHOT="$final_snapshot" TARGET_NAME="$target_name" node -e '
const input = JSON.parse(process.env.FINAL_SNAPSHOT)
const target = process.env.TARGET_NAME
const title = input.data.window.title

if (title !== target) {
  console.error(`Expected "${target}", but Finder reached "${title}".`)
  process.exitCode = 1
} else {
  console.log(`Confirmed Finder reached "${target}".`)
}
'

What should match?

Your file should contain the snapshot helper first. The ref lookup should appear next.

The click sequence should use the lookup's fresh ref. The final Node.js check should compare Finder's window title with Downloads.

You now own an executable Finder automation script that refreshes its refs before acting. Its final snapshot proves the desktop app reached the expected state.

Secret mission

Prove the Script Survives UI Drift

Your script works from the Finder state you tested earlier. Now prove it can recover when Finder starts in two different folders. You will run the same one-command automation twice to confirm that fresh refs still reach the expected end state.

Clean Up Your Resources

Clean Up Your Resources

Your Finder automation runs locally with no ongoing cloud costs. Choose whether to keep your resources ready, pause your work, or remove the project artifacts.

Resources you used:

  • Executable automation script named operate-finder.sh.
  • macOS Accessibility permission granted to Terminal or the process running agent-desktop.
  • Any Finder snapshots or screenshots you saved during testing.

Snapshots Can Contain Private Content

Accessibility snapshots can include private content from your screen. Screenshots can capture the same kind of information.

The cleanup options below treat saved captures as temporary files. Your reusable operate-finder.sh script can stay separate from them.

Keep everything running

No action is needed for operate-finder.sh. Choose this if you plan to keep experimenting with Finder automation.

  • Keep operate-finder.sh in its current location.
  • Keep the macOS Accessibility permission enabled.
  • Delete any Finder snapshots you saved during testing.
  • Delete any screenshots you saved during testing.

Your script stays ready for another Finder run. Private captures are no longer stored alongside it.

Pause - I'll come back to this later

Each agent-desktop command ends after it finishes its task. There is no background process to stop.

  • Keep operate-finder.sh in its current location.
  • Keep the macOS Accessibility permission enabled for your next session.
  • Delete any saved Finder snapshots.
  • Delete any saved screenshots.
  • Close the Terminal window after finishing the cleanup.

Your reusable script remains on your Mac. You can continue from the same setup when you return.

Delete - I don't want to use this again

Deleting operate-finder.sh permanently removes your automation. Revoking Accessibility access removes the permission you granted for this project.

The installed agent-desktop CLI remains available because this project does not establish a verified uninstall path.

  • Return to Finder.
  • Search your Mac for operate-finder.sh.
  • Delete the matching operate-finder.sh file.
  • Return to each folder where you saved Finder snapshots during this project.
  • Delete every saved snapshot file from this project.
  • Return to each folder where you saved screenshots during this project.
  • Delete every saved screenshot from this project.
  • Clear those deleted files from the Trash.
  • Return to the Terminal window from this project.
  • Open the macOS permission guide by running this command:
agent-desktop permissions --guide

What Does This Command Do?

The permission guide points you back to the macOS permission control used earlier. This gives you a direct route to the access granted for this project.

  • Follow the guide until the relevant permission control is visible.
  • Disable access for Terminal or the process running agent-desktop.
  • Confirm that the permission is no longer enabled.

The automation script is gone. Saved captures are gone. macOS no longer grants the project process Accessibility access.

Nice Work!

Nice Work!

You did it! Your operate-finder.sh script now uses agent-desktop to operate Finder through its accessibility tree from one verified command.

You've learned how to:

  • Capture an interactive snapshot of Finder. You used each actionable element's full ref to target the interface without matching pixels.
  • Operate Finder without moving your physical pointer. You proved that refs expire by deliberately reusing an old ref and observing STALE_REF.
  • Build an executable automation script that takes a fresh snapshot before every action. The script finishes with a final state check that confirms Finder reached the expected result.
  • Extended your desktop automation skills through an optional Secret Mission challenge.

Ready to quiz yourself?