Control Your Mac Remotely via Telegram
Build an AI agent that controls your Mac via Telegram using Gemini and Playwright.
Introduction
30 Second Summary
You're away from your desk and remember you need to move a file, open an app, or fill out a form on your computer. What if you could just send a chat message and your Mac would handle it?
In this project, you will build a personal AI agent that runs on your Mac and listens to your Telegram messages. You write what you want done in plain language, and the agent executes it on your computer and sends you a screenshot as proof.
What You'll Build
A Telegram bot powered by Google Gemini that translates your natural language messages into Mac system commands and browser automation actions.
By the end of this project, you'll have:
- A working Telegram bot connected to Gemini AI that parses your natural language messages into structured actions.
- A Mac automation layer that moves files, launches apps, and takes screenshots on your behalf.
- A browser automation module using Playwright that navigates websites, fills input fields, and clicks buttons.
- Secret Mission: A macro system that records multi-step workflows and replays them with a single command.
Are there any prerequisites?
No prior experience with Telegram bots, AI APIs, or browser automation is needed. This guide walks you through setting up every tool from scratch, including Python, your API keys, and Playwright.
Set Up Your Environment and API Keys
Before you can build your AI agent, you need a working Python environment and two API keys: one for Telegram (so your bot can receive messages) and one for Gemini (so your agent can understand natural language).
In this step, you will create your project folder, install all dependencies, and get both API keys verified and ready to go.
In this step, get ready to:
- Create a Python project with a virtual environment and install all dependencies.
- Create a Telegram bot and retrieve your user ID.
- Get a Gemini API key and verify it works.
Create a Python virtual environment and install dependencies
A virtual environment keeps your project's packages isolated from the rest of your system. This way, the libraries you install here won't conflict with anything else on your Mac.
- Press Cmd+Space to open Spotlight Search.
- Type Terminal and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir mac-remote-agent
cd mac-remote-agent- Confirm the folder exists by running:
pwdYou should see /Users/yourname/Desktop/mac-remote-agent in the output.
- Create and activate a virtual environment by running these commands:
python3 -m venv venv
source venv/bin/activateWhat does this do?
The first command creates a folder called venv inside your project. It contains a private copy of Python and a place to install packages.
The second command activates the environment. You will see (venv) appear at the start of your terminal prompt, confirming it is active.
You should see (venv) at the beginning of your terminal prompt.
Don't see (venv) in your prompt?
Make sure you ran source venv/bin/activate (not just python3 -m venv venv). The first command creates the environment, the second activates it.
If you see a "command not found" error for python3, confirm Python is installed by running python3 --version.
help me activate my Python virtual environment
- Install the three project dependencies by running:
pip install python-telegram-bot==22.7 google-genai==2.8.0 playwright==1.60.0What are these packages?
- python-telegram-bot connects your Python code to the Telegram Bot API so you can send and receive messages.
- google-genai is Google's official SDK for calling the Gemini API from Python.
- playwright automates a real browser (Chromium) so your agent can navigate websites, fill forms, and click buttons.
You should see Successfully installed followed by a list of package names near the end of the output.
Seeing an error during install?
Make sure your virtual environment is active (you see (venv) in the prompt). If you run pip install without activation, it may try to install globally and fail due to permissions.
If you see a network error, check your internet connection and try again.
help me install my Python dependencies
- Install the Playwright Chromium browser by running:
python -m playwright install chromiumWhy install a browser separately?
Playwright automates a real browser to interact with websites. The pip install command only installs the Python library. This command downloads the actual Chromium browser binary that Playwright will control.
You should see a download progress bar, followed by a confirmation that Chromium was installed.
Download failing?
This command downloads approximately 150 MB. Make sure you have a stable internet connection. If it times out, simply run the command again.
help me install Playwright Chromium
Create your Telegram bot and find your user ID
Your agent needs two pieces of information from Telegram: a bot token (so it can send and receive messages as a bot) and your personal user ID (so it only responds to you).
- Open Telegram on your phone or desktop.
- Search for @BotFather and open a chat with it.
- Send the message:
/newbot- BotFather will ask for a name for your bot. Type a display name (for example, Mac Remote Agent).
- BotFather will ask for a username. This must end in bot (for example, mac_remote_agent_bot). The username cannot be changed later.
What are the username rules?
Telegram bot usernames must be 5-32 characters long. They can only include Latin letters, numbers, and underscores. They must end with bot. If the username is taken, try adding your name or numbers.
- After you confirm the username, BotFather replies with a message containing your bot token. It looks like 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11.
- Copy the token and save it somewhere safe (a notes app or text file). You will need it in the next step.
Keep your bot token secret
Anyone who has your token can control your bot. Do not share it publicly or commit it to version control.
Now retrieve your personal Telegram user ID so the bot can verify that messages come from you.
- Search for @userinfobot in Telegram and open a chat with it.
- Send any message (for example, hi).
- The bot replies with your user information, including a numeric Id field. Copy this number and save it alongside your bot token.
Why do you need your user ID?
Telegram bots are publicly discoverable. Anyone can find your bot and send it messages. By checking the user ID of every incoming message, your agent will ignore everyone except you.
Get your Gemini API key and verify it works
The Gemini API is what gives your agent the ability to understand natural language. It is free to use on the free tier with no credit card required.
- Open your browser and go to ai.google.dev.
- Click Sign in to Google AI Studio and sign in with your Google account.
- Click Get API key (or navigate to the API Keys page from the left sidebar).
- Click Create API key.
- Copy the key that appears. Save it in the same place as your Telegram bot token.
Keep your API key secret
Your Gemini API key grants access to your account. Do not commit it to Git or share it publicly.
Now verify the key actually works by running a quick test script.
- In your terminal (still inside the mac-remote-agent folder with the venv active), create a test file by running:
touch test_gemini.py- Open test_gemini.py in any text editor and paste the following code:
from google import genai
# Initialize the Gemini client with your API key
client = genai.Client(api_key="your-gemini-api-key-here")
# Send a test prompt to verify the connection works
response = client.models.generate_content(
model="gemini-2.5-flash",
contents="Sag Hallo in einem Satz.",
)
print(response.text)What does this script do?
- from google import genai imports the official Google GenAI SDK.
- genai.Client(api_key=...) creates an authenticated client using your key.
- client.models.generate_content(...) sends a prompt to the gemini-2.5-flash model and returns the AI's response.
- print(response.text) prints the model's reply so you can confirm everything is connected.
- Replace your-gemini-api-key-here with the actual API key you just copied from Google AI Studio.
- Save the file.
- Run the test by executing:
python test_gemini.pyYou should see a short German greeting printed in your terminal (something like "Hallo, wie geht es dir?").
Seeing an error instead of a greeting?
If you see API key not valid, double-check that you pasted the full key with no extra spaces.
If you see ModuleNotFoundError: No module named 'google', make sure your virtual environment is active (you see (venv) in the prompt).
help me fix my Gemini API test script
Once you see a response, your Gemini API key is working. You can delete the test file if you like, since you won't need it going forward.
Your environment is fully set up. Next up, you will build the actual Telegram bot that listens for your messages and responds.
Build the Telegram Bot with Auth
Your environment is ready with all dependencies installed, your bot token saved, and your Gemini API key verified. Now it's time to bring your bot to life.
In this step, you'll create the actual Python file that connects to Telegram and listens for your messages. Security is the priority here. Since Telegram bots are publicly discoverable, your bot will only respond to your specific user ID.
In this step, get ready to:
- Create bot.py with authentication that locks down the bot to your user ID.
- Add command handlers and verify the bot responds in Telegram.
- Add a screenshot function that captures your Mac screen and sends it back as proof.
Create bot.py with the auth foundation
Your bot needs a skeleton that imports the Telegram library, stores your credentials, and checks every incoming message against your user ID before responding.
- Create a new file called bot.py in your project folder and add the following code:
import asyncio
import json
import subprocess
import os
from telegram import Update
from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes
# Your bot token from @BotFather
TELEGRAM_TOKEN = "your-telegram-bot-token-here"
# Your personal Telegram user ID from @userinfobot
ALLOWED_USER_ID = 123456789 # Replace with your Telegram user ID
def is_authorized(update: Update) -> bool:
# Only allow messages from your account
return update.effective_user.id == ALLOWED_USER_ID
def main():
app = Application.builder().token(TELEGRAM_TOKEN).build()
print("Bot laeuft... Druecke Ctrl+C zum Stoppen.")
app.run_polling()
if __name__ == "__main__":
main()What does this code do?
This sets up the skeleton of your Telegram bot with a security gate.
- TELEGRAM_TOKEN is the token you received from @BotFather in Step 1.
- ALLOWED_USER_ID is your personal numeric Telegram ID. Anyone else who messages the bot gets silently ignored.
- is_authorized() compares the sender's ID to your allowed ID. Every handler will call this first.
- Application.builder() creates the bot connection using the python-telegram-bot library's async architecture.
- Replace "your-telegram-bot-token-here" with the bot token you saved from @BotFather.
- Replace 123456789 with the numeric user ID you received from @userinfobot.
Finding your Telegram user ID
If you haven't found your user ID yet, open Telegram and search for @userinfobot. Start a chat with it and it will reply with your numeric user ID.
- Save bot.py.
You won't see a change yet. The bot doesn't have any handlers to respond to messages. The next chunk adds those.
Add command handlers and run the bot
A Telegram bot needs handler functions that tell it what to do when it receives a message. You'll add a /start command (the greeting) and a text handler that echoes messages back.
- In bot.py, add the following two functions above the main() function (below is_authorized):
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
await update.message.reply_text(
"Hey! Ich bin dein Mac Agent. Schreib mir was ich tun soll."
)
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text(user_message)What does this code do?
- start() handles the /start command. It checks authorization first, then sends a German greeting.
- handle_message() catches all regular text messages. It checks authorization, then echoes back whatever the user sent.
- Both functions are async because python-telegram-bot uses Python's asyncio under the hood.
Now you need to register these handlers with the application so it knows when to call them.
- In bot.py, find your main() function. Find this line:
app = Application.builder().token(TELEGRAM_TOKEN).build()- Add these two lines directly below it (above the print line):
app.add_handler(CommandHandler("start", start))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))What do these lines do?
- CommandHandler("start", start) routes /start messages to your start() function.
- MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message) routes all regular text (that isn't a slash command) to handle_message().
- Save bot.py.
- Run the bot by executing this command in your terminal:
python bot.pyYou should see Bot laeuft... Druecke Ctrl+C zum Stoppen. printed in the terminal.
- Open Telegram on your phone or desktop.
- Find your bot by searching for the username you gave it during @BotFather setup.
- Send /start to the bot.
- You should see the reply: "Hey! Ich bin dein Mac Agent. Schreib mir was ich tun soll."
- Send any text message (like "hallo") and the bot should echo it back.
Bot not responding?
- Check that TELEGRAM_TOKEN is correct. Even one missing character will cause a silent failure.
- Make sure ALLOWED_USER_ID matches your actual numeric Telegram ID (not your username). The bot silently ignores unauthorized users.
- Confirm the terminal still shows the bot running. If you see an error, check for typos in your imports.
Help me debug why my Telegram bot isn't responding to messages.
- Stop the bot by pressing Ctrl+C in the terminal. You'll restart it after the next substep.
Add the screenshot function
The whole point of this agent is that it can show you what's happening on your Mac. You'll add a function that captures your screen using the macOS screencapture command, then modify the message handler to send that screenshot back as a photo.
- In bot.py, add this function above is_authorized() (below the constants):
def take_screenshot():
# Capture the screen silently and save to temp file
path = "/tmp/mac_agent_screenshot.png"
subprocess.run(["screencapture", "-x", path])
return pathWhat does this code do?
- screencapture -x is a macOS command that takes a silent screenshot (no shutter sound). The -x flag suppresses the sound.
- The screenshot is saved to /tmp/mac_agent_screenshot.png and the function returns that path so other code can find the file.
Now update handle_message() to take a screenshot and send it after echoing the message.
- Find your handle_message() function. It currently looks like this:
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text(user_message)- Replace the entire handle_message() function with this updated version that includes screenshot capture:
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text(user_message)
# Take a screenshot and send it back as proof
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Screenshot!")What changed?
After echoing the message, the handler now calls take_screenshot() to capture the screen. It then opens the screenshot file in binary read mode and sends it as a photo using reply_photo().
The os.path.exists() check prevents a crash if the screenshot fails for any reason.
- Save bot.py.
- Run the bot again:
python bot.py- In Telegram, send any text message to your bot.
- You should see the bot echo your message, followed by a screenshot of your Mac screen.
macOS might ask for screen recording permission
The first time screencapture runs from a terminal application, macOS may ask you to grant Screen Recording permission. Go to System Settings > Privacy & Security > Screen Recording and enable it for your Terminal app.
Screenshot not arriving in Telegram?
- Check that your Terminal app has Screen Recording permission in System Settings.
- Verify that /tmp/mac_agent_screenshot.png exists after running the bot by checking in a second terminal window.
- If you see a permissions error in the terminal, restart your Terminal app after granting Screen Recording access.
Help me fix screenshot sending in my Telegram bot.
- Stop the bot with Ctrl+C when you're done testing.
✔️ Awesome, I've got everything!
Great. Make sure you've saved bot.py with all the changes.
ⓧ I'd like to double check the full code
Here is the complete bot.py at this point:
import asyncio
import json
import subprocess
import os
from telegram import Update
from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes
TELEGRAM_TOKEN = "your-telegram-bot-token-here"
ALLOWED_USER_ID = 123456789 # Replace with your Telegram user ID
def take_screenshot():
path = "/tmp/mac_agent_screenshot.png"
subprocess.run(["screencapture", "-x", path])
return path
def is_authorized(update: Update) -> bool:
return update.effective_user.id == ALLOWED_USER_ID
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
await update.message.reply_text(
"Hey! Ich bin dein Mac Agent. Schreib mir was ich tun soll."
)
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text(user_message)
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Screenshot!")
def main():
app = Application.builder().token(TELEGRAM_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))
print("Bot laeuft... Druecke Ctrl+C zum Stoppen.")
app.run_polling()
if __name__ == "__main__":
main()Your bot is alive and responding to your commands only. Next up, you'll connect Gemini as the AI brain so the bot can understand natural language commands instead of just echoing text.
Add the AI Brain with Gemini
Your bot can receive messages and send screenshots. Now it needs a brain that understands what you actually want it to do.
In this step, you will connect Google Gemini as the AI layer that parses your natural language messages into structured actions the bot can execute.
In this step, get ready to:
- Create a brain module that sends your messages to Gemini and gets back structured JSON actions.
- Implement a system prompt that teaches Gemini how to parse commands into action types.
- Connect the brain to your Telegram bot so messages flow through Gemini before any action runs.
Set up the Gemini client
The google-genai SDK provides a Client object that handles all communication with the Gemini API. You will initialize it with your API key at module level so every function in the file can use it.
- Create a new file called brain.py in your project folder (the same folder where bot.py lives).
- Add the imports and client initialization by pasting the following code:
import json
from google import genai
# Your Gemini API key from Google AI Studio
GEMINI_API_KEY = "your-gemini-api-key-here"
# Initialize the Gemini client
client = genai.Client(api_key=GEMINI_API_KEY)What does this code do?
The genai.Client object is the single entry point for all Gemini API calls. By passing your API key here, every request the client makes is automatically authenticated.
The json import is for parsing the structured responses Gemini will send back later.
- Replace your-gemini-api-key-here with the actual API key you saved from Google AI Studio in Step 1.
- Save brain.py.
Not sure where your API key is?
Check the Google AI Studio page where you generated it in Step 1. If you lost it, you can create a new one at ai.google.dev.
help me find my Gemini API key
Define the system prompt
A system prompt tells the AI model how to behave. Your system prompt will instruct Gemini to always respond with structured JSON instead of free-form text.
- In brain.py, add the system prompt below the client initialization by pasting this code:
SYSTEM_PROMPT = """Du bist ein Mac-Steuerungs-Agent. Der Nutzer schickt dir Befehle in natuerlicher Sprache.
Deine Aufgabe: Parse den Befehl in eine strukturierte JSON-Aktion.
Moegliche Aktionen:
1. {"type": "move_file", "source": "/pfad/zur/datei", "destination": "/ziel/pfad/"}
2. {"type": "open_app", "app_name": "Safari"}
3. {"type": "browse_website", "url": "https://example.com", "fill_selector": "CSS-Selektor oder null", "fill_text": "Text zum eingeben oder null", "click_selector": "CSS-Selektor zum klicken oder null"}
4. {"type": "screenshot"}
Regeln:
- Antworte NUR mit dem JSON-Objekt, kein anderer Text.
- Wenn der Nutzer eine Datei verschieben will, nutze den vollen Pfad. ~ bedeutet /Users/USERNAME.
- Wenn der Nutzer eine Website besuchen und etwas eingeben will, nutze browse_website.
- Wenn du den Befehl nicht verstehst, antworte mit: {"type": "unknown"}
"""What does this system prompt do?
This prompt defines four action types the agent can perform: move_file, open_app, browse_website, and screenshot. Each action type has a specific JSON structure Gemini must follow.
The rules section ensures Gemini only returns JSON (no extra text) and falls back to {"type": "unknown"} when it cannot understand a command. This makes the response predictable and parseable every time.
- Save brain.py.
Why is the prompt in German?
The prompt is written in German because the agent is designed to understand German commands. Gemini handles multilingual instructions natively. You can also send it English commands and it will still parse them correctly.
Implement the think function
The think() function is the core of the brain. It sends the user message to Gemini, receives the JSON response, handles edge cases (like markdown code fences or unparseable text), and returns a clean action dictionary.
- In brain.py, add the think() function below the SYSTEM_PROMPT by pasting this code:
def think(user_message: str) -> dict | None:
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=f"{SYSTEM_PROMPT}\n\nNutzer-Befehl: {user_message}",
)
text = response.text.strip()
# Remove markdown code fences if present
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
try:
action = json.loads(text)
except json.JSONDecodeError:
return None
if action.get("type") == "unknown":
return None
return actionWhat does this function do?
The function calls client.models.generate_content() with the gemini-2.5-flash model. It combines the system prompt with the user's message so Gemini has full context on how to respond.
Gemini sometimes wraps JSON in markdown code fences (```json ... ```). The stripping logic removes these so json.loads() can parse the raw JSON cleanly.
If parsing fails or Gemini returns "unknown", the function returns None so the bot knows the command was not understood.
- Save brain.py.
Seeing an error about response.text?
Make sure you replaced your-gemini-api-key-here with your actual Gemini API key. An invalid key causes the API call to fail before returning any text.
help me debug my Gemini API connection in brain.py
✔️ Awesome, I've got everything!
Great. Double check you have saved brain.py with all three sections: imports and client, the system prompt, and the think() function.
ⓧ I'd like to double check the full code
import json
from google import genai
GEMINI_API_KEY = "your-gemini-api-key-here"
client = genai.Client(api_key=GEMINI_API_KEY)
SYSTEM_PROMPT = """Du bist ein Mac-Steuerungs-Agent. Der Nutzer schickt dir Befehle in natuerlicher Sprache.
Deine Aufgabe: Parse den Befehl in eine strukturierte JSON-Aktion.
Moegliche Aktionen:
1. {"type": "move_file", "source": "/pfad/zur/datei", "destination": "/ziel/pfad/"}
2. {"type": "open_app", "app_name": "Safari"}
3. {"type": "browse_website", "url": "https://example.com", "fill_selector": "CSS-Selektor oder null", "fill_text": "Text zum eingeben oder null", "click_selector": "CSS-Selektor zum klicken oder null"}
4. {"type": "screenshot"}
Regeln:
- Antworte NUR mit dem JSON-Objekt, kein anderer Text.
- Wenn der Nutzer eine Datei verschieben will, nutze den vollen Pfad. ~ bedeutet /Users/USERNAME.
- Wenn der Nutzer eine Website besuchen und etwas eingeben will, nutze browse_website.
- Wenn du den Befehl nicht verstehst, antworte mit: {"type": "unknown"}
"""
def think(user_message: str) -> dict | None:
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=f"{SYSTEM_PROMPT}\n\nNutzer-Befehl: {user_message}",
)
text = response.text.strip()
# Remove markdown code fences if present
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
try:
action = json.loads(text)
except json.JSONDecodeError:
return None
if action.get("type") == "unknown":
return None
return actionConnect the brain to your bot
Right now bot.py just echoes messages back. You need to route incoming messages through the think() function so Gemini decides what action to take.
- Switch back to bot.py in your editor.
- Find this line near the top of the file:
from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes- Add this import directly below it:
from brain import thinkWhat does this import do?
This pulls in the think() function from your new brain.py module. Now bot.py can send any user message to Gemini for parsing.
Next, you will replace the handle_message function so it uses the brain instead of echoing.
- Find the existing handle_message function in bot.py:
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text(user_message)
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Screenshot!")- Replace the entire function with this updated version:
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text("Denke nach...")
# Send the message to Gemini for parsing
action = think(user_message)
if action is None:
await update.message.reply_text(
"Ich habe nicht verstanden was du willst. Versuch es anders zu formulieren."
)
return
# Confirm what action the agent is about to take
await update.message.reply_text(f"Aktion: {action.get('type')} - Wird ausgefuehrt...")
# Take a screenshot and send it as confirmation
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Erledigt!")What changed in handle_message?
Instead of echoing the message back, the bot now sends "Denke nach..." (thinking) as immediate feedback. Then it passes the message to think() which calls Gemini.
If Gemini cannot parse the command (returns None), the bot tells the user to try again. Otherwise, it confirms the action type before executing and sends a screenshot as proof.
- Save bot.py.
- Run the bot by executing this command in your terminal:
python bot.py- Open Telegram and send your bot a message like "mach einen Screenshot". You should see "Denke nach..." followed by "Aktion: screenshot - Wird ausgefuehrt..." and then a screenshot photo.
Bot not responding or crashing?
Check that your brain.py file is in the same folder as bot.py. Python can only import modules from the same directory.
If you see an import error mentioning actions, that is expected for now. You will create the actions.py module in the next step.
help me fix my bot.py import or Gemini connection error
✔️ Awesome, I've got everything!
Great. Make sure you have saved both bot.py and brain.py.
ⓧ I'd like to double check the full code
import asyncio
import json
import subprocess
import os
from telegram import Update
from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes
from brain import think
TELEGRAM_TOKEN = "your-telegram-bot-token-here"
ALLOWED_USER_ID = 123456789 # Replace with your Telegram user ID
def take_screenshot():
path = "/tmp/mac_agent_screenshot.png"
subprocess.run(["screencapture", "-x", path])
return path
def is_authorized(update: Update) -> bool:
return update.effective_user.id == ALLOWED_USER_ID
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
await update.message.reply_text(
"Hey! Ich bin dein Mac Agent. Schreib mir was ich tun soll."
)
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text("Denke nach...")
action = think(user_message)
if action is None:
await update.message.reply_text(
"Ich habe nicht verstanden was du willst. Versuch es anders zu formulieren."
)
return
await update.message.reply_text(f"Aktion: {action.get('type')} - Wird ausgefuehrt...")
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Erledigt!")
def main():
app = Application.builder().token(TELEGRAM_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))
print("Bot laeuft... Druecke Ctrl+C zum Stoppen.")
app.run_polling()
if __name__ == "__main__":
main()Your bot now has a brain. It receives your messages, sends them to Gemini, and gets back structured actions it can act on. Next up, you will build the action layer that actually moves files, opens apps, and automates your browser.
Implement Mac Actions and Browser Automation
Your agent can think, but it cannot act yet. Right now, Gemini parses your messages into action dictionaries, but nothing actually happens on your Mac.
In this step, you will give the agent hands. You will build an actions module for file and app operations, a browser automation module using Playwright, and then wire everything together so the bot executes real actions and sends you screenshot proof.
In this step, get ready to:
- Create an actions module for file operations and app launching.
- Build a browser automation module with Playwright.
- Wire the bot to dispatch actions and send confirmation screenshots.
Create the actions module
The actions module translates action dictionaries from Gemini into real macOS operations. It uses Python's built-in subprocess and shutil libraries to move files and launch apps.
- Create a new file called actions.py in your project folder.
- Add the following code to actions.py:
import subprocess
import shutil
import os
def execute_action(action: dict) -> str:
action_type = action.get("type")
if action_type == "move_file":
return move_file(action.get("source", ""), action.get("destination", ""))
elif action_type == "open_app":
return open_app(action.get("app_name", ""))
elif action_type == "screenshot":
return "Screenshot wird gemacht..."
else:
return f"Unbekannte Aktion: {action_type}"What does this code do?
The execute_action() function is a dispatcher. It reads the type field from the action dictionary and routes to the correct handler function.
For move_file actions, it extracts source and destination paths. For open_app actions, it extracts the app_name. Unknown action types get a fallback error message.
- Add the move_file() function below execute_action():
def move_file(source: str, destination: str) -> str:
source = os.path.expanduser(source)
destination = os.path.expanduser(destination)
if not os.path.exists(source):
return f"Fehler: Datei '{source}' existiert nicht."
try:
shutil.move(source, destination)
return f"Datei verschoben: {source} -> {destination}"
except Exception as e:
return f"Fehler beim Verschieben: {e}"What does this code do?
The os.path.expanduser() call converts ~ shortcuts into full paths (like /Users/yourname). This is important because Gemini's responses often include ~ in file paths.
The function checks if the source file exists before attempting the move. If anything goes wrong, it returns a descriptive error message instead of crashing.
- Add the open_app() function below move_file():
def open_app(app_name: str) -> str:
try:
subprocess.run(["open", "-a", app_name], check=True)
return f"App '{app_name}' gestartet."
except subprocess.CalledProcessError:
return f"Fehler: App '{app_name}' konnte nicht gestartet werden."What does open -a do?
The open -a command is a macOS-native way to launch applications by name. You can open Safari, Finder, Notes, or any app installed on your Mac.
Setting check=True makes subprocess.run() raise a CalledProcessError if the app name is invalid, which the except block catches gracefully.
- Save actions.py.
- Confirm the file exists by running:
ls actions.pyYou should see actions.py listed in the output.
File not found?
Make sure you are in your project folder (the same folder where bot.py and brain.py live). If you opened a new terminal, cd back into your project directory.
help me find my project folder
Build the browser automation module
The browser module uses Playwright's async API to control a real Chromium browser. It can navigate to websites, fill in form fields, click buttons, and take a screenshot of the result.
- Create a new file called browser.py in your project folder.
- Add the following code to browser.py:
import asyncio
from playwright.async_api import async_playwright
async def browse(
url: str,
fill_selector: str | None = None,
fill_text: str | None = None,
click_selector: str | None = None,
) -> str:
screenshot_path = "/tmp/mac_agent_browser.png"
async with async_playwright() as p:
browser = await p.chromium.launch(headless=False)
page = await browser.new_page()
await page.goto(url, wait_until="domcontentloaded")
await page.wait_for_timeout(2000)
if fill_selector and fill_text:
await page.fill(fill_selector, fill_text)
await page.wait_for_timeout(500)
if click_selector:
await page.click(click_selector)
await page.wait_for_timeout(2000)
await page.screenshot(path=screenshot_path, full_page=False)
await browser.close()
return screenshot_pathWhat does this code do?
The browse() function is async because Playwright's Python API uses asyncio. It launches a visible Chromium window (headless=False) so you can watch the agent work.
The function navigates to the URL, waits for the page to load, optionally fills an input field using a CSS selector, optionally clicks a button, then takes a screenshot. The screenshot is saved to /tmp/mac_agent_browser.png and the path is returned for sending back to Telegram.
The wait_for_timeout() calls give pages time to render JavaScript content before interacting or capturing a screenshot.
Why Playwright over AppleScript for browser control?
AppleScript browser control only works with Safari and is fragile. Playwright gives you reliable, cross-browser CSS-selector-based automation that can fill forms, click buttons, and take page screenshots natively.
- Save browser.py.
- Confirm both new files exist by running:
ls actions.py browser.pyYou should see both filenames listed.
Seeing 'No such file or directory'?
Double check you saved both files in the same folder as bot.py. If you used a different directory, move the files or re-create them in the correct location.
help me check where my files are saved
Wire the bot to execute actions
Now you will connect the action modules to the bot. The updated handle_message() function will dispatch actions to the correct module and send screenshot proof back to Telegram.
- Open bot.py in your editor.
- Find the import section at the top of the file. Locate this line:
from brain import think- Add these two imports directly below it:
from actions import execute_action
from browser import browse- Find the line that says ALLOWED_USER_ID = 123456789 and add the following lines directly below it:
MACROS_FILE = "macros.json"
recording = {}
def load_macros():
if os.path.exists(MACROS_FILE):
with open(MACROS_FILE, "r") as f:
return json.load(f)
return {}
def save_macros(macros):
with open(MACROS_FILE, "w") as f:
json.dump(macros, f, indent=2)What is this macro infrastructure?
This sets up the foundation for a macro recording system. The recording dictionary will track which users are currently recording action sequences. The load_macros() and save_macros() functions persist macros to a JSON file so they survive bot restarts.
You will build the full macro commands (record, stop, play, list) later in the Secret Mission.
- Now replace the entire handle_message() function with this updated version:
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text("Denke nach...")
action = think(user_message)
if action is None:
await update.message.reply_text(
"Ich habe nicht verstanden was du willst. Versuch es anders zu formulieren."
)
return
# Record macro if active
user_id = update.effective_user.id
if user_id in recording:
recording[user_id]["actions"].append(action)
await update.message.reply_text(f"Aktion: {action.get('type')} - Wird ausgefuehrt...")
if action["type"] == "browse_website":
screenshot_path = await browse(
url=action.get("url", ""),
fill_selector=action.get("fill_selector"),
fill_text=action.get("fill_text"),
click_selector=action.get("click_selector"),
)
else:
result = execute_action(action)
await update.message.reply_text(result)
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Erledigt!")What changed in handle_message?
The key change is the dispatch logic. For browse_website actions, the bot calls the async browse() function from your browser module. For all other action types (move_file, open_app, screenshot), it calls execute_action() and takes a Mac screenshot.
The macro recording hook (if user_id in recording) silently appends each action to the recording list when active. This does nothing yet until the macro commands are built.
- Add the macro command handlers below handle_message() (above the main() function):
async def save_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
if not context.args:
await update.message.reply_text("Nutzung: /save macroname")
return
name = context.args[0]
user_id = update.effective_user.id
recording[user_id] = {"name": name, "actions": []}
await update.message.reply_text(f"Recording gestartet: {name}")
async def stop_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_id = update.effective_user.id
if user_id not in recording:
await update.message.reply_text("Kein Recording aktiv.")
return
macro_data = recording.pop(user_id)
macros = load_macros()
macros[macro_data["name"]] = macro_data["actions"]
save_macros(macros)
await update.message.reply_text(
f"Macro '{macro_data['name']}' gespeichert mit {len(macro_data['actions'])} Aktionen."
)
async def play_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
if not context.args:
await update.message.reply_text("Nutzung: /play macroname")
return
name = context.args[0]
macros = load_macros()
if name not in macros:
await update.message.reply_text(f"Macro '{name}' nicht gefunden.")
return
actions = macros[name]
await update.message.reply_text(f"Spiele Macro '{name}' ab ({len(actions)} Aktionen)...")
for action in actions:
if action["type"] == "browse_website":
await browse(
url=action.get("url", ""),
fill_selector=action.get("fill_selector"),
fill_text=action.get("fill_text"),
click_selector=action.get("click_selector"),
)
else:
execute_action(action)
screenshot_path = take_screenshot()
if os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption=f"Macro '{name}' fertig!")
async def list_macros(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
macros = load_macros()
if not macros:
await update.message.reply_text("Keine Macros gespeichert.")
return
lines = [f"- {name} ({len(actions)} Aktionen)" for name, actions in macros.items()]
await update.message.reply_text("Gespeicherte Macros:\n" + "\n".join(lines))What do these macro commands do?
These four handlers power the macro system. /save starts recording actions into a named macro. /stop ends recording and saves to macros.json. /play replays a saved macro's actions in sequence. /macros lists all stored macros.
- Now replace the main() function with this updated version that registers all handlers:
def main():
app = Application.builder().token(TELEGRAM_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(CommandHandler("save", save_macro))
app.add_handler(CommandHandler("stop", stop_macro))
app.add_handler(CommandHandler("play", play_macro))
app.add_handler(CommandHandler("macros", list_macros))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))
print("Bot laeuft... Druecke Ctrl+C zum Stoppen.")
app.run_polling()
if __name__ == "__main__":
main()- Save bot.py.
- Run the bot by executing:
python bot.pyYou should see Bot laeuft... Druecke Ctrl+C zum Stoppen. in your terminal.
- Open Telegram and send your bot a message like oeffne Safari.
- Watch Safari open on your Mac, then check Telegram for the screenshot confirmation.
Bot not responding or crashing?
Check your terminal for error messages. A common issue is ModuleNotFoundError if your virtual environment is not activated. Run source venv/bin/activate (or your venv activation command) and try again.
If you see an import error for actions or browser, make sure all four files (bot.py, brain.py, actions.py, browser.py) are in the same folder.
help me debug my bot startup error
✔️ Awesome, I've got everything!
Great. Double check you have saved all files before moving on.
ⓧ I'd like to double check the full code
Here is the complete code for each file at this point:
import subprocess
import shutil
import os
def execute_action(action: dict) -> str:
action_type = action.get("type")
if action_type == "move_file":
return move_file(action.get("source", ""), action.get("destination", ""))
elif action_type == "open_app":
return open_app(action.get("app_name", ""))
elif action_type == "screenshot":
return "Screenshot wird gemacht..."
else:
return f"Unbekannte Aktion: {action_type}"
def move_file(source: str, destination: str) -> str:
source = os.path.expanduser(source)
destination = os.path.expanduser(destination)
if not os.path.exists(source):
return f"Fehler: Datei '{source}' existiert nicht."
try:
shutil.move(source, destination)
return f"Datei verschoben: {source} -> {destination}"
except Exception as e:
return f"Fehler beim Verschieben: {e}"
def open_app(app_name: str) -> str:
try:
subprocess.run(["open", "-a", app_name], check=True)
return f"App '{app_name}' gestartet."
except subprocess.CalledProcessError:
return f"Fehler: App '{app_name}' konnte nicht gestartet werden."import asyncio
from playwright.async_api import async_playwright
async def browse(
url: str,
fill_selector: str | None = None,
fill_text: str | None = None,
click_selector: str | None = None,
) -> str:
screenshot_path = "/tmp/mac_agent_browser.png"
async with async_playwright() as p:
browser = await p.chromium.launch(headless=False)
page = await browser.new_page()
await page.goto(url, wait_until="domcontentloaded")
await page.wait_for_timeout(2000)
if fill_selector and fill_text:
await page.fill(fill_selector, fill_text)
await page.wait_for_timeout(500)
if click_selector:
await page.click(click_selector)
await page.wait_for_timeout(2000)
await page.screenshot(path=screenshot_path, full_page=False)
await browser.close()
return screenshot_pathimport asyncio
import json
import subprocess
import os
from telegram import Update
from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes
from brain import think
from actions import execute_action
from browser import browse
TELEGRAM_TOKEN = "your-telegram-bot-token-here"
ALLOWED_USER_ID = 123456789 # Replace with your Telegram user ID
MACROS_FILE = "macros.json"
recording = {}
def load_macros():
if os.path.exists(MACROS_FILE):
with open(MACROS_FILE, "r") as f:
return json.load(f)
return {}
def save_macros(macros):
with open(MACROS_FILE, "w") as f:
json.dump(macros, f, indent=2)
def take_screenshot():
path = "/tmp/mac_agent_screenshot.png"
subprocess.run(["screencapture", "-x", path])
return path
def is_authorized(update: Update) -> bool:
return update.effective_user.id == ALLOWED_USER_ID
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
await update.message.reply_text(
"Hey! Ich bin dein Mac Agent. Schreib mir was ich tun soll."
)
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_message = update.message.text
await update.message.reply_text("Denke nach...")
action = think(user_message)
if action is None:
await update.message.reply_text(
"Ich habe nicht verstanden was du willst. Versuch es anders zu formulieren."
)
return
# Record macro if active
user_id = update.effective_user.id
if user_id in recording:
recording[user_id]["actions"].append(action)
await update.message.reply_text(f"Aktion: {action.get('type')} - Wird ausgefuehrt...")
if action["type"] == "browse_website":
screenshot_path = await browse(
url=action.get("url", ""),
fill_selector=action.get("fill_selector"),
fill_text=action.get("fill_text"),
click_selector=action.get("click_selector"),
)
else:
result = execute_action(action)
await update.message.reply_text(result)
screenshot_path = take_screenshot()
if screenshot_path and os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption="Erledigt!")
async def save_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
if not context.args:
await update.message.reply_text("Nutzung: /save macroname")
return
name = context.args[0]
user_id = update.effective_user.id
recording[user_id] = {"name": name, "actions": []}
await update.message.reply_text(f"Recording gestartet: {name}")
async def stop_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
user_id = update.effective_user.id
if user_id not in recording:
await update.message.reply_text("Kein Recording aktiv.")
return
macro_data = recording.pop(user_id)
macros = load_macros()
macros[macro_data["name"]] = macro_data["actions"]
save_macros(macros)
await update.message.reply_text(
f"Macro '{macro_data['name']}' gespeichert mit {len(macro_data['actions'])} Aktionen."
)
async def play_macro(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
if not context.args:
await update.message.reply_text("Nutzung: /play macroname")
return
name = context.args[0]
macros = load_macros()
if name not in macros:
await update.message.reply_text(f"Macro '{name}' nicht gefunden.")
return
actions = macros[name]
await update.message.reply_text(f"Spiele Macro '{name}' ab ({len(actions)} Aktionen)...")
for action in actions:
if action["type"] == "browse_website":
await browse(
url=action.get("url", ""),
fill_selector=action.get("fill_selector"),
fill_text=action.get("fill_text"),
click_selector=action.get("click_selector"),
)
else:
execute_action(action)
screenshot_path = take_screenshot()
if os.path.exists(screenshot_path):
with open(screenshot_path, "rb") as photo:
await update.message.reply_photo(photo=photo, caption=f"Macro '{name}' fertig!")
async def list_macros(update: Update, context: ContextTypes.DEFAULT_TYPE):
if not is_authorized(update):
return
macros = load_macros()
if not macros:
await update.message.reply_text("Keine Macros gespeichert.")
return
lines = [f"- {name} ({len(actions)} Aktionen)" for name, actions in macros.items()]
await update.message.reply_text("Gespeicherte Macros:\n" + "\n".join(lines))
def main():
app = Application.builder().token(TELEGRAM_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(CommandHandler("save", save_macro))
app.add_handler(CommandHandler("stop", stop_macro))
app.add_handler(CommandHandler("play", play_macro))
app.add_handler(CommandHandler("macros", list_macros))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))
print("Bot laeuft... Druecke Ctrl+C zum Stoppen.")
app.run_polling()
if __name__ == "__main__":
main()import json
from google import genai
GEMINI_API_KEY = "your-gemini-api-key-here"
client = genai.Client(api_key=GEMINI_API_KEY)
SYSTEM_PROMPT = """Du bist ein Mac-Steuerungs-Agent. Der Nutzer schickt dir Befehle in natuerlicher Sprache.
Deine Aufgabe: Parse den Befehl in eine strukturierte JSON-Aktion.
Moegliche Aktionen:
1. {"type": "move_file", "source": "/pfad/zur/datei", "destination": "/ziel/pfad/"}
2. {"type": "open_app", "app_name": "Safari"}
3. {"type": "browse_website", "url": "https://example.com", "fill_selector": "CSS-Selektor oder null", "fill_text": "Text zum eingeben oder null", "click_selector": "CSS-Selektor zum klicken oder null"}
4. {"type": "screenshot"}
Regeln:
- Antworte NUR mit dem JSON-Objekt, kein anderer Text.
- Wenn der Nutzer eine Datei verschieben will, nutze den vollen Pfad. ~ bedeutet /Users/USERNAME.
- Wenn der Nutzer eine Website besuchen und etwas eingeben will, nutze browse_website.
- Wenn du den Befehl nicht verstehst, antworte mit: {"type": "unknown"}
"""
def think(user_message: str) -> dict | None:
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=f"{SYSTEM_PROMPT}\n\nNutzer-Befehl: {user_message}",
)
text = response.text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
try:
action = json.loads(text)
except json.JSONDecodeError:
return None
if action.get("type") == "unknown":
return None
return actionYour agent can now think and act. It moves files, launches apps, controls the browser, and sends you proof. Next up, you will unlock the Secret Mission and teach your agent to remember and replay entire workflows.
Secret mission
Build a Macro Recorder
Your agent can execute single commands. But what about complex workflows that take multiple steps? Build a macro system that records sequences of actions and replays them with a single command.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely on your Mac with no cloud infrastructure. The Gemini API uses the free tier, so there are no ongoing costs to worry about.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Telegram bot (runs locally, free).
- Gemini API key (free tier, no charges).
- Playwright Chromium browser (installed locally, no cost).
- Project folder on your Desktop with bot.py, brain.py, actions.py, browser.py, and macros.json.
Keep everything running
No action needed. Choose this if you are still actively building or want the bot available at all times.
- Leave Terminal open with the bot running. It continues to respond to your Telegram messages.
- Your Gemini API key stays active on the free tier. No charges accumulate.
- The Playwright Chromium browser stays installed on your machine for future browser automation.
Pause - I'll come back to this later
Shut down the running process to free up memory. Your files and credentials stay in place for when you return.
- In the Terminal window running your bot, press Ctrl+C to stop the bot process.
- Your project files remain on your Desktop. Restart anytime by running:
python bot.py- Your Gemini API key and Telegram bot token stay valid. No need to recreate them.
Delete - I don't want to use this again
Remove all project resources and revoke credentials to start fresh.
- In the Terminal running your bot, press Ctrl+C to stop the process.
- Remove the Playwright Chromium browser by running:
playwright uninstall- Delete your project folder by running:
cd ~/Desktop
rm -rf mac-remote-agent- Revoke your Telegram bot token. Open Telegram, message @BotFather, and send /deletebot. Select your bot from the list. This permanently deletes the bot and frees its username.
- Delete your Gemini API key. Open the Google Cloud API Credentials page, find your key, click the Actions icon, and select Delete API key. Confirm the deletion in the modal.
This cannot be undone
Deleting your bot via @BotFather is permanent. You cannot recover the bot or its username after deletion. If you think you might want the bot again, choose Pause instead.
That's a Wrap!
That's a Wrap!
Nice work! You just built a personal AI agent that controls your Mac remotely through Telegram messages.
You've learned how to:
- Built a Telegram bot with Gemini AI that parses natural language commands into structured JSON actions and only responds to your authenticated user ID.
- Implemented Mac automation for file operations, app launching, and screenshot capture using native macOS commands like screencapture and open -a.
- Built browser automation with Playwright that navigates websites, fills form inputs, clicks buttons, and sends you screenshot confirmations.
- Secret Mission: Created a macro system that records multi-step action sequences to macros.json and replays them with a single command.
Ready to quiz yourself?