AI Screen Reader with Gemini

Build a desktop AI assistant that captures your screen and answers questions about it.

Introduction

30 Second Summary

You know that moment when you're staring at an error message, a confusing dashboard, or a wall of code and you just wish someone could look at your screen and tell you what's going on? That's exactly what you're building today.

In this project, you will build a desktop AI assistant that captures your screen with a keyboard shortcut and uses Google Gemini's vision capabilities to analyze what it sees. The AI streams its response into a floating chat window where you can ask follow-up questions.

What You'll Build

A Python system tray application that captures your screen on a global hotkey, sends it to Gemini 2.5 Flash, and streams back a helpful analysis in a floating chat panel.

By the end of this project, you'll have:

  • A PyQt6 system tray application that runs quietly in the background and opens a floating chat panel.
  • Screen capture on a global hotkey using the mss library, with screenshots compressed to JPEG and sent to Gemini's vision API.
  • Streaming responses from Gemini 2.5 Flash displayed in real time, with a text input for follow-up questions about what's on your screen.
  • Secret Mission: add multi-monitor support and let the user select which monitor to capture.

Are there any prerequisites?

This project assumes you have Python programming experience and familiarity with virtual environments using uv. You will also need a free Gemini API key from Google AI Studio. No credit card is required.

Set Up Your Project

Before you can build an AI that sees your screen, you need a clean workspace with all the right tools installed. A dedicated project folder and isolated Python environment keep your dependencies organized and prevent conflicts with other projects on your machine.

In this step, you'll create your project folder, install six Python packages, and confirm your Gemini API key is working. By the end, you'll be ready to start writing application code.

In this step, get ready to:
  • Create a project directory and virtual environment.
  • Install all project dependencies.
  • Verify your Gemini API key with a test script.
Create your project directory and virtual environment

A virtual environment is an isolated copy of Python where you install only the packages your project needs. This prevents version clashes between different projects on the same machine.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-screen-reader
cd ai-screen-reader
  • Confirm the folder was created by running:
ls

The output should be empty because the folder is brand new.

  • Initialize a Python virtual environment by running:
uv venv

You should see output confirming the virtual environment was created, including the path to .venv.

Seeing 'uv: command not found'?

This means uv is not installed or not in your system PATH. Follow the official installation instructions at docs.astral.sh/uv and try again.

Help me install uv on my operating system.

  • Activate the virtual environment by running this command:

macOS

source .venv/bin/activate

Linux users can use the same command.

Windows

.venv\Scripts\activate

Your terminal prompt now shows (.venv) at the beginning. This confirms the virtual environment is active.

Don't see (.venv) in your prompt?

Make sure you ran the activation command for your operating system. macOS and Linux use source while Windows uses a backslash path.

If your terminal still does not show (.venv), confirm the .venv folder exists by running ls -a (macOS) or dir (Windows) in the ai-screen-reader folder.

Help me activate my Python virtual environment.

Install your dependencies

This project uses six Python packages. Each handles a specific job, from screen capture to AI communication to the desktop UI.

  • Install all six packages by running this command:
uv pip install google-genai PyQt6 mss keyboard Pillow python-dotenv

What do these packages do?

  • google-genai is Google's official SDK for the Gemini API. It handles sending prompts and receiving responses.
  • PyQt6 is a framework for building desktop applications with Python. You'll use it for the system tray icon and chat panel.
  • mss is a fast, cross-platform screen capture library.
  • keyboard registers global hotkeys so you can trigger a capture from anywhere.
  • Pillow is a Python image processing library. It converts raw screenshots into compressed JPEG bytes.
  • python-dotenv loads environment variables from a .env file so you can keep secrets out of your code.
  • Look for a Successfully installed message in the terminal output listing all six packages.

Seeing install errors?

Make sure your virtual environment is active. You should see (.venv) in your terminal prompt before running the install command.

If you see errors about building wheels for PyQt6, confirm you are running Python 3.10 or newer by running python --version.

Help me fix my dependency install errors.

Now open VS Code to create the project files you need.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to Desktop, select the ai-screen-reader folder, and click Open.

Your project needs a requirements.txt file to record which packages it depends on. This lets anyone recreate your exact environment later.

  • In the Explorer sidebar on the left, right-click in the empty space below .venv.
  • Click New File.
  • Name it requirements.txt and press Enter.
  • Paste the following into requirements.txt:
google-genai>=1.74.0
PyQt6>=6.7.0
mss>=10.0.0
keyboard>=0.13.5
Pillow>=10.0.0
python-dotenv>=1.0.0

Why pin version numbers?

The >= syntax means "this version or newer." It ensures anyone installing from this file gets a compatible version of each package. Without version pins, a future breaking change could silently break your project.

  • Save requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you've saved requirements.txt. The white dot on the file tab disappears once saved.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file. Make sure yours matches exactly.

google-genai>=1.74.0
PyQt6>=6.7.0
mss>=10.0.0
keyboard>=0.13.5
Pillow>=10.0.0
python-dotenv>=1.0.0
Create your .env file and verify your API key

Your Gemini API key is a secret credential that authenticates your app with Google's servers. Storing it in a .env file keeps it out of your source code so you never accidentally commit it to version control.

  • In the Explorer sidebar, right-click in the empty space and click New File.
  • Name it .env and press Enter.
  • Paste the following into .env, replacing your-api-key-here with the actual Gemini API key from your Google AI Studio dashboard:
GEMINI_API_KEY=your-api-key-here

Keep your API key safe

Never share your API key or commit it to GitHub. If you plan to use Git for this project, add .env to your .gitignore file.

  • Save .env by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Make sure the white dot on the .env tab has disappeared, confirming the file is saved.

ⓧ I'd like to double check the full code

Your .env file should have exactly one line. Replace your-api-key-here with your actual key.

GEMINI_API_KEY=your-api-key-here

Now verify your API key actually works by sending a test prompt to gemini-2.5-flash.

  • In the Explorer sidebar, right-click in the empty space and click New File.
  • Name it test_gemini.py and press Enter.
  • Paste the following test script into test_gemini.py:
# Test script to verify your Gemini API key
from dotenv import load_dotenv
import os
from google import genai

# Load the API key from your .env file
load_dotenv()

# Create a Gemini client using your API key
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

# Send a simple text prompt to Gemini 2.5 Flash
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Say hello in one sentence."
)

# Print Gemini's response
print(response.text)

What does this test script do?

  • load_dotenv() reads the GEMINI_API_KEY value from your .env file into an environment variable.
  • genai.Client() creates a connection to the Gemini API using your key.
  • generate_content() sends a text prompt to the gemini-2.5-flash model and returns the full response.
  • Save test_gemini.py.
  • Go back to the terminal where you activated your virtual environment.
  • Run the test script:
python test_gemini.py

You should see a short greeting from Gemini printed in your terminal. The exact wording varies, but any text response confirms your API key is working.

Seeing an API key error?

If you see API key not valid or a 400/403 error, double check that your .env file has no spaces around the = sign. It should look like GEMINI_API_KEY=AIza... with no quotes and no spaces.

If you see ModuleNotFoundError, your virtual environment may not be active. Check for (.venv) in your terminal prompt and re-activate if needed.

Help me fix my Gemini API connection error.

Your environment is ready and Gemini is responding. Next up, you'll capture your screen on a hotkey and turn those pixels into bytes that Gemini can analyze.

Capture Your Screen on a Hotkey

Your environment is set up and your API key is verified. Now it's time to give your AI assistant the ability to see your screen.

Before Gemini can analyze anything, it needs an image to look at. In this step, you'll use the mss library to capture your screen and compress it into a compact JPEG ready for the API.

In this step, get ready to:
  • Build a screenshot capture function using mss.
  • Convert raw screen data to compressed JPEG format with Pillow.
  • Test the capture on your machine.
Set up main.py with your imports

Your main.py file will hold the entire application by the end of this project. Start by adding all the imports you'll need, plus one line to load your API key from the .env file.

  • Create a new file called main.py inside your ai-screen-reader folder.
  • Add the following imports to main.py:
# Standard library imports
import sys
import os
import io
import base64
import threading
from pathlib import Path

# Third-party libraries for screen capture, hotkeys, and AI
from dotenv import load_dotenv
from PIL import Image
from mss import mss
import keyboard
from google import genai
from google.genai import types

# PyQt6 widgets for the desktop UI
from PyQt6.QtWidgets import (
    QApplication, QWidget, QSystemTrayIcon, QMenu,
    QTextEdit, QLineEdit, QVBoxLayout
)
from PyQt6.QtCore import Qt, pyqtSignal, QObject
from PyQt6.QtGui import QIcon, QPixmap, QAction, QPainter, QColor

# Load environment variables (GEMINI_API_KEY) from .env
load_dotenv()

What does this code do?

  • Standard library imports (sys, os, io, threading) handle system operations, byte buffers, and background threads.
  • load_dotenv() reads your .env file so GEMINI_API_KEY is available as an environment variable throughout the script.
  • mss captures your screen. PIL.Image converts the raw pixels into a compressed image format.
  • keyboard registers global hotkeys that work even when your app is not in focus. google.genai and google.genai.types talk to the Gemini API.
  • The PyQt6 imports are for the system tray app and chat panel you'll build in Step 4.
  • Save main.py.

You won't see any output yet. The next chunk adds the function that puts these imports to work.

Build the screenshot capture function

The mss library captures your screen as raw BGRA pixel data. A single screenshot at full resolution can be several megabytes.

Converting to JPEG shrinks it down to 100-300 KB, which is much faster to send to Gemini.

  • In main.py, add the following function below load_dotenv():
def capture_screenshot(monitor_index: int = 1) -> bytes:
    """Capture a screenshot and return it as JPEG bytes."""
    with mss() as sct:
        # Select the target monitor (index 1 = primary monitor)
        monitor = sct.monitors[monitor_index]
        screenshot = sct.grab(monitor)

        # Convert raw BGRA pixels to RGB, then compress as JPEG
        img = Image.frombytes("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX")
        buffer = io.BytesIO()
        img.save(buffer, format="JPEG", quality=70)
        return buffer.getvalue()

What does this code do?

  • with mss() as sct: creates a screen capture session. sct.monitors[1] targets your primary monitor. Index 0 is a virtual screen that combines all monitors.
  • sct.grab(monitor) captures the raw pixel data from that monitor.
  • Image.frombytes("RGB", ..., "raw", "BGRX") converts the BGRA pixel data into an RGB image. The BGRX decoder tells Pillow to read blue, green, red channels and skip the alpha channel.
  • The image is saved as JPEG with quality 70, then returned as raw bytes. Quality 70 is a good balance between small file size and image clarity.

Why mss over other screenshot libraries?

mss is pure Python, cross-platform, and thread-safe. It handles Hi-DPI displays on Windows automatically.

Other options like pyautogui or PIL.ImageGrab are slower and less reliable for background capture. Since your app will grab screenshots in a background thread while the UI stays responsive, thread safety matters.

  • Save main.py.

You won't see a visible change yet. The next section tests this function to confirm it's working.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py before moving on.

ⓧ I'd like to double check the full code

Here's the complete main.py so far. Compare it to yours and make sure everything matches.

import sys
import os
import io
import base64
import threading
from pathlib import Path

from dotenv import load_dotenv
from PIL import Image
from mss import mss
import keyboard
from google import genai
from google.genai import types

from PyQt6.QtWidgets import (
    QApplication, QWidget, QSystemTrayIcon, QMenu,
    QTextEdit, QLineEdit, QVBoxLayout
)
from PyQt6.QtCore import Qt, pyqtSignal, QObject
from PyQt6.QtGui import QIcon, QPixmap, QAction, QPainter, QColor

load_dotenv()


def capture_screenshot(monitor_index: int = 1) -> bytes:
    """Capture a screenshot and return it as JPEG bytes."""
    with mss() as sct:
        monitor = sct.monitors[monitor_index]
        screenshot = sct.grab(monitor)

        img = Image.frombytes("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX")
        buffer = io.BytesIO()
        img.save(buffer, format="JPEG", quality=70)
        return buffer.getvalue()
Test your screenshot capture

The function is defined, but you need to confirm it actually captures your screen. You'll run a quick test from the terminal.

  • Make sure your terminal is in the ai-screen-reader folder with your virtual environment activated.
  • Run this command to capture a screenshot and print its size:
python -c "from main import capture_screenshot; img = capture_screenshot(); print('Screenshot captured! Size:', len(img), 'bytes')"

You should see output like Screenshot captured! Size: 187432 bytes. The exact number depends on your screen resolution and what's on your display.

Seeing an error?

  • If you see ModuleNotFoundError, make sure your virtual environment is activated. You should see (.venv) at the start of your terminal prompt.
  • On Linux, the keyboard library requires root access. Try running the command with sudo if you get a permissions error.
  • If you see a size of 0 bytes, check that no other application is blocking screen capture access on your system.
  • Help me troubleshoot my screenshot capture error.

What about the global hotkey?

The keyboard library can register a global hotkey like Ctrl+Alt that triggers your capture function even when your app is running in the background. You'll wire this up in Step 4 when you build the system tray app.

Your screenshot function is working. Next up, you'll send these screenshots to Gemini and stream back AI-powered analysis in real time.

Send Screenshots to Gemini and Stream the Response

Your app can capture the screen as a JPEG image on demand. That's a great start, but a screenshot sitting in memory doesn't help anyone.

In this step, you'll send those screenshots to Gemini 2.5 Flash, Google's multimodal AI model that can understand images. The model will analyze what's on your screen and stream its response back chunk by chunk, so you see text appear in real time instead of waiting for the whole answer.

In this step, get ready to:
  • Create a GeminiClient class that manages your API connection and conversation history.
  • Send screenshots to Gemini and stream the response text as it arrives.
  • Test the full capture-to-response flow end to end.
Create the GeminiClient class

Every call to Gemini needs a client object and a conversation history so the model remembers what it has already seen. A dedicated class keeps all of that state in one place.

  • In main.py, find the line load_dotenv().
  • Add the following class below it, above the existing capture_screenshot function:
class GeminiClient:
    """Handles communication with the Gemini API."""

    def __init__(self, api_key: str):
        # Create a Gemini client using the provided API key
        self.client = genai.Client(api_key=api_key)
        # Set the model to Gemini 2.5 Flash (free tier, vision-capable)
        self.model = "gemini-2.5-flash"
        # Store conversation history so Gemini remembers previous messages
        self.history = []

What does this code do?

genai.Client(api_key=api_key) creates a connection to Google's Gemini API using the key from your .env file.

self.history is a list that accumulates every message (yours and the model's). Gemini reads this list on each call so it has context from earlier in the conversation.

You won't see a change yet. The next chunk adds the method that actually sends images to Gemini.

Add the streaming methods

Gemini's streaming API sends the response in small text chunks as the model generates them. This means your app can show text appearing in real time instead of freezing while it waits for the full answer.

  • Inside the GeminiClient class, add the analyze_screenshot method below __init__:
    def analyze_screenshot(self, image_bytes: bytes, prompt: str):
        """Send a screenshot to Gemini and yield streamed text chunks."""
        # Wrap the raw JPEG bytes as an image part Gemini can read
        image_part = types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg")
        # Wrap the text prompt as a separate part
        text_part = types.Part.from_text(text=prompt)

        # Bundle both parts into a single user message
        user_content = types.Content(
            role="user",
            parts=[image_part, text_part]
        )
        self.history.append(user_content)

        # Stream the response so chunks arrive as they are generated
        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        # Yield each text chunk as it arrives and accumulate the full response
        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        # Save the model's complete reply to history for context in follow-ups
        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

What does this code do?

  • types.Part.from_bytes wraps your JPEG screenshot into a format the Gemini API understands. The mime_type tells Gemini it's receiving a JPEG image.
  • generate_content_stream returns a generator that yields text as Gemini produces it. Each chunk.text is a small piece of the answer.
  • After all chunks arrive, the full response is saved to self.history so Gemini has context for follow-up questions.

You won't see a change yet. The next chunk adds support for follow-up questions.

Your app also needs a way to ask follow-up questions without sending another screenshot, and a way to reset the conversation when the user wants to start fresh.

  • Still inside the GeminiClient class, add the ask_followup and clear_history methods below analyze_screenshot:
    def ask_followup(self, question: str):
        """Send a text-only follow-up question."""
        # Create a text-only message (no image this time)
        user_content = types.Content(
            role="user",
            parts=[types.Part.from_text(text=question)]
        )
        self.history.append(user_content)

        # Stream the response the same way as analyze_screenshot
        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        # Save the model's reply to history
        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

    def clear_history(self):
        """Reset conversation history."""
        self.history = []

What does this code do?

ask_followup follows the same streaming pattern as analyze_screenshot, but sends only text. Because the screenshot is already stored in self.history, Gemini still has the image context when answering follow-ups.

clear_history resets the conversation to an empty list. This is useful when the user wants a completely fresh analysis of a new screen.

  • Save main.py.

You won't see a change yet. The next substep wires everything together for a live test.

Test the full capture-to-response flow

Time to see your AI in action. You'll add a small test block at the bottom of main.py that registers the hotkey, captures your screen, and prints Gemini's streamed response to the terminal. This test block is temporary. You'll replace it with the full GUI in the next step.

  • At the very bottom of main.py, below the capture_screenshot function, add this temporary test code:
# --- Temporary test code (remove in Step 4) ---
if __name__ == "__main__":
    api_key = os.getenv("GEMINI_API_KEY")
    if not api_key:
        print("Error: GEMINI_API_KEY not found. Check your .env file.")
        sys.exit(1)

    gemini = GeminiClient(api_key)

    def on_hotkey():
        print("Capturing screen...")
        image_bytes = capture_screenshot()
        print("Sending to Gemini...")
        for chunk in gemini.analyze_screenshot(
            image_bytes, "What do you see on this screen? Be concise."
        ):
            print(chunk, end="", flush=True)
        print("\n\nDone!")

    keyboard.add_hotkey("ctrl+alt", on_hotkey)
    print("Press Ctrl+Alt to capture and analyze your screen.")
    print("Press Ctrl+C to quit.")
    keyboard.wait()

What does this code do?

  • It reads your API key from the .env file and creates a GeminiClient.
  • keyboard.add_hotkey("ctrl+alt", on_hotkey) registers a global hotkey. When you press Ctrl+Alt, the on_hotkey function captures your screen, sends it to Gemini, and prints each text chunk as it arrives.
  • keyboard.wait() keeps the script alive and listening for hotkey presses until you press Ctrl+C.
  • Save main.py.
  • Run the script by running this command:
uv run main.py
  • You should see Press Ctrl+Alt to capture and analyze your screen. printed in the terminal.
  • Press Ctrl+Alt on your keyboard.
  • Wait a few seconds. You should see Capturing screen... followed by Sending to Gemini..., then Gemini's response appearing word by word in your terminal.
  • Press Ctrl+C to stop the script when you're done.

Not seeing a response from Gemini?

  • Check that your .env file contains a valid API key. You can verify it at https://aistudio.google.com/apikey.
  • If you see Error: GEMINI_API_KEY not found, make sure your .env file is in the same folder as main.py and has no spaces around the = sign.
  • On Linux, the keyboard library requires root access. Try running sudo uv run main.py instead.
  • If you see a rate limit error, the Gemini free tier allows 10 requests per minute. Wait a moment and try again.

Still stuck? Ask for help.

The test block you just added is temporary. You'll replace it with a full PyQt6 GUI in the next step.

  • In main.py, delete everything from the # --- Temporary test code (remove in Step 4) --- comment down to the end of the file.
  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py with the temporary test code removed.

ⓧ I'd like to double check the full code

Here is the complete main.py at this point. Make sure your file matches this exactly (without the temporary test code):

import sys
import os
import io
import base64
import threading
from pathlib import Path

from dotenv import load_dotenv
from PIL import Image
from mss import mss
import keyboard
from google import genai
from google.genai import types

from PyQt6.QtWidgets import (
    QApplication, QWidget, QSystemTrayIcon, QMenu,
    QTextEdit, QLineEdit, QVBoxLayout
)
from PyQt6.QtCore import Qt, pyqtSignal, QObject
from PyQt6.QtGui import QIcon, QPixmap, QAction, QPainter, QColor

load_dotenv()


class GeminiClient:
    """Handles communication with the Gemini API."""

    def __init__(self, api_key: str):
        self.client = genai.Client(api_key=api_key)
        self.model = "gemini-2.5-flash"
        self.history = []

    def analyze_screenshot(self, image_bytes: bytes, prompt: str):
        """Send a screenshot to Gemini and yield streamed text chunks."""
        image_part = types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg")
        text_part = types.Part.from_text(text=prompt)

        user_content = types.Content(
            role="user",
            parts=[image_part, text_part]
        )
        self.history.append(user_content)

        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

    def ask_followup(self, question: str):
        """Send a text-only follow-up question."""
        user_content = types.Content(
            role="user",
            parts=[types.Part.from_text(text=question)]
        )
        self.history.append(user_content)

        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

    def clear_history(self):
        """Reset conversation history."""
        self.history = []


def capture_screenshot(monitor_index: int = 1) -> bytes:
    """Capture a screenshot and return it as JPEG bytes."""
    with mss() as sct:
        monitor = sct.monitors[monitor_index]
        screenshot = sct.grab(monitor)

        img = Image.frombytes("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX")
        buffer = io.BytesIO()
        img.save(buffer, format="JPEG", quality=70)
        return buffer.getvalue()

Your AI can now see your screen and tell you about it. Next up, you'll build a proper desktop GUI so those streamed responses appear in a floating chat panel instead of the terminal.

Build the System Tray App with a Chat Panel

Your Gemini integration is working. Press the hotkey, capture the screen, and watch a streamed response print to the console.

Printing to the console is great for testing, but a real desktop assistant needs a proper GUI. In this step, you will build a PyQt6 system tray application with a floating chat panel that streams Gemini's responses in real time.

In this step, get ready to:
  • Create a signal bridge so background threads can safely update the GUI.
  • Build a floating chat panel that displays streamed AI responses and accepts follow-up questions.
  • Wire a system tray icon and global hotkey to bring the entire app together.
Set up the signal bridge for thread safety

PyQt6 has a strict rule: only the main thread can update the GUI. Your Gemini API calls run in background threads to avoid freezing the window.

To bridge the gap, you will create a SignalBridge class. It uses Qt's signal/slot system to safely pass data from a background thread to the main thread.

  • In main.py, find the line load_dotenv() near the top of the file.
  • Add the following class below it, before the GeminiClient class:
class SignalBridge(QObject):
    """Bridge to send signals from background threads to the Qt main thread."""
    # Carries a streamed text chunk from Gemini
    text_chunk = pyqtSignal(str)
    # Fires when the full response has finished streaming
    response_done = pyqtSignal()
    # Carries status messages like "Capturing screen..."
    status_update = pyqtSignal(str)

What does this code do?

Each pyqtSignal defines a message channel. A background thread emits a signal, and the main thread receives it safely.

  • text_chunk carries each piece of Gemini's streamed response so the chat panel can display text as it arrives.
  • response_done fires once the full response is finished, so the input field can re-enable.
  • status_update carries status messages like "Capturing screen..." to keep the user informed.

You will not see a change yet. The next chunk uses this bridge inside the chat panel.

Build the chat panel UI

The chat panel is a floating window that stays on top of other apps. It has a read-only text area for AI responses and a text input for follow-up questions.

  • In main.py, scroll to the bottom of the file, after the capture_screenshot function.
  • Add the ChatPanel class with its __init__ and display helper methods by copying the following code:
class ChatPanel(QWidget):
    """Floating chat panel for displaying AI responses."""

    def __init__(self, gemini_client: GeminiClient, signal_bridge: SignalBridge):
        super().__init__()
        self.gemini_client = gemini_client
        self.signals = signal_bridge
        self.monitor_index = 1

        # Window title and behavior flags
        self.setWindowTitle("AI Screen Reader")
        self.setWindowFlags(
            Qt.WindowType.WindowStaysOnTopHint
            | Qt.WindowType.Tool
        )
        self.setFixedSize(450, 500)

        # Vertical layout with some padding
        layout = QVBoxLayout()
        layout.setContentsMargins(10, 10, 10, 10)

        # Read-only text area for AI responses
        self.chat_display = QTextEdit()
        self.chat_display.setReadOnly(True)
        self.chat_display.setPlaceholderText(
            "Press Ctrl+Alt to capture your screen...\n"
            "AI analysis will appear here."
        )
        layout.addWidget(self.chat_display)

        # Text input for follow-up questions
        self.input_field = QLineEdit()
        self.input_field.setPlaceholderText("Ask a follow-up question...")
        self.input_field.returnPressed.connect(self.send_followup)
        layout.addWidget(self.input_field)

        self.setLayout(layout)

        # Connect signals from background threads to display methods
        self.signals.text_chunk.connect(self.append_text)
        self.signals.response_done.connect(self.on_response_done)
        self.signals.status_update.connect(self.show_status)

    def append_text(self, text: str):
        """Append a text chunk to the chat display."""
        cursor = self.chat_display.textCursor()
        cursor.movePosition(cursor.MoveOperation.End)
        cursor.insertText(text)
        self.chat_display.setTextCursor(cursor)
        self.chat_display.ensureCursorVisible()

    def on_response_done(self):
        """Called when streaming is complete."""
        self.append_text("\n\n")
        self.input_field.setEnabled(True)
        self.input_field.setFocus()

    def show_status(self, message: str):
        """Show a status message in the chat."""
        self.chat_display.append(f"\n--- {message} ---\n")

What does this code do?

The __init__ method builds the window layout and connects the signal bridge to display methods.

  • WindowStaysOnTopHint keeps the chat panel visible over other windows. Tool makes it a lightweight utility window that does not show up in the taskbar.
  • append_text() moves the cursor to the end and inserts a text chunk. This is how streamed text appears character by character.
  • on_response_done() re-enables the input field so the user can type follow-up questions once Gemini finishes.

You will not see a change yet. The next chunk adds the methods that trigger screen capture and stream Gemini's response.

  • Still inside the ChatPanel class, add the following methods directly below show_status:
    def trigger_capture(self):
        """Capture screen and send to Gemini."""
        # Bring the chat panel to the front
        self.show()
        self.raise_()
        self.activateWindow()
        self.input_field.setEnabled(False)
        self.signals.status_update.emit("Capturing screen...")

        # Run capture and analysis in a background thread
        thread = threading.Thread(target=self._capture_and_analyze, daemon=True)
        thread.start()

    def _capture_and_analyze(self):
        """Background thread: capture and stream Gemini response."""
        try:
            image_bytes = capture_screenshot(self.monitor_index)
            self.signals.status_update.emit("Analyzing with Gemini...")

            prompt = "What do you see on this screen? Be helpful and concise."
            for chunk in self.gemini_client.analyze_screenshot(image_bytes, prompt):
                self.signals.text_chunk.emit(chunk)

            self.signals.response_done.emit()
        except Exception as e:
            self.signals.status_update.emit(f"Error: {e}")
            self.signals.response_done.emit()

    def send_followup(self):
        """Send a follow-up text question."""
        question = self.input_field.text().strip()
        if not question:
            return

        self.input_field.clear()
        self.input_field.setEnabled(False)
        self.chat_display.append(f"\nYou: {question}\n")

        # Stream the follow-up response in a background thread
        thread = threading.Thread(
            target=self._stream_followup, args=(question,), daemon=True
        )
        thread.start()

    def _stream_followup(self, question: str):
        """Background thread: stream follow-up response."""
        try:
            for chunk in self.gemini_client.ask_followup(question):
                self.signals.text_chunk.emit(chunk)
            self.signals.response_done.emit()
        except Exception as e:
            self.signals.status_update.emit(f"Error: {e}")
            self.signals.response_done.emit()

What does this code do?

These four methods form two pairs that handle the main interaction flows.

  • trigger_capture() shows the panel, disables the input, and kicks off _capture_and_analyze() in a daemon thread. The background thread captures the screen, sends it to Gemini, and emits text chunks through the signal bridge.
  • send_followup() reads the text input and starts _stream_followup() in a daemon thread. This lets the user ask questions about the screenshot without recapturing.
  • Both background methods use try/except to catch API errors and display them in the chat panel instead of crashing silently.
  • Save main.py.

You will not see a change yet. The next substep creates the tray icon and the main() function that launches everything.

Why use daemon threads?

Setting daemon=True means the thread automatically stops when the main application quits. Without it, a long-running Gemini call could keep the process alive even after you close the app.

Wire the system tray and launch the app

A system tray icon lets your app run quietly in the background. The user right-clicks it to access a menu with options like showing the panel, clearing history, or quitting.

  • In main.py, scroll below the ChatPanel class you just added. Add this helper function that draws a simple tray icon:
def create_tray_icon() -> QPixmap:
    """Create a simple colored icon for the system tray."""
    # Start with a transparent 32x32 canvas
    pixmap = QPixmap(32, 32)
    pixmap.fill(QColor(0, 0, 0, 0))
    painter = QPainter(pixmap)
    # Draw a blue circle as the outer ring
    painter.setBrush(QColor(66, 133, 244))
    painter.setPen(Qt.PenStyle.NoPen)
    painter.drawEllipse(2, 2, 28, 28)
    # Draw a white circle as the inner eye
    painter.setBrush(QColor(255, 255, 255))
    painter.drawEllipse(10, 10, 12, 12)
    painter.end()
    return pixmap

What does this code do?

This draws a small blue-and-white "eye" icon in memory using Qt's painting API. It is a 32x32 pixel image with a blue outer circle and a white inner circle. This icon appears in your system tray so you can find the app at a glance.

Now add the main() function that creates the application, wires up the tray icon and hotkey, and launches everything.

  • Directly below create_tray_icon, add the following code at the bottom of main.py:
def main():
    # Load the API key from .env
    api_key = os.getenv("GEMINI_API_KEY")
    if not api_key:
        print("Error: GEMINI_API_KEY not found in environment.")
        print("Create a .env file with: GEMINI_API_KEY=your-key-here")
        print("Get your key at: https://aistudio.google.com/apikey")
        sys.exit(1)

    # Create the Qt application (keeps running even if all windows close)
    app = QApplication(sys.argv)
    app.setQuitOnLastWindowClosed(False)

    # Set up the core components
    signal_bridge = SignalBridge()
    gemini_client = GeminiClient(api_key)
    chat_panel = ChatPanel(gemini_client, signal_bridge)

    # Create the system tray icon
    tray_icon = QSystemTrayIcon()
    tray_icon.setIcon(QIcon(create_tray_icon()))
    tray_icon.setToolTip("AI Screen Reader - Ctrl+Alt to capture")

    # Build the right-click context menu
    tray_menu = QMenu()
    show_action = QAction("Show Panel")
    show_action.triggered.connect(chat_panel.show)
    tray_menu.addAction(show_action)

    clear_action = QAction("Clear History")
    clear_action.triggered.connect(gemini_client.clear_history)
    clear_action.triggered.connect(chat_panel.chat_display.clear)
    tray_menu.addAction(clear_action)

    tray_menu.addSeparator()

    quit_action = QAction("Quit")
    quit_action.triggered.connect(app.quit)
    tray_menu.addAction(quit_action)

    tray_icon.setContextMenu(tray_menu)
    tray_icon.show()

    # Register the global hotkey to trigger screen capture
    keyboard.add_hotkey("ctrl+alt", chat_panel.trigger_capture)

    # Show the panel on startup and start the event loop
    chat_panel.show()
    sys.exit(app.exec())


if __name__ == "__main__":
    main()

What does this code do?

The main() function is the entry point that ties every component together.

  • setQuitOnLastWindowClosed(False) prevents the app from exiting when you close the chat panel. The tray icon keeps the app alive in the background.
  • The tray menu has three actions: Show Panel reopens the chat window, Clear History resets the conversation, and Quit exits the application.
  • keyboard.add_hotkey("ctrl+alt", ...) registers the global hotkey that triggers a screen capture from anywhere on your desktop.
  • app.exec() starts the Qt event loop. The app keeps running until you select Quit from the tray menu.
  • Save main.py.
  • Run your app by running this command:
python main.py

A floating chat panel appears on screen with placeholder text that says "Press Ctrl+Alt to capture your screen...". You should also see a small blue circle icon in your system tray.

  • Press Ctrl+Alt on your keyboard.

The chat panel shows "Capturing screen..." followed by "Analyzing with Gemini...", then Gemini's response streams into the text area word by word.

  • Type a follow-up question in the text input at the bottom (for example, What app is open in the top left?) and press Enter.

Gemini responds based on the screenshot it already analyzed. The conversation context is preserved.

  • Right-click the blue tray icon. You should see a menu with Show Panel, Clear History, and Quit.
  • Click Quit to stop the application.

App not starting or seeing an error?

  • If you see GEMINI_API_KEY not found, make sure your .env file is in the same folder as main.py and contains GEMINI_API_KEY=your-key with no spaces around the equals sign.
  • If the window does not appear, check for indentation errors in the ChatPanel class. All methods must be indented one level inside the class body.
  • On Linux, the keyboard library requires root privileges. Run with sudo python main.py if you see a permissions error.

Why isn't my AI Screen Reader app launching or showing the chat panel?

Hotkey not triggering a capture?

  • Make sure you are pressing Ctrl+Alt together (not Ctrl+Alt plus another key). The keyboard library on Windows can register this two-key combo on its own.
  • If another application uses the same shortcut, it may intercept it. Try closing other apps that use global hotkeys.

Why isn't my Ctrl+Alt hotkey triggering screen capture in my PyQt6 app?

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete main.py file. Compare your code against this and fix any differences.

import sys
import os
import io
import base64
import threading
from pathlib import Path

from dotenv import load_dotenv
from PIL import Image
from mss import mss
import keyboard
from google import genai
from google.genai import types

from PyQt6.QtWidgets import (
    QApplication, QWidget, QSystemTrayIcon, QMenu,
    QTextEdit, QLineEdit, QVBoxLayout
)
from PyQt6.QtCore import Qt, pyqtSignal, QObject
from PyQt6.QtGui import QIcon, QPixmap, QAction, QPainter, QColor

load_dotenv()


class SignalBridge(QObject):
    """Bridge to send signals from background threads to the Qt main thread."""
    text_chunk = pyqtSignal(str)
    response_done = pyqtSignal()
    status_update = pyqtSignal(str)


class GeminiClient:
    """Handles communication with the Gemini API."""

    def __init__(self, api_key: str):
        self.client = genai.Client(api_key=api_key)
        self.model = "gemini-2.5-flash"
        self.history = []

    def analyze_screenshot(self, image_bytes: bytes, prompt: str):
        """Send a screenshot to Gemini and yield streamed text chunks."""
        image_part = types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg")
        text_part = types.Part.from_text(text=prompt)

        user_content = types.Content(
            role="user",
            parts=[image_part, text_part]
        )
        self.history.append(user_content)

        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

    def ask_followup(self, question: str):
        """Send a text-only follow-up question."""
        user_content = types.Content(
            role="user",
            parts=[types.Part.from_text(text=question)]
        )
        self.history.append(user_content)

        response = self.client.models.generate_content_stream(
            model=self.model,
            contents=self.history,
        )

        full_response = ""
        for chunk in response:
            if chunk.text:
                full_response += chunk.text
                yield chunk.text

        assistant_content = types.Content(
            role="model",
            parts=[types.Part.from_text(text=full_response)]
        )
        self.history.append(assistant_content)

    def clear_history(self):
        """Reset conversation history."""
        self.history = []


def capture_screenshot(monitor_index: int = 1) -> bytes:
    """Capture a screenshot and return it as JPEG bytes."""
    with mss() as sct:
        monitor = sct.monitors[monitor_index]
        screenshot = sct.grab(monitor)

        img = Image.frombytes("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX")
        buffer = io.BytesIO()
        img.save(buffer, format="JPEG", quality=70)
        return buffer.getvalue()


class ChatPanel(QWidget):
    """Floating chat panel for displaying AI responses."""

    def __init__(self, gemini_client: GeminiClient, signal_bridge: SignalBridge):
        super().__init__()
        self.gemini_client = gemini_client
        self.signals = signal_bridge
        self.monitor_index = 1

        self.setWindowTitle("AI Screen Reader")
        self.setWindowFlags(
            Qt.WindowType.WindowStaysOnTopHint
            | Qt.WindowType.Tool
        )
        self.setFixedSize(450, 500)

        layout = QVBoxLayout()
        layout.setContentsMargins(10, 10, 10, 10)

        self.chat_display = QTextEdit()
        self.chat_display.setReadOnly(True)
        self.chat_display.setPlaceholderText(
            "Press Ctrl+Alt to capture your screen...\n"
            "AI analysis will appear here."
        )
        layout.addWidget(self.chat_display)

        self.input_field = QLineEdit()
        self.input_field.setPlaceholderText("Ask a follow-up question...")
        self.input_field.returnPressed.connect(self.send_followup)
        layout.addWidget(self.input_field)

        self.setLayout(layout)

        self.signals.text_chunk.connect(self.append_text)
        self.signals.response_done.connect(self.on_response_done)
        self.signals.status_update.connect(self.show_status)

    def append_text(self, text: str):
        """Append a text chunk to the chat display."""
        cursor = self.chat_display.textCursor()
        cursor.movePosition(cursor.MoveOperation.End)
        cursor.insertText(text)
        self.chat_display.setTextCursor(cursor)
        self.chat_display.ensureCursorVisible()

    def on_response_done(self):
        """Called when streaming is complete."""
        self.append_text("\n\n")
        self.input_field.setEnabled(True)
        self.input_field.setFocus()

    def show_status(self, message: str):
        """Show a status message in the chat."""
        self.chat_display.append(f"\n--- {message} ---\n")

    def trigger_capture(self):
        """Capture screen and send to Gemini."""
        self.show()
        self.raise_()
        self.activateWindow()
        self.input_field.setEnabled(False)
        self.signals.status_update.emit("Capturing screen...")

        thread = threading.Thread(target=self._capture_and_analyze, daemon=True)
        thread.start()

    def _capture_and_analyze(self):
        """Background thread: capture and stream Gemini response."""
        try:
            image_bytes = capture_screenshot(self.monitor_index)
            self.signals.status_update.emit("Analyzing with Gemini...")

            prompt = "What do you see on this screen? Be helpful and concise."
            for chunk in self.gemini_client.analyze_screenshot(image_bytes, prompt):
                self.signals.text_chunk.emit(chunk)

            self.signals.response_done.emit()
        except Exception as e:
            self.signals.status_update.emit(f"Error: {e}")
            self.signals.response_done.emit()

    def send_followup(self):
        """Send a follow-up text question."""
        question = self.input_field.text().strip()
        if not question:
            return

        self.input_field.clear()
        self.input_field.setEnabled(False)
        self.chat_display.append(f"\nYou: {question}\n")

        thread = threading.Thread(
            target=self._stream_followup, args=(question,), daemon=True
        )
        thread.start()

    def _stream_followup(self, question: str):
        """Background thread: stream follow-up response."""
        try:
            for chunk in self.gemini_client.ask_followup(question):
                self.signals.text_chunk.emit(chunk)
            self.signals.response_done.emit()
        except Exception as e:
            self.signals.status_update.emit(f"Error: {e}")
            self.signals.response_done.emit()


def create_tray_icon() -> QPixmap:
    """Create a simple colored icon for the system tray."""
    pixmap = QPixmap(32, 32)
    pixmap.fill(QColor(0, 0, 0, 0))
    painter = QPainter(pixmap)
    painter.setBrush(QColor(66, 133, 244))
    painter.setPen(Qt.PenStyle.NoPen)
    painter.drawEllipse(2, 2, 28, 28)
    painter.setBrush(QColor(255, 255, 255))
    painter.drawEllipse(10, 10, 12, 12)
    painter.end()
    return pixmap


def main():
    api_key = os.getenv("GEMINI_API_KEY")
    if not api_key:
        print("Error: GEMINI_API_KEY not found in environment.")
        print("Create a .env file with: GEMINI_API_KEY=your-key-here")
        print("Get your key at: https://aistudio.google.com/apikey")
        sys.exit(1)

    app = QApplication(sys.argv)
    app.setQuitOnLastWindowClosed(False)

    signal_bridge = SignalBridge()
    gemini_client = GeminiClient(api_key)
    chat_panel = ChatPanel(gemini_client, signal_bridge)

    tray_icon = QSystemTrayIcon()
    tray_icon.setIcon(QIcon(create_tray_icon()))
    tray_icon.setToolTip("AI Screen Reader - Ctrl+Alt to capture")

    tray_menu = QMenu()
    show_action = QAction("Show Panel")
    show_action.triggered.connect(chat_panel.show)
    tray_menu.addAction(show_action)

    clear_action = QAction("Clear History")
    clear_action.triggered.connect(gemini_client.clear_history)
    clear_action.triggered.connect(chat_panel.chat_display.clear)
    tray_menu.addAction(clear_action)

    tray_menu.addSeparator()

    quit_action = QAction("Quit")
    quit_action.triggered.connect(app.quit)
    tray_menu.addAction(quit_action)

    tray_icon.setContextMenu(tray_menu)
    tray_icon.show()

    keyboard.add_hotkey("ctrl+alt", chat_panel.trigger_capture)

    chat_panel.show()
    sys.exit(app.exec())


if __name__ == "__main__":
    main()

Secret mission

Multi-Monitor Capture

Your AI assistant always captures the primary monitor. What if you have two or three screens? Add a monitor selector to the system tray menu so you can choose which display Gemini analyzes.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, and the Gemini API free tier has no ongoing costs, so there is nothing to worry about billing-wise.

Resources you used:

  • Python virtual environment (created with uv).
  • Project files: main.py, requirements.txt, and .env.
  • Installed packages: google-genai, PyQt6, mss, keyboard, Pillow, python-dotenv.
  • Gemini API key (from Google AI Studio).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue with the next project in this series.

  • Your project folder, virtual environment, and all files stay on your Desktop.
  • Your Gemini API key in .env remains ready to use.
  • You can relaunch the app anytime by activating your virtual environment and running python main.py.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files and virtual environment stay intact so you can pick up where you left off.

  • Close the AI Screen Reader app by right-clicking the system tray icon and selecting Quit.
  • Close your terminal window.
  • Your project folder, virtual environment, and .env file remain on disk for next time.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes everything you built in this project.

  • Close the AI Screen Reader app if it is still running.
  • Delete your entire project folder from the Desktop.
  • If you no longer need your Gemini API key, go to https://aistudio.google.com/apikey and delete the key.

Mission accomplished

You Built an AI That Can See Your Screen

Nice work! You just built a desktop AI assistant that lives in your system tray, captures your screen on a hotkey, and streams back intelligent analysis from Gemini 2.5 Flash.

You've learned how to:

  • Built a PyQt6 system tray application that runs in the background and captures screenshots on a global hotkey using mss.
  • Sent images to Gemini 2.5 Flash and streamed responses in real time using the google-genai SDK.
  • Created a floating chat panel for typing follow-up questions about what's on your screen.
  • Secret Mission: Added multi-monitor support with a monitor selector dropdown in the system tray menu.

Ready to quiz yourself?