Build an AI QA Agent from JIRA Tickets

Build an AI agent that reads JIRA bugs and auto-tests your web app.

Introduction

30 Second Summary

Every developer knows the pain of manual bug testing. Someone files a ticket, you read it, open the browser, click through the same forms, and try to reproduce the issue. Multiply that by dozens of tickets and your day disappears.

In this project, you will build an AI-powered QA agent that reads bug tickets from JIRA, uses Google Gemini to generate test steps, executes those tests in a real browser, and posts the results back to JIRA automatically.

What You'll Build

You will type a single JIRA ticket number into your terminal, walk away, and come back to find a detailed test report posted as a comment on that ticket, confirming whether the bug exists.

By the end of this project, you'll have:

  • A command-line agent you run with any JIRA ticket number that fetches the bug description and transforms it into structured, executable test steps using Google Gemini.
  • An automated browser test powered by Playwright that launches a real browser, fills forms, clicks buttons, and checks for expected behavior.
  • A closed feedback loop that posts pass/fail results back to JIRA as a comment, so your team sees findings without leaving their workflow.
  • Secret Mission: Add screenshot capture on test failure so your JIRA comments include visual evidence of what went wrong.

Are there any prerequisites?

You need Python and VS Code installed on your machine. This project walks you through setting up everything else, including your free Jira Cloud account and Gemini API key.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

This is the moment you commit to the project goal so every step that follows has a clear purpose.

Set Up Your Tools

Before you can build your AI QA agent, you need the right packages and accounts in place.

In this step, you will install Python libraries for browser automation, AI, and web serving. You will also create accounts for Google Gemini and Jira Cloud so your agent can talk to real services.

In this step, get ready to:
  • Install Python packages and a browser for automated testing.
  • Get a Gemini API key for AI-powered test generation.
  • Set up a Jira Cloud account with a test bug ticket.
Create your project folder and install packages
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open a terminal inside VS Code by clicking Terminal in the top menu bar, then New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir qa-agent
cd qa-agent
  • Install the Python packages your project needs by running this command:
pip install flask==3.1.3 playwright==1.62.0 google-genai==2.17.0 requests

What are these packages?

  • flask is a lightweight web framework. You will use it to build the buggy test target app.
  • playwright automates a real browser. Your agent will use it to click buttons and fill forms.
  • google-genai is the official SDK for the Gemini API. It turns bug descriptions into test steps.
  • requests makes HTTP calls. Your agent uses it to talk to Jira's REST API.

You will see pip downloading and installing each package. Once that finishes, you need to download a browser binary for Playwright to automate.

  • Download the Chromium browser binary by running:
playwright install chromium

You will see Playwright downloading Chromium. This may take a minute depending on your connection.

Before you run the next command, what do you think will happen if all the packages installed correctly?

  • Verify all packages are installed correctly by running:
python -c "from google import genai; from playwright.sync_api import sync_playwright; from flask import Flask; print('All packages ready!')"

You should see All packages ready! printed in the terminal. This confirms every package imported successfully.

Seeing an import error?

  • Make sure you ran the pip install command in the same terminal session. If you opened a new terminal, re-run it.
  • If you see a playwright error, double-check you ran playwright install chromium after the pip install.
  • help me fix my package installation error
  • Open the project folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the qa-agent folder, and click Open.

Why open the folder?

Opening the folder in VS Code gives you the file explorer sidebar. You will create multiple files here in the next steps, and the explorer makes them easy to manage.

Get your Gemini API key

Your agent needs a Gemini API key to ask Google's AI model to generate test steps from bug descriptions.

  • Open your browser and go to aistudio.google.com.
  • Sign in with your Google account.
  • Click Get API key in the left sidebar or on the landing page.
  • Click Create API key.
  • Follow the dialog to create a new key. Google AI Studio may auto-create a project for you.
  • Copy the key and save it somewhere safe (a text file or password manager). You will paste it into your code later.

Is this free?

Yes. The Gemini API free tier provides free input and output tokens with no billing setup required. The key you just created is an auth key by default, which is the current standard for Gemini API authentication.

Can't create a key?

  • If you see a 403 Access Restricted error, you may not be in a supported region. Check Google's supported region list.
  • If the Create API key button is greyed out, make sure you have accepted the Terms of Service in Google AI Studio.
  • help me troubleshoot my Gemini API key creation
Set up Jira Cloud and create a test ticket

Your agent reads bug reports from Jira, so you need a Jira Cloud account with a project and a test ticket to work with.

  • Open your browser and go to atlassian.com/try/cloud/signup.
  • Sign up with your email address. No credit card is required for the Free plan.
  • Follow the setup wizard to create your Jira site. Choose any site name you like.

What is Jira Cloud Free?

Jira Cloud Free supports up to 10 users, unlimited projects, and includes full REST API access. It is completely free with no time limit.

Once your Jira site is ready, create a project for your QA agent to work with.

  • In Jira, click Projects in the top navigation.
  • Click Create project.
  • Choose a Scrum or Kanban template (either works).
  • Set the project name to QA and confirm the project key is QA.
  • Click Create.

Now create the bug ticket your agent will read and test against.

  • Click Create (the button in the top navigation bar) to open the issue creation dialog.
  • Set the issue type to Bug.
  • Set the summary to: Registration form accepts empty password.
  • In the description field, paste the following text:
Steps: 1. Go to registration page 2. Enter username but leave password empty 3. Click Register. Expected: Error message shown. Actual: Form submits successfully.
  • Click Create to save the ticket.

Your ticket should now appear as QA-1 in your project. This is the ticket ID your agent will use.

Finally, you need a Jira API token so your agent can authenticate with Jira's REST API.

  • Open a new browser tab and go to id.atlassian.com/manage-profile/security/api-tokens.
  • Click Create API token.
  • Enter a label like qa-agent and click Create.
  • Click Copy and save the token somewhere safe alongside your Gemini API key. You cannot view this token again after closing the dialog.

Save both credentials now

You should now have two credentials saved: your Gemini API key and your Jira API token. You will also need your Jira site URL (something like your-site.atlassian.net) and the email address you used to sign up. Keep all four values handy for the next steps.

Your environment is ready. You have packages installed, an AI key for test generation, and a Jira account with a real bug ticket. Next up, you will build the buggy web app that your agent will test against.

Build the Buggy Web App

Your tools are installed and your Jira ticket is waiting. Now your AI agent needs a target to test against.

In this step, you will build a simple Flask web app with a registration form. The twist: you will plant intentional bugs in the form validation so your agent has real defects to discover later.

In this step, get ready to:
  • Create a Flask app with intentional validation bugs.
  • Add HTML templates for the registration form and success page.
  • Run the app and confirm the bugs exist by submitting invalid data.
Create the Flask application

Flask needs a templates folder to store HTML files. You will create that folder first, then build the Python app.

  • Create the templates folder by running this command in your VS Code terminal:
mkdir templates
  • Confirm the folder exists by running:
ls

You should see templates listed in the output.

  • Create a new file called app.py in your project folder. In VS Code, click File in the top menu, then New File.
  • Save the file as app.py in your project folder (not inside templates).
  • Add the following code to app.py to set up the Flask app with its routes:
from flask import Flask, render_template, request, redirect, url_for

app = Flask(__name__)

users = []


@app.route("/")
def home():
    return redirect(url_for("register"))


@app.route("/register", methods=["GET", "POST"])
def register():
    if request.method == "POST":
        username = request.form.get("username", "")
        email = request.form.get("email", "")
        password = request.form.get("password", "")

        # BUG: No validation! Accepts empty passwords, short usernames, etc.
        users.append({"username": username, "email": email, "password": password})
        return redirect(url_for("success"))

    return render_template("register.html")

What does this code do?

  • The / route redirects visitors straight to the registration form.
  • The /register route handles both showing the form (GET) and processing submissions (POST).
  • The intentional bug: when a form is submitted, the code saves the data without checking if the password is empty or if the username is too short. A real app would validate these fields.
  • Add the success route and the startup block below the register function in app.py:
@app.route("/success")
def success():
    return render_template("success.html")


if __name__ == "__main__":
    app.run(debug=True, port=5000)

What does this code do?

  • The /success route shows a confirmation page after registration.
  • The if __name__ == "__main__" block starts the Flask development server on port 5000 with debug mode enabled for auto-reloading.
  • Save app.py.
  • Verify the app loads without errors by running this in your terminal:
python app.py

You should see output including Running on http://127.0.0.1:5000. The server starts even though templates do not exist yet. Flask only looks for them when a page is requested.

  • Press Ctrl+C in the terminal to stop the server. You will restart it after adding templates.

Server not starting?

  • Check that app.py is saved in your project folder, not inside templates.
  • If you see ModuleNotFoundError: No module named 'flask', confirm you installed Flask in the previous step.
  • help me debug why my Flask app won't start
Add the HTML templates

Flask renders HTML templates from the templates folder. You need two pages: the registration form and a success confirmation.

  • Create a new file called register.html inside the templates folder. In VS Code, right-click the templates folder in the Explorer sidebar and select New File.
  • Name the file register.html and paste the following code:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Register</title>
    <style>
        body { font-family: Arial, sans-serif; max-width: 400px; margin: 50px auto; padding: 20px; }
        form { display: flex; flex-direction: column; gap: 15px; }
        label { font-weight: bold; }
        input { padding: 8px; border: 1px solid #ccc; border-radius: 4px; }
        button { padding: 10px; background-color: #007bff; color: white; border: none; border-radius: 4px; cursor: pointer; }
        button:hover { background-color: #0056b3; }
        .error { color: red; font-size: 14px; }
    </style>
</head>
<body>
    <h1>Create Account</h1>
    <form method="POST" action="/register" id="register-form">
        <label for="username">Username</label>
        <input type="text" id="username" name="username" placeholder="Enter username">

        <label for="email">Email</label>
        <input type="email" id="email" name="email" placeholder="Enter email">

        <label for="password">Password</label>
        <input type="password" id="password" name="password" placeholder="Enter password">

        <button type="submit" id="register-btn">Register</button>
    </form>
</body>
</html>

What does this template do?

  • The form POSTs to /register, which hits the Flask route you just wrote.
  • Each input has an id attribute (like #username, #password, #register-btn) that Playwright will use later to target elements during testing.
  • Notice the form has no client-side validation either. Both the front end and back end are wide open.
  • Save register.html.
  • Create another new file inside templates called success.html and paste this code:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Registration Successful</title>
    <style>
        body { font-family: Arial, sans-serif; max-width: 400px; margin: 50px auto; padding: 20px; text-align: center; }
        .success-message { color: green; font-size: 24px; margin-top: 50px; }
    </style>
</head>
<body>
    <div class="success-message" id="success-msg">Registration Successful!</div>
    <p>Your account has been created.</p>
    <a href="/register">Register another account</a>
</body>
</html>

What does this template do?

This is the page Flask shows after a successful registration. The #success-msg element gives Playwright a selector to check whether the form submission went through. If the agent sees this page after submitting invalid data, it confirms the bug.

  • Save success.html.
  • Start the Flask server again by running:
python app.py

You should see the server running on http://127.0.0.1:5000 again.

  • Open your browser and go to http://localhost:5000.

You should see a clean registration form with Username, Email, and Password fields.

Seeing a TemplateNotFound error?

  • Make sure your templates folder is in the same directory as app.py, not nested inside another folder.
  • Confirm the file is named exactly register.html (lowercase, no typos).
  • help me fix the TemplateNotFound error in my Flask app

✔️ Awesome, I've got everything!

Great. Make sure both files are saved in templates/ and the form is visible in your browser.

ⓧ I'd like to double check the full code

Here is the full register.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Register</title>
    <style>
        body { font-family: Arial, sans-serif; max-width: 400px; margin: 50px auto; padding: 20px; }
        form { display: flex; flex-direction: column; gap: 15px; }
        label { font-weight: bold; }
        input { padding: 8px; border: 1px solid #ccc; border-radius: 4px; }
        button { padding: 10px; background-color: #007bff; color: white; border: none; border-radius: 4px; cursor: pointer; }
        button:hover { background-color: #0056b3; }
        .error { color: red; font-size: 14px; }
    </style>
</head>
<body>
    <h1>Create Account</h1>
    <form method="POST" action="/register" id="register-form">
        <label for="username">Username</label>
        <input type="text" id="username" name="username" placeholder="Enter username">

        <label for="email">Email</label>
        <input type="email" id="email" name="email" placeholder="Enter email">

        <label for="password">Password</label>
        <input type="password" id="password" name="password" placeholder="Enter password">

        <button type="submit" id="register-btn">Register</button>
    </form>
</body>
</html>

And here is success.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Registration Successful</title>
    <style>
        body { font-family: Arial, sans-serif; max-width: 400px; margin: 50px auto; padding: 20px; text-align: center; }
        .success-message { color: green; font-size: 24px; margin-top: 50px; }
    </style>
</head>
<body>
    <div class="success-message" id="success-msg">Registration Successful!</div>
    <p>Your account has been created.</p>
    <a href="/register">Register another account</a>
</body>
</html>

Scroll up and confirm your app.py contains both code chunks (the imports through the register route, then the success route and main block).

Verify the bugs exist

The whole point of this app is that it is broken on purpose. Your AI agent will discover these bugs later, but first you need to confirm they are real.

Before you test, what do you think will happen if you submit the form with an empty password field? Will the app reject it, or let it through?

  • In your browser at http://localhost:5000, type testuser in the Username field.
  • Type test@example.com in the Email field.
  • Leave the Password field completely empty.
  • Click Register.

You land on the success page showing "Registration Successful!" even though the password was empty. That is the bug.

Why plant bugs on purpose?

Your AI agent needs defects to find. If the app worked perfectly, the agent would have nothing to report. By planting known bugs (empty passwords accepted, short usernames allowed, no duplicate checking), you create a controlled testing ground where you can verify the agent actually catches real issues.

  • Return to your terminal and press Ctrl+C to stop the Flask server.

Your buggy web app is ready. Next up, you will build the agent that reads your JIRA ticket and uses AI to generate test steps for this exact form.

Build the JIRA Reader and AI Test Generator

Your buggy web app is running and ready to be tested. Now you need something that can understand what to test.

The agent needs to read a real bug report from JIRA, understand what it describes, then figure out the exact steps to verify whether that bug exists. In this step, you will build two functions: one that fetches a ticket from JIRA's REST API, and another that sends the bug description to Google Gemini to generate structured, executable test steps.

In this step, get ready to:
  • Build a JIRA client function that fetches bug ticket details using the REST API.
  • Build a Gemini-powered function that transforms bug descriptions into structured test steps.
  • Wire both functions together and confirm the agent prints a test plan to the console.
Create agent.py with imports and configuration

Your agent needs several libraries: requests for HTTP calls to JIRA, google-genai for Gemini, and Python's built-in json and sys modules. The configuration block stores your personal credentials so the rest of the code can reference them.

  • Create a new file called agent.py in your project folder (the same folder that contains app.py) by adding the following code:
import sys
import json
import requests
from google import genai
from playwright.sync_api import sync_playwright

# Configuration - replace with your values
JIRA_BASE_URL = "https://your-domain.atlassian.net"
JIRA_EMAIL = "your-email@example.com"
JIRA_API_TOKEN = "your-api-token-here"
GEMINI_API_KEY = "your-gemini-api-key-here"
APP_URL = "http://localhost:5000"

What does this code do?

  • The imports bring in everything the agent will need: sys for command-line arguments, json for parsing AI responses, requests for HTTP calls, genai for Gemini, and sync_playwright for browser automation later.
  • JIRA_BASE_URL is your Jira Cloud site address. JIRA_EMAIL and JIRA_API_TOKEN are used together for Basic authentication against the JIRA API.
  • APP_URL points to your local Flask app so the AI knows where to direct test steps.
  • Replace your-domain in JIRA_BASE_URL with your actual Jira Cloud domain (the part before .atlassian.net in your browser's address bar when you are logged in).
  • Replace your-email@example.com with the email address you used to create your Jira account.
  • Replace your-api-token-here with the Jira API token you saved in Step 1.
  • Replace your-gemini-api-key-here with the Gemini API key you saved in Step 1.
  • Save agent.py.
  • Confirm the file appears in VS Code's Explorer sidebar alongside app.py.

Don't see agent.py in the Explorer?

Make sure you saved the file in the same folder as app.py. If you accidentally saved it somewhere else, delete it and recreate it in the correct location.

help me find where my agent.py file was saved

Build the JIRA client function

JIRA Cloud stores ticket descriptions in a nested JSON structure called Atlassian Document Format (ADF). Your function needs to call the API, then walk through that nested structure to extract the plain text.

  • In agent.py, add the following function below the configuration constants:
def fetch_jira_ticket(ticket_key):
    """Fetch ticket details from JIRA REST API."""
    url = f"{JIRA_BASE_URL}/rest/api/3/issue/{ticket_key}"
    auth = (JIRA_EMAIL, JIRA_API_TOKEN)
    headers = {"Accept": "application/json"}

    response = requests.get(url, auth=auth, headers=headers)
    response.raise_for_status()

    data = response.json()
    summary = data["fields"]["summary"]

    # Extract description text from ADF format
    description_content = data["fields"].get("description", {})
    description_text = ""
    if description_content and "content" in description_content:
        for block in description_content["content"]:
            if "content" in block:
                for item in block["content"]:
                    if item.get("type") == "text":
                        description_text += item.get("text", "")
                description_text += "\n"

    print(f"Ticket: {summary}")
    print(f"Description: {description_text}")
    return summary, description_text

What does this function do?

  • It calls GET /rest/api/3/issue/{ticket_key} using Basic authentication (your email + API token as a tuple).
  • raise_for_status() will throw an error immediately if the request fails (wrong credentials, ticket not found, etc.) instead of silently returning bad data.
  • The nested for loops walk through the ADF structure to pull out plain text from the description field. JIRA wraps each paragraph in content blocks, so the code digs into each one.

The function is defined but not callable yet. You need a main() entry point to run it from the terminal.

  • Add this main function and entry point below fetch_jira_ticket:
def main():
    if len(sys.argv) < 2:
        print("Usage: python agent.py <TICKET-KEY>")
        print("Example: python agent.py QA-1")
        sys.exit(1)

    ticket_key = sys.argv[1]
    print(f"AI QA Agent starting for ticket: {ticket_key}\n")

    # Step 1: Fetch ticket from JIRA
    summary, description = fetch_jira_ticket(ticket_key)


if __name__ == "__main__":
    main()

What does main do?

  • It reads the ticket key (like QA-1) from the command line arguments. If no argument is provided, it prints usage instructions and exits.
  • The if __name__ == "__main__" guard ensures main() only runs when you execute the file directly.
  • Save agent.py.
  • Test the JIRA connection by running this command in your terminal:
python agent.py QA-1

You should see output like this:

AI QA Agent starting for ticket: QA-1

Ticket: Registration form accepts empty password
Description: Steps: 1. Go to registration page 2. Enter username but leave password empty 3. Click Register. Expected: Error message shown. Actual: Form submits successfully.

Seeing a 401 Unauthorized error?

This means your JIRA credentials are wrong. Double check that JIRA_EMAIL matches the email you used to sign up for Jira, and that JIRA_API_TOKEN is the API token (not your account password). Also confirm JIRA_BASE_URL uses https:// and ends with .atlassian.net with no trailing slash.

If you need a new API token, go to id.atlassian.com/manage-profile/security/api-tokens and create one.

help me debug my JIRA API authentication error

Add the AI test generator

The agent can now read from JIRA, but a raw bug description is not something a browser can execute. You need Gemini to transform that human-written text into a structured list of test steps: what to click, what to type, and what to check.

  • In agent.py, add the first half of the generate_test_steps function between fetch_jira_ticket and main. This defines the function and builds the prompt:
def generate_test_steps(summary, description):
    """Use Gemini to generate structured test steps from the bug description."""
    client = genai.Client(api_key=GEMINI_API_KEY)

    prompt = f"""You are a QA automation engineer. Given this bug report, generate structured test steps to verify the bug exists.

Bug Title: {summary}
Bug Description: {description}

Target app URL: {APP_URL}

Respond with ONLY a JSON array of test steps. Each step should have:
- "action": one of "navigate", "fill", "click", "assert_visible", "assert_not_visible", "assert_url"
- "selector": CSS selector or URL (use id selectors like #username, #password, #register-btn, #success-msg)
- "value": value to fill or expected value (empty string if not applicable)
- "description": human-readable description of what this step does

Example format:
[
  {{"action": "navigate", "selector": "{APP_URL}/register", "value": "", "description": "Go to registration page"}},
  {{"action": "fill", "selector": "#username", "value": "testuser", "description": "Enter a username"}},
  {{"action": "click", "selector": "#register-btn", "value": "", "description": "Click register button"}}
]"""

What does this prompt do?

The prompt gives Gemini a clear role (QA automation engineer) and asks it to output a specific JSON structure. Each test step has an action (what to do), a selector (which element to target), a value (what to type or expect), and a description (human-readable summary).

The example format teaches Gemini the exact shape you need. It references the real CSS selectors from your registration form (#username, #register-btn) so Gemini produces selectors that actually match your HTML.

  • Immediately below the prompt (still inside the same function), add the code that sends the prompt to Gemini and parses the response:
    response = client.models.generate_content(
        model="gemini-2.0-flash",
        contents=prompt
    )

    response_text = response.text.strip()
    # Remove markdown code fences if present
    if response_text.startswith("```"):
        response_text = response_text.split("\n", 1)[1]
        response_text = response_text.rsplit("```", 1)[0]

    test_steps = json.loads(response_text)
    print("\nGenerated Test Steps:")
    for i, step in enumerate(test_steps, 1):
        print(f"  {i}. {step['description']}")
    return test_steps

What does this code do?

  • client.models.generate_content sends the prompt to the gemini-2.0-flash model and gets back a text response.
  • LLMs sometimes wrap JSON output in markdown code fences (```json ... ```). The cleanup code strips those so json.loads can parse the raw JSON.
  • The parsed test steps are printed to the console so you can see what the AI came up with before the agent tries to execute them.

Now you need to tell main() to actually call this new function after fetching the ticket.

  • In agent.py, scroll down to your main() function and find this line:
    summary, description = fetch_jira_ticket(ticket_key)
  • Add these two lines directly below it:
    # Step 2: Generate test steps with AI
    test_steps = generate_test_steps(summary, description)
  • Save agent.py.

Before you run this, what do you think the output will look like? The agent will fetch the ticket from JIRA, send its description to Gemini, and print whatever test steps the AI generates.

  • Make sure your Flask app from Step 2 is NOT running (it is not needed yet). Run the agent:
python agent.py QA-1

You should see the ticket details followed by a numbered list of test steps generated by Gemini. The output looks something like this:

AI QA Agent starting for ticket: QA-1

Ticket: Registration form accepts empty password
Description: Steps: 1. Go to registration page 2. Enter username but leave password empty ...

Generated Test Steps:
  1. Go to registration page
  2. Enter a username
  3. Leave password field empty
  4. Click Register button
  5. Assert error message is visible

Why might your output look slightly different?

Gemini is non-deterministic. It might generate 4 steps or 7 steps, and the wording will vary each time you run it. What matters is that you see structured steps that relate to the bug description. If the steps mention navigating to the registration page and testing with an empty password, the AI understood the ticket correctly.

Seeing a JSON parsing error?

If you see json.JSONDecodeError, Gemini returned text that was not valid JSON. Run the command again. LLM outputs vary between calls, and a retry usually produces valid JSON.

If the error persists, check that your GEMINI_API_KEY is correct and that you have not exceeded the free tier rate limit (wait a minute and try again).

help me debug why Gemini is not returning valid JSON test steps

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and that running python agent.py QA-1 prints both the ticket details and the AI-generated test steps.

ⓧ I'd like to double check the full code

Your agent.py should contain these sections in order. Scroll through your file and confirm each one is present:

  • Imports (sys, json, requests, genai, sync_playwright).
  • Configuration constants (JIRA_BASE_URL, JIRA_EMAIL, JIRA_API_TOKEN, GEMINI_API_KEY, APP_URL) with your real values filled in.
  • fetch_jira_ticket(ticket_key) function that builds a URL, calls the JIRA API, and parses the ADF description.
  • generate_test_steps(summary, description) function that creates a Gemini client, builds a prompt, sends it, strips code fences, parses JSON, and prints the steps.
  • main() function that reads sys.argv[1], calls fetch_jira_ticket, then calls generate_test_steps.
  • if __name__ == "__main__": main() entry point at the bottom.

Your agent can now read bug tickets and think about how to test them. The test steps exist as a plan in the console, but nothing has actually been executed against your web app yet. Next, you will give the agent hands: it will launch a browser, perform those test steps, and report the results back to JIRA.

Execute Tests and Report to JIRA

In the last step, you gave your agent the ability to read a JIRA ticket and ask Gemini to generate a test plan. The problem is that a test plan sitting in the console is useless.

The agent needs to actually run those tests against the live app and report what it finds back to where the team works. In this step, you will build the execution engine and the JIRA reporter that close this loop.

In this step, get ready to:
  • Build a Playwright test executor that launches a browser and performs each AI-generated test step.
  • Post a formatted test report back to JIRA as a comment on the original ticket.
  • Run the full agent end-to-end and confirm the bug is reported in JIRA.
Build the Playwright test executor

Playwright can launch a real browser and perform actions like clicking buttons, filling forms, and checking for elements. Your executor function will loop through each AI-generated test step and translate it into a browser action.

  • In agent.py, find the end of the generate_test_steps function. Add the first part of the execute_tests function below it:
def execute_tests(test_steps):
    """Execute test steps using Playwright and return results."""
    results = []

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        for step in test_steps:
            action = step["action"]
            selector = step["selector"]
            value = step["value"]
            description = step["description"]

            try:
                if action == "navigate":
                    page.goto(selector)
                elif action == "fill":
                    if value:
                        page.fill(selector, value)
                    else:
                        page.fill(selector, "")
                elif action == "click":
                    page.click(selector)
                    page.wait_for_load_state("networkidle")

What does this code do?

This function launches a visible Chromium browser and iterates over each test step from the AI-generated plan.

  • headless=False means you can watch the browser perform each action in real time.
  • Each action type maps to a Playwright method: goto for navigation, fill for typing into inputs, click for buttons.
  • wait_for_load_state("networkidle") pauses after a click until the page finishes loading.
  • Directly below the click handler (still inside the try block), add the assertion handlers and result collection:
                elif action == "assert_visible":
                    element = page.locator(selector)
                    assert element.is_visible(), f"Element {selector} is not visible"
                elif action == "assert_not_visible":
                    element = page.locator(selector)
                    assert not element.is_visible(), f"Element {selector} is visible but should not be"
                elif action == "assert_url":
                    current_url = page.url
                    assert value in current_url, f"Expected URL to contain '{value}', got '{current_url}'"

                results.append({"step": description, "status": "PASS", "details": ""})
                print(f"  PASS: {description}")

            except Exception as e:
                results.append({"step": description, "status": "FAIL", "details": str(e)})
                print(f"  FAIL: {description} - {e}")

        browser.close()

    return results

What does the result collection do?

The assertion handlers check whether elements are visible or the URL changed as expected. If any assertion fails, Python raises an exception.

The try/except block catches failures gracefully. Each step gets recorded as PASS or FAIL with error details, so the agent never crashes mid-run.

  • Now update main() to call the new function. Find the comment # Step 2: Generate test steps with AI and the line below it. Add the following after that block:
    # Step 3: Execute tests with Playwright
    print("\nExecuting tests...")
    results = execute_tests(test_steps)
  • Save agent.py.
  • Make sure your Flask app is running. If it is not running, open a new terminal and start it:
python app.py

Before you run the agent, what do you think will happen when it hits the empty-password test? Will the assertion pass or fail?

  • In your other terminal (not the one running Flask), run the agent:
python agent.py QA-1

A Chromium browser window opens and you can watch the agent fill in the form, click buttons, and check results. In your terminal, you see PASS and FAIL lines printed for each test step.

Why run the browser in visible mode?

Running headed (headless=False) lets you watch every action the agent performs. This is invaluable for debugging test steps that target the wrong selector. Production agents typically run headless for speed.

Browser not opening or tests failing immediately?

  • Make sure your Flask app is running in a separate terminal. The agent navigates to http://localhost:5000, so the server must be active.
  • If you see a timeout error, your Flask app may have crashed. Check the Flask terminal for errors and restart it with python app.py.
  • If Chromium fails to launch, try running playwright install chromium again to re-download the browser.

Help me debug why my Playwright tests are not executing

Post results back to JIRA

Right now the test results only appear in your terminal. The team needs to see them where they actually work: on the JIRA ticket itself.

  • In agent.py, add the post_results_to_jira function after the execute_tests function:
def post_results_to_jira(ticket_key, results):
    """Post test results as a comment on the JIRA ticket."""
    passed = sum(1 for r in results if r["status"] == "PASS")
    failed = sum(1 for r in results if r["status"] == "FAIL")
    total = len(results)

    # Build comment body in Atlassian Document Format (ADF)
    report_lines = f"AI QA Agent Test Report\n\nResults: {passed}/{total} passed, {failed}/{total} failed\n\n"
    for r in results:
        icon = "PASS" if r["status"] == "PASS" else "FAIL"
        report_lines += f"[{icon}] {r['step']}\n"
        if r["details"]:
            report_lines += f"    Details: {r['details']}\n"

    report_lines += f"\nConclusion: {'Bug CONFIRMED - test failures match the reported behavior.' if failed > 0 else 'All tests passed - bug may be fixed.'}"

    # Use v2 API for simple text comments
    url = f"{JIRA_BASE_URL}/rest/api/2/issue/{ticket_key}/comment"
    auth = (JIRA_EMAIL, JIRA_API_TOKEN)
    headers = {"Content-Type": "application/json"}
    payload = {"body": report_lines}

    response = requests.post(url, auth=auth, headers=headers, json=payload)
    response.raise_for_status()
    print(f"\nResults posted to JIRA ticket {ticket_key}")

What does this function do?

  • It counts how many steps passed and failed, then builds a human-readable report string.
  • The conclusion line automatically states whether the bug was confirmed based on whether any tests failed.
  • It uses the JIRA REST API v2 endpoint for comments, which accepts plain text. The v3 endpoint requires a complex JSON document format.
  • In the main() function, find the line results = execute_tests(test_steps). Add the following lines after it:
    # Step 4: Report results back to JIRA
    post_results_to_jira(ticket_key, results)

    print("\nAgent complete!")
  • Save agent.py.

Why use the v2 API for comments?

JIRA Cloud's v3 API requires comments in Atlassian Document Format (ADF), a nested JSON structure that is complex to construct. The v2 API accepts a simple text string in the body field, which is much simpler for agent-generated reports.

Run the full agent end-to-end

Before you run this final test, predict: will the agent confirm the bug or report all tests passing?

  • Make sure your Flask app is still running in its terminal.
  • Run the agent from your other terminal:
python agent.py QA-1

The browser opens, the agent runs each test step, and your terminal prints PASS/FAIL results followed by "Results posted to JIRA ticket QA-1" and "Agent complete!".

  • Open your JIRA project in the browser.
  • Click on ticket QA-1.
  • Scroll down to the comments section. You should see a new comment titled "AI QA Agent Test Report" with pass/fail results and a conclusion stating "Bug CONFIRMED - test failures match the reported behavior."

Comment not appearing in JIRA?

  • Check that your JIRA_API_TOKEN and JIRA_EMAIL in agent.py are correct. Basic auth requires your Atlassian account email and an API token generated from id.atlassian.com.
  • Verify JIRA_BASE_URL matches your site exactly, including the https:// prefix and .atlassian.net suffix.
  • If you see a 404 error, make sure the ticket key QA-1 exists in your project and matches the project key you created.

Help me debug why my agent cannot post comments to JIRA

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py with all the changes from this step.

ⓧ I'd like to double check the full code

Your agent.py is long now. Scroll up and confirm each of these sections is present, in order:

  • Imports and configuration variables at the top.
  • fetch_jira_ticket(ticket_key) function.
  • generate_test_steps(summary, description) function.
  • execute_tests(test_steps) function with the browser launch, action handlers (navigate, fill, click, assert_visible, assert_not_visible, assert_url), result collection, and browser.close().
  • post_results_to_jira(ticket_key, results) function with the report builder and v2 API POST.
  • main() function calling all four steps: fetch ticket, generate steps, execute tests, and post results.
  • The if __name__ == "__main__": main() block at the very end.

Your AI agent now does everything: reads a JIRA ticket, generates a test plan with Gemini, executes real browser tests, and reports findings back to JIRA. Next up, you can take this further by adding screenshot evidence when tests fail.

Secret mission

Add Screenshot Evidence on Failure

Your agent posts text results to JIRA, but QA teams need more than words. When a test fails, what did the browser actually show? Add automatic screenshot capture on failure so every bug report comes with visual proof.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally with free-tier services, so there are no ongoing costs.

Resources you used:

  • Local project folder (app.py, agent.py, templates/, screenshots/).
  • Python packages (flask, playwright, google-genai, requests).
  • Playwright Chromium browser binary.
  • Gemini API key in Google AI Studio.
  • Jira Cloud Free account with test project.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your AI QA agent or reuse parts of this project.

  • Your Jira Cloud Free account stays active for future projects at no cost.
  • Your Gemini API key remains on the free tier with no billing required.
  • The Flask app only runs when you start it manually, so nothing is consuming resources in the background.

Pause - I'll come back to this later

No running services to pause. Flask only runs when you explicitly start it with python app.py. If you've stopped that process already, everything is paused.

  • Make sure you've pressed Ctrl+C in any terminal where the Flask app was running.
  • Your project files, API key, and Jira account will all be waiting for you when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to clean up completely.

  • Delete your local project folder by running this command in your terminal:
  • macOS/Linux:
  • Windows:
cd ~/Desktop
rm -rf qa-agent

On Windows, run:

cd ~\Desktop
rmdir /s /q qa-agent
  • Uninstall the Playwright browser binary by running:
playwright uninstall
  • Uninstall the Python packages by running:
pip uninstall flask playwright google-genai requests -y
  • Delete your Gemini API key. Open the Google Cloud API Credentials page at console.cloud.google.com/apis/credentials. Find the API key you created for this project, click the Actions icon next to it, select Delete API key, and confirm.
  • Delete your Jira API token. Go to id.atlassian.com/manage-profile/security/api-tokens and revoke the token you created for this project.
  • Optionally, delete your Jira Cloud site. Go to admin.atlassian.com, select your site, navigate to Settings, and look for the option to delete the site under Billing. Free sites may also be automatically deactivated by Atlassian after a period of inactivity.

Nice Work!

Nice Work!

You just built an AI-powered QA agent that reads bug tickets, generates test plans, executes browser tests, and reports results. That is a full automation loop from ticket to verified finding.

You've learned how to:

  • Integrate with JIRA's REST API to read bug tickets and post automated test results back as comments.
  • Use Google Gemini to transform human-written bug descriptions into structured, executable test steps.
  • Automate browser testing with Playwright to verify bugs against a live web application without manual clicking.
  • Secret Mission: Added screenshot capture on failure to provide visual debugging evidence alongside test reports.

Ready to quiz yourself?