Build an Autonomous Student Agent

Deploy a serverless AI agent that emails you daily hackathon picks.

Introduction

30 Second Summary

You know that feeling when you discover an amazing hackathon the day after registration closed? Or when a friend mentions a perfect opportunity you never even heard about? The best opportunities don't wait for you to find them.

In this project, you will build a personal AI agent that wakes up every morning, scrapes live hackathon listings from the internet, uses AI to filter them based on your interests, and delivers a polished email digest straight to your inbox.

What You'll Build

You run the agent once, walk away, and every morning at 7 AM a curated list of hackathons and opportunities lands in your inbox without you lifting a finger.

By the end of this project, you'll have:

A daily email digest arriving in your inbox each morning with hackathons and opportunities ranked by how well they match your personal interests.
An AI-powered filter that cuts through dozens of irrelevant listings and surfaces only the events worth your time, with a one-sentence reason for each pick.
A fully autonomous serverless pipeline on AWS that runs daily at zero cost, combining web scraping, AI reasoning, and email delivery without your laptop needing to be on.
Secret Mission: Add S3-based history tracking so the agent remembers what it already sent you and never shows the same opportunity twice.

Are there any prerequisites?

You need an existing AWS account, Python installed on your machine, and VS Code ready to go. Step 1 walks you through everything else, including setting up the AWS CLI and getting your free Google Gemini API key.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters to you personally.

Set Up Your Tools and Project

Your agent needs two things to work: a way to deploy code to the cloud, and an AI brain to do the filtering. The AWS CLI lets you manage AWS resources from your terminal. Google Gemini provides the intelligence that turns a wall of raw hackathon listings into a curated shortlist.

In this step, you will install and configure both tools, then record your project's Python dependencies so the agent is ready to build.

In this step, get ready to:

Configure the AWS CLI to deploy resources from your terminal.
Obtain a Google Gemini API key for AI-powered filtering.
Install the Python packages your agent depends on.

Install and configure the AWS CLI

The AWS CLI is a command-line tool that lets you create, configure, and delete AWS resources without clicking through the web console. You need it to deploy your Lambda function and schedule it later.

Check if the AWS CLI is already installed by running this command in your terminal:

aws --version

✔️ I see a version number

You are good to go. The AWS CLI is already installed on your machine.

ⓧ Command not found

You need to install the AWS CLI. Follow these steps for your operating system.

macOS

Download the macOS installer by opening this URL in your browser:
https://awscli.amazonaws.com/AWSCLIV2.pkg
Double-click the downloaded AWSCLIV2.pkg file and follow the installer prompts.
Once the installer finishes, close and reopen your terminal.
Confirm the install by running:

aws --version

You should see a version number printed.

Windows

Download the Windows installer by opening this URL in your browser:
https://awscli.amazonaws.com/AWSCLIV2.msi
Run the downloaded AWSCLIV2.msi file and follow the installer prompts.
Once the installer finishes, close and reopen your terminal.
Confirm the install by running:

aws --version

You should see a version number printed.

Now you need to connect the CLI to your AWS account. The aws configure command will prompt you for four values one at a time.

Run the configuration command:

aws configure

What will this command ask?

The command prompts you for four values, one per line. Enter them in this order:

AWS Access Key ID - paste your access key from the AWS IAM console.
AWS Secret Access Key - paste your secret key from the AWS IAM console.
Default region name - type us-east-1.
Default output format - type json.

Find your access keys

If you do not have an Access Key ID and Secret Access Key yet, create them in the AWS Console under IAM > Users > your username > Security credentials > Create access key.

Copy both values immediately. The Secret Access Key is only shown once.

Before you run the next command, what do you think it will print if your credentials are valid?

Verify your configuration by running:

aws sts get-caller-identity

You should see a JSON response containing your AWS Account ID, UserId, and ARN. This proves your CLI can talk to AWS.

Seeing an error instead of your account info?

Double-check that you pasted the Access Key ID and Secret Access Key correctly. Re-run aws configure if needed.
Make sure your terminal session is fresh. Close and reopen it after installing the CLI.
help me debug my AWS CLI configuration

Get your Gemini API key and install packages

Google Gemini is the AI model your agent uses to decide which opportunities match your interests. You need an API key to authenticate requests to the Gemini API.

Open Google AI Studio in your browser.
Sign in with your Google account.
Click Create API key on the API keys page.
Select a project in the dialog that appears (AI Studio creates a default project for new users automatically).
Copy the generated API key and save it somewhere safe (a notes app, a password manager, or a local text file that is not in your project folder).

Why is the Gemini API free?

The Gemini API Free Tier charges nothing for input or output tokens. Your agent makes one small request per day, well within the free limits.

Cannot create an API key?

If you see a permissions error, try creating a new Google Cloud project that is not linked to an organization. Personal Gmail accounts typically work without issues.
Try opening AI Studio in an incognito window or a different browser.
help me get a Gemini API key

Now install the four Python packages your agent needs.

Install the project dependencies by running:

pip install google-generativeai requests beautifulsoup4 boto3

What are these packages?

google-generativeai - the official Python SDK for calling the Gemini API.
requests - a simple HTTP library for fetching web pages.
beautifulsoup4 - an HTML parser for extracting data from web pages.
boto3 - the AWS SDK for Python, used to send emails via SES.

What do you expect to see if the Gemini SDK installed correctly?

Verify the packages installed successfully by running:

python -c "import google.generativeai; print('Gemini SDK ready')"

You should see Gemini SDK ready printed with no errors.

Seeing a ModuleNotFoundError?

Make sure you used the same Python that pip installed packages into. Try pip3 or python3 if you are on macOS.
If you use a virtual environment, make sure it is activated before running both commands.
help me fix my Python import error for google-generativeai

Finally, record these dependencies in your project's requirements file so the Lambda deployment package knows what to include.

Open requirements.txt in the opportunity-agent folder in VS Code.
Paste the following content:

google-generativeai
requests
beautifulsoup4
boto3

What does this file do?

A requirements.txt file lists every Python package your project depends on. When you deploy to Lambda later, you will use this file to install dependencies into the deployment package.

Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Double check you have saved requirements.txt with all four package names.

ⓧ I'd like to double check the full code

Your requirements.txt should contain exactly this:

google-generativeai
requests
beautifulsoup4
boto3

Your tools are locked and loaded. Next up, you will write the scraper functions that pull live hackathon listings from the internet.

Scrape Live Opportunities

Your environment is wired up with the AWS CLI, a Gemini API key, and all the Python packages you need. Now it is time to give your agent something to work with: raw data.

Before the AI can filter anything, the agent needs live hackathon listings pulled from the internet. In this step, you will write scrapers that fetch real event data from Devpost and MLH, giving you something tangible to see immediately.

In this step, get ready to:

Build a scraper that pulls open hackathons from Devpost.
Build a scraper that pulls upcoming events from MLH.
Combine both sources and print live opportunity data from your terminal.

Build the Devpost scraper

Devpost hosts thousands of hackathon listings on a public page. Since Devpost has no official API, you will use web scraping to pull data directly from the HTML.

Open lambda_function.py in VS Code and add the imports along with the beginning of the scrape_devpost() function:

import json
import os
import re
import requests
from bs4 import BeautifulSoup
import google.generativeai as genai
import boto3


def scrape_devpost():
    """Scrape open hackathons from Devpost."""
    url = "https://devpost.com/hackathons?status=open"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    response = requests.get(url, headers=headers, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")

What do these imports do?

requests fetches web pages over HTTP.
BeautifulSoup parses HTML so you can extract specific elements by CSS selector.
json, os, re, genai, and boto3 are needed for later steps. Including them now avoids editing the top of the file again.

Directly below, add the parsing loop that completes the function:

    hackathons = []
    tiles = soup.select("a.hackathon-tile")
    for tile in tiles:
        title_el = tile.select_one("h3")
        if not title_el:
            continue
        title = title_el.get_text(strip=True)
        link = tile.get("href", "")
        subtitle = tile.select_one(".submission-period")
        deadline = subtitle.get_text(strip=True) if subtitle else "No deadline listed"
        prize_el = tile.select_one(".prize-amount")
        prize = prize_el.get_text(strip=True) if prize_el else "No prize listed"

        hackathons.append({
            "source": "Devpost",
            "title": title,
            "url": link,
            "deadline": deadline,
            "prize": prize,
        })

    return hackathons

What does this parsing loop do?

It selects each hackathon tile using the CSS selector a.hackathon-tile and loops through them.
For each tile, it extracts the title, URL, submission deadline, and prize amount into a dictionary.
The function returns a list of these dictionaries so other code can use the scraped data.

Save lambda_function.py.
Test your Devpost scraper by running this command in your terminal:

python -c "from lambda_function import scrape_devpost; print(len(scrape_devpost()), 'hackathons found')"

You should see a number followed by hackathons found. The exact count depends on how many open hackathons Devpost has listed today.

Seeing 0 hackathons or an error?

Make sure you are running the command from inside your opportunity-agent folder where lambda_function.py lives.
If you see ModuleNotFoundError, confirm that you installed the packages with pip install requests beautifulsoup4 in your current Python environment.
A count of 0 can happen if Devpost changed their page structure. The agent handles this gracefully in the next substep.

Help me debug why my Devpost scraper returns 0 results.

Build the MLH scraper

MLH (Major League Hacking) lists their season events on a public page. Like Devpost, MLH has no public API for event discovery, so you will scrape it the same way.

In lambda_function.py, add the beginning of the scrape_mlh() function below scrape_devpost() (leave two blank lines between functions):

def scrape_mlh():
    """Scrape upcoming hackathons from MLH."""
    url = "https://mlh.io/na"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    response = requests.get(url, headers=headers, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")

Directly below, add the parsing loop that completes the function:

    hackathons = []
    events = soup.select(".event-wrapper")
    for event in events:
        name_el = event.select_one(".event-name")
        if not name_el:
            continue
        name = name_el.get_text(strip=True)
        link_el = event.select_one("a[href]")
        link = link_el.get("href", "") if link_el else ""
        date_el = event.select_one(".event-date")
        date = date_el.get_text(strip=True) if date_el else "No date listed"
        location_el = event.select_one(".event-location")
        location = location_el.get_text(strip=True) if location_el else "Online"

        hackathons.append({
            "source": "MLH",
            "title": name,
            "url": link,
            "deadline": date,
            "prize": f"Location: {location}",
        })

    return hackathons

What does this function do?

It fetches the MLH North America events page and parses each .event-wrapper element.
For each event, it extracts the name, link, date, and location into the same dictionary shape as the Devpost scraper.
The unified shape means both scrapers produce data that can be combined into a single list without any transformation.

Save lambda_function.py.
Verify the MLH scraper works by running:

python -c "from lambda_function import scrape_mlh; print(len(scrape_mlh()), 'MLH events found')"

You should see a number followed by MLH events found. Even a small count means the scraper is working.

Getting 0 events or a connection error?

Check your internet connection and try visiting https://mlh.io/na in your browser to confirm the page loads.
If only one source returns results (Devpost works but MLH does not, or vice versa), that is fine. The agent keeps going even if one source is temporarily unavailable.

Help me debug why my MLH scraper returns 0 results.

Wire both scrapers together and run

Each scraper works on its own, but the agent needs a single entry point that combines results from every source. The get_opportunities() function handles this and catches failures gracefully so one broken source does not crash the entire agent.

Below scrape_mlh() (with two blank lines separating), add the get_opportunities() function and the __main__ block:

def get_opportunities():
    """Combine opportunities from all sources."""
    opportunities = []
    try:
        opportunities.extend(scrape_devpost())
    except Exception as e:
        print(f"Devpost scraping failed: {e}")
    try:
        opportunities.extend(scrape_mlh())
    except Exception as e:
        print(f"MLH scraping failed: {e}")
    return opportunities


if __name__ == "__main__":
    os.environ.setdefault("GEMINI_API_KEY", "your-gemini-api-key-here")
    os.environ.setdefault("RECIPIENT_EMAIL", "your-email@example.com")
    os.environ.setdefault("INTERESTS_JSON", open("interests.json").read())
    results = get_opportunities()
    print(json.dumps(results, indent=2))

What does this code do?

get_opportunities() calls both scrapers inside try/except blocks. If Devpost is down or MLH changes their HTML, the other source still delivers results.
The __main__ block sets default environment variables (used by later steps) and prints the combined results as formatted JSON so you can inspect the data.

Save lambda_function.py.

Before you run this, how many total hackathons do you think both sources will return combined? Make a guess.

Run the full scraper pipeline by running:

python lambda_function.py

You should see a JSON array printed to your terminal with 10 or more entries. Each entry contains a source, title, url, deadline, and prize field.

These are real, live hackathon listings happening right now.

Seeing an empty list or an error?

If you see [] (empty list), both scrapers returned nothing. Check your internet connection and try visiting https://devpost.com/hackathons?status=open in your browser to confirm the page loads.
If you see a FileNotFoundError about interests.json, make sure the file exists in the same folder as lambda_function.py. It can be empty for now.
If only one source returned results (you see a "scraping failed" message for the other), that is fine. The agent is designed to keep going even if one source is temporarily unavailable.

Help me debug why my opportunity scraper is not returning results.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your file contains these sections in order:

The 7-line import block at the top.
The scrape_devpost() function (fetches the page, parses tiles, returns a list of dicts).
The scrape_mlh() function (fetches the page, parses event wrappers, returns a list of dicts).
The get_opportunities() function (calls both scrapers inside try/except).
The if __name__ == "__main__": block at the very bottom.

Make sure there are two blank lines between each function definition. Save the file and rerun python lambda_function.py to confirm it works.

Your agent can now pull live data from the internet on its own. Next up, you will feed this raw dump to Gemini AI so it can filter out the noise and rank only the opportunities that match your interests.

Add AI-Powered Filtering

Your scraper pulls real hackathon data from the web. The problem is you now have dozens of events dumped into a wall of text.

Without filtering, this is no better than visiting the websites yourself. In this step, you will connect Google Gemini to your agent so it can intelligently filter, rank, and summarize only the opportunities that match your personal interests.

In this step, get ready to:

Define your interests profile so the AI knows what to look for.
Build a Gemini-powered filter function that ranks opportunities by relevance.
Run the full pipeline locally and compare raw vs. filtered results.

Create your interests profile

The AI needs decision criteria to know what counts as relevant. Your interests.json file acts as that criteria. It tells Gemini which topics matter to you, which formats you prefer, and which keywords to ignore.

Open interests.json in VS Code and paste the following content:

{
  "topics": ["AI", "machine learning", "web development", "beginner-friendly"],
  "format_preference": "online",
  "min_prize_usd": 0,
  "exclude_keywords": ["blockchain", "crypto", "web3"],
  "max_results": 10
}

What does this file do?

topics lists the subjects you want to see in your digest.
format_preference tells the AI you prefer online events over in-person.
exclude_keywords filters out anything containing those terms, regardless of other matches.
max_results caps the output so your email stays digestible.

Feel free to customize these values to match your actual interests. The more specific you are, the better Gemini's filtering becomes.

Save interests.json.
Confirm the file saved correctly by running:

cat interests.json

You should see your JSON printed back with your topics, format preference, and exclude keywords.

Seeing an empty file or an error?

Make sure you saved the file in VS Code before running cat.
Confirm you are in the opportunity-agent folder in your terminal.
Check that the JSON is valid. Every key and string value needs double quotes, and arrays use square brackets.Help me fix my interests.json file

Build the AI filter function

Now you will write a function that sends the raw opportunities to Gemini along with your interests profile. Gemini acts as a personal scout, returning only the events worth your attention, each with a one-sentence reason.

In lambda_function.py, find the get_opportunities() function. Add the following new function directly below it (before the if __name__ block):

def filter_with_ai(opportunities, interests):
    """Use Gemini to filter and rank opportunities based on interests."""
    genai.configure(api_key=os.environ["GEMINI_API_KEY"])
    model = genai.GenerativeModel("gemini-2.5-flash")

    prompt = f"""You are a personal opportunity scout for a student.

Here are the student's interests:
{json.dumps(interests, indent=2)}

Here are the raw opportunities found today:
{json.dumps(opportunities, indent=2)}

Your job:
1. Filter out opportunities that don't match the student's interests or topics.
2. Filter out anything containing excluded keywords.
3. Rank the remaining opportunities by relevance (best match first).
4. Return at most {interests.get('max_results', 10)} opportunities.

Return ONLY a valid JSON array. Each item must have:
- "title": string
- "url": string
- "deadline": string
- "prize": string
- "source": string
- "reason": a one-sentence explanation of why this matches the student

Return the JSON array and nothing else. No markdown formatting."""

What does this prompt do?

The prompt feeds Gemini two pieces of context: the student's interests and the raw scraped data. It then gives Gemini four explicit instructions (filter, exclude, rank, cap) and a strict output format.

Asking for structured JSON with a reason field forces Gemini to justify each pick. This makes your email digest more useful because you can see why each opportunity was selected.

Directly below the prompt (still inside the same function), add the API call and response parsing:

    response = model.generate_content(prompt)
    text = response.text.strip()
    # Strip markdown code fences that Gemini sometimes wraps around JSON
    text = re.sub(r"^```json\s*", "", text)
    text = re.sub(r"\s*```$", "", text)

    try:
        filtered = json.loads(text)
    except json.JSONDecodeError:
        filtered = []

    return filtered

Why strip markdown fences?

Even when you ask Gemini to return raw JSON, it sometimes wraps the response in ```json fences. The two re.sub lines remove those fences so json.loads can parse the response cleanly.

The try/except block handles cases where Gemini returns something unparseable. Instead of crashing, the function returns an empty list.

Save lambda_function.py.
Verify the function loaded without syntax errors by running:

python -c "from lambda_function import filter_with_ai; print('filter_with_ai loaded')"

You should see filter_with_ai loaded printed with no errors.

Seeing a SyntaxError or IndentationError?

Make sure the triple-quoted prompt string ends with """ on its own line, indented at the same level as the prompt = assignment.
Confirm all lines after the prompt (response = , text = , etc.) are indented with 4 spaces inside the function.
Check that return filtered is the last line of the function, indented once (4 spaces).Help me fix my filter_with_ai function

Test the full pipeline locally

The filter function is ready but the script still prints raw results. You need to update the __main__ block so it scrapes, filters with AI, and shows you the difference.

In lambda_function.py, scroll to the bottom and find these two lines inside the if __name__ == "__main__": block:

    results = get_opportunities()
    print(json.dumps(results, indent=2))

Replace those two lines with the following:

    opportunities = get_opportunities()
    print(f"Raw count: {len(opportunities)}")
    interests = json.loads(os.environ["INTERESTS_JSON"])
    filtered = filter_with_ai(opportunities, interests)
    print(f"Filtered count: {len(filtered)}")
    print(json.dumps(filtered, indent=2))

What changed?

Instead of printing all raw results, the script now loads your interests from the environment variable, passes them to filter_with_ai(), and prints both the raw count and filtered count. This lets you see exactly how much noise the AI removed.

Save lambda_function.py.

Before you run this, what do you think will happen? Will the filtered count be lower than the raw count?

Set your Gemini API key and run the full pipeline:

export GEMINI_API_KEY=your-gemini-api-key-here
python lambda_function.py

Replace your-gemini-api-key-here with the actual API key you saved in Step 1.

You should see output showing a raw count (10+ events), a lower filtered count, and a JSON array where each opportunity includes a reason explaining why it matched your interests.

Getting an API error or empty results?

If you see a 400 or 403 error, double-check that you copied your full Gemini API key with no extra spaces.
If the filtered count is 0 but the raw count is high, your interests might be too narrow. Try broadening topics in interests.json or removing some exclude_keywords.
If the raw count is 0, the scraper could not reach Devpost or MLH. Check your internet connection and try again.Help me debug my Gemini API call

✔️ Awesome, I've got everything!

Great. Double check you have saved both interests.json and lambda_function.py.

ⓧ I'd like to double check the full code

Here is the complete interests.json:

{
  "topics": ["AI", "machine learning", "web development", "beginner-friendly"],
  "format_preference": "online",
  "min_prize_usd": 0,
  "exclude_keywords": ["blockchain", "crypto", "web3"],
  "max_results": 10
}

For lambda_function.py, confirm the following pieces are present in this order:

Imports at the top (json, os, re, requests, BeautifulSoup, genai, boto3).
scrape_devpost() function from Step 2.
scrape_mlh() function from Step 2.
get_opportunities() function from Step 2.
filter_with_ai(opportunities, interests) function you just added (prompt + API call + JSON parsing).
Updated if __name__ == "__main__": block that calls both get_opportunities() and filter_with_ai().

Your agent now has a brain. It scrapes raw data and distils it into a personalized shortlist. Next up, you will turn that shortlist into a polished HTML email and deliver it straight to your inbox.

Build and Send the Email Digest

Your agent can scrape and filter opportunities, but a printed terminal output is useless when you are asleep.

In this step, you will turn the filtered results into a polished HTML email delivered straight to your inbox via Amazon SES. This completes the autonomous loop: scrape, filter, deliver.

In this step, get ready to:

Build an HTML email template that renders each opportunity as a styled card with clickable links.
Verify your email address in Amazon SES so your agent can send to your inbox.
Wire up the full pipeline and receive your first digest email.

Build the opportunity card template

Email clients strip external stylesheets and most CSS selectors. The only reliable way to style an HTML email is with inline CSS on every element.

In lambda_function.py, scroll down below the filter_with_ai function.
Add the first part of a new build_html_email function that builds individual opportunity cards:

def build_html_email(opportunities, date_str):
    """Build a styled HTML email from filtered opportunities."""
    if not opportunities:
        cards_html = "<p>No new opportunities matched your interests today. Check back tomorrow!</p>"
    else:
        cards = []
        for i, opp in enumerate(opportunities, 1):
            card = f"""
            <div style="background:#ffffff;border:1px solid #e0e0e0;border-radius:8px;padding:16px;margin-bottom:12px;">
                <h3 style="margin:0 0 8px 0;color:#1a1a2e;">#{i} {opp['title']}</h3>
                <p style="margin:4px 0;color:#555;font-size:14px;">Source: {opp.get('source', 'Unknown')} | Deadline: {opp.get('deadline', 'N/A')}</p>
                <p style="margin:4px 0;color:#555;font-size:14px;">Prize: {opp.get('prize', 'N/A')}</p>
                <p style="margin:8px 0 8px 0;color:#2d6a4f;font-size:14px;font-style:italic;">{opp.get('reason', '')}</p>
                <a href="{opp['url']}" style="color:#0066cc;font-weight:bold;text-decoration:none;">View Opportunity &rarr;</a>
            </div>"""
            cards.append(card)
        cards_html = "".join(cards)

What does this code do?

This function handles two scenarios. If no opportunities matched your interests, it displays a friendly fallback message.

Otherwise, it loops through each opportunity and builds a styled card with the title, source, deadline, prize, the AI's reasoning for including it, and a clickable link. Every style is applied inline because email clients ignore external CSS.

Directly below the cards_html = "".join(cards) line (still inside the same function), add the outer HTML template that wraps the cards:

    html = f"""
    <html>
    <body style="font-family:Arial,sans-serif;background:#f5f5f5;padding:20px;margin:0;">
        <div style="max-width:600px;margin:0 auto;">
            <div style="background:#1a1a2e;color:#ffffff;padding:20px;border-radius:8px 8px 0 0;text-align:center;">
                <h1 style="margin:0;font-size:22px;">Your Daily Opportunity Digest</h1>
                <p style="margin:8px 0 0 0;opacity:0.8;font-size:14px;">{date_str}</p>
            </div>
            <div style="background:#f9f9f9;padding:20px;border-radius:0 0 8px 8px;">
                <p style="color:#333;margin-bottom:16px;">Found <strong>{len(opportunities)}</strong> opportunities matching your interests:</p>
                {cards_html}
            </div>
            <p style="text-align:center;color:#999;font-size:12px;margin-top:16px;">
                Sent by your Autonomous Opportunity Agent running on AWS Lambda
            </p>
        </div>
    </body>
    </html>"""
    return html

What does the template do?

The outer wrapper creates a centered 600px container (the standard email width). It has a dark header with the digest title and today's date, a light body area where the opportunity cards render, and a footer crediting your agent.

The date_str parameter gets passed in later so the email always shows when it was generated.

Save lambda_function.py.
Confirm there are no syntax errors by running this quick import check:

python -c "from lambda_function import build_html_email; print('Email template loaded')"

You should see Email template loaded printed with no errors.

Seeing a SyntaxError?

Check that the second code block is indented with 4 spaces (it is inside the build_html_email function body, not at the top level).
Make sure the triple-quote f-strings use three double quotes f""" and close with """ on the same indentation level.
Verify there is no blank line between the first code block (cards logic) and the second code block (html template). They are one continuous function.

Still stuck? Help me fix a SyntaxError in my build_html_email function

Set up SES and verify your email

Amazon SES needs to confirm you own the email address before it will send on your behalf. In sandbox mode, you can only send to verified addresses. Since you are sending to yourself, one verification covers both sender and recipient.

Open the AWS Management Console in your browser.
In the search bar at the top, type SES and select Amazon Simple Email Service.
Confirm the region selector in the top-right shows us-east-1. If not, switch to US East (N. Virginia).
In the left navigation pane under Configuration, click Identities.
Click Create identity.
Under Identity details, select Email address.
Enter your personal email address (the one you want to receive the daily digest at).
Click Create identity at the bottom of the page.

What happens next?

SES sends a verification email from no-reply-aws@amazon.com to the address you entered. It usually arrives within a few minutes. Check your spam folder if you do not see it.

Open your email inbox and find the verification email from AWS.
Click the verification link in the email.
Return to the SES console and refresh the page. Your email should now show a Verified status in the Status column.

Still showing Pending?

Check your spam or junk folder for an email from no-reply-aws@amazon.com.
Make sure you are looking at the SES console in the us-east-1 region (the same region you created the identity in).
The verification link expires after 24 hours. If it has been longer, delete the identity and create it again.

Help me verify my email identity in SES

Add the SES sending function and wire up the pipeline

With SES verified, you can now write the function that delivers the email. Then you will connect all the pieces into a single lambda_handler function that AWS Lambda will invoke.

In lambda_function.py, add the send_email function below build_html_email:

def send_email(html_body, recipient):
    """Send the HTML email via Amazon SES."""
    ses = boto3.client("ses", region_name=os.environ.get("AWS_REGION", "us-east-1"))
    ses.send_email(
        Source=recipient,
        Destination={"ToAddresses": [recipient]},
        Message={
            "Subject": {"Data": "Your Daily Opportunity Digest", "Charset": "UTF-8"},
            "Body": {"Html": {"Data": html_body, "Charset": "UTF-8"}},
        },
    )

What does this function do?

It creates a boto3 SES client in the us-east-1 region. The Source and Destination are both the same address (yours) because SES sandbox mode requires both to be verified.

The Message payload wraps the HTML body you built earlier. SES handles the actual SMTP delivery to your inbox.

Save lambda_function.py and confirm the new function loads cleanly:

python -c "from lambda_function import send_email; print('SES function loaded')"

You should see SES function loaded with no errors.

Now add the lambda_handler function below send_email. This is the orchestration function that Lambda will call:

def lambda_handler(event, context):
    """Main Lambda entry point."""
    from datetime import datetime

    email = os.environ["RECIPIENT_EMAIL"]
    interests = json.loads(os.environ["INTERESTS_JSON"])

    print("Scraping opportunities...")
    opportunities = get_opportunities()
    print(f"Found {len(opportunities)} raw opportunities")

    if not opportunities:
        print("No opportunities scraped. Skipping AI filter and email.")
        return {"statusCode": 200, "body": "No opportunities found."}

    print("Filtering with Gemini AI...")
    filtered = filter_with_ai(opportunities, interests)
    print(f"AI selected {len(filtered)} relevant opportunities")

    date_str = datetime.now().strftime("%B %d, %Y")
    html = build_html_email(filtered, date_str)

    print("Sending email via SES...")
    send_email(html, email)
    print("Email sent successfully!")

    return {"statusCode": 200, "body": f"Sent {len(filtered)} opportunities."}

What does lambda_handler do?

This is the full pipeline in one function. It reads your email and interests from environment variables, scrapes opportunities, sends them to Gemini for filtering, builds the HTML email, and delivers it via SES.

The event and context parameters are required by AWS Lambda. When EventBridge triggers your function later, Lambda passes scheduling metadata through these parameters.

Now update the if __name__ == "__main__": block at the bottom of the file. Find these lines:

if __name__ == "__main__":
    os.environ.setdefault("GEMINI_API_KEY", "your-gemini-api-key-here")
    os.environ.setdefault("RECIPIENT_EMAIL", "your-email@example.com")
    os.environ.setdefault("INTERESTS_JSON", open("interests.json").read())
    opportunities = get_opportunities()
    print(f"Raw count: {len(opportunities)}")
    interests = json.loads(os.environ["INTERESTS_JSON"])
    filtered = filter_with_ai(opportunities, interests)
    print(f"Filtered count: {len(filtered)}")
    print(json.dumps(filtered, indent=2))

Replace the entire block with this simplified version that calls lambda_handler directly:

if __name__ == "__main__":
    os.environ.setdefault("GEMINI_API_KEY", "your-gemini-api-key-here")
    os.environ.setdefault("RECIPIENT_EMAIL", "your-email@example.com")
    os.environ.setdefault("INTERESTS_JSON", open("interests.json").read())
    lambda_handler({}, None)

Why change the main block?

Previously, the __main__ block called each function individually for testing. Now that lambda_handler orchestrates the full pipeline, you only need to call it once. This also means running locally exercises the exact same code path that Lambda will use in the cloud.

Before you run, update the RECIPIENT_EMAIL default value in the __main__ block to your actual verified email address (the one you just verified in SES).
Also update GEMINI_API_KEY with your real Gemini API key if you have not already set it as an environment variable.
Save lambda_function.py.

Before you run this, what do you expect to happen? Think about each stage: scraping, AI filtering, email building, and SES delivery.

Run the full pipeline:

python lambda_function.py

You should see output like this in your terminal:

Scraping opportunities...
Found 24 raw opportunities
Filtering with Gemini AI...
AI selected 7 relevant opportunities
Sending email via SES...
Email sent successfully!

Check your email inbox. You should find a formatted HTML email titled Your Daily Opportunity Digest with clickable opportunity cards, deadlines, and the AI's reasoning for each pick.

Email not arriving?

Check your spam folder. SES sandbox emails sometimes land there on first send.
Make sure the email in your __main__ block's RECIPIENT_EMAIL exactly matches the address you verified in SES (case-sensitive).
Confirm you are running in the us-east-1 region. The SES client in your code defaults to us-east-1, and your verified identity must be in the same region.
If you see MessageRejected in the terminal, your email is not verified yet. Go back to the SES console and confirm the status is Verified.

Help me troubleshoot why my SES email is not being delivered

✔️ Awesome, I've got everything!

Great. Double check you have saved lambda_function.py and your email arrived successfully.

ⓧ I'd like to double check the full code

Your lambda_function.py should now contain these sections in order. Scroll up and confirm each is present:

Imports (json, os, re, requests, BeautifulSoup, genai, boto3).
scrape_devpost() function.
scrape_mlh() function.
get_opportunities() function.
filter_with_ai() function.
build_html_email() function (cards logic + outer HTML template + return html).
send_email() function.
lambda_handler() function.
if __name__ == "__main__": block calling lambda_handler({}, None).

Your agent now scrapes, filters with AI, and delivers a polished email to your inbox. Next up, you will deploy this to AWS Lambda and schedule it to run every morning without any action on your part.

Deploy to AWS Lambda and Schedule It

Your agent works perfectly on your laptop. But it only runs when your laptop is on, when you remember to run it, and when you have internet.

Deploying to AWS Lambda with an EventBridge Scheduler schedule makes the agent truly autonomous. It runs whether you are awake, asleep, or on vacation.

In this step, get ready to:

Package your code and dependencies into a Lambda deployment zip.
Create and deploy the Lambda function with environment variables.
Schedule the function to run every morning at 7 AM.

Create the Lambda deployment package

Lambda needs all your Python dependencies bundled into a single zip file alongside your code. The python3.12 runtime includes boto3 natively, so you only need to package the other three libraries.

Install your dependencies into a local package/ directory by running this command:

pip install -t package/ google-generativeai requests beautifulsoup4

Copy your Lambda function into the package directory by running:

cp lambda_function.py package/

Zip the entire package into a deployment file by running:

cd package && zip -r ../deployment.zip . && cd ..

Confirm the zip was created by running:

ls deployment.zip

You should see deployment.zip listed in the output.

Zip command not found?

On Windows, use PowerShell's Compress-Archive instead. Or install a zip utility. On macOS and Linux, zip is available by default.

help me create a Lambda deployment zip on my operating system

Now you need an IAM role that gives Lambda permission to execute and send emails via SES.

Create a trust policy file by running:

cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

What does this file do?

This trust policy tells AWS that the Lambda service is allowed to assume (use) this role. Without it, Lambda cannot run your function.

Create the IAM execution role by running:

aws iam create-role \
  --role-name opportunity-agent-role \
  --assume-role-policy-document file://trust-policy.json

Copy the Role.Arn value from the output. You will need it in the next command.
Attach the basic Lambda execution policy by running:

aws iam attach-role-policy \
  --role-name opportunity-agent-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Create an inline policy that grants SES send permission by running:

aws iam put-role-policy \
  --role-name opportunity-agent-role \
  --policy-name SESSendEmail \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"ses:SendEmail","Resource":"*"}]}'

Why two separate policies?

The managed AWSLambdaBasicExecutionRole gives Lambda permission to write logs to CloudWatch. The inline SESSendEmail policy grants permission to send emails through SES. Your agent needs both.

Wait 10 seconds for IAM to propagate the role, then create the Lambda function by running:

aws lambda create-function \
  --function-name opportunity-agent \
  --runtime python3.12 \
  --handler lambda_function.lambda_handler \
  --role arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/opportunity-agent-role \
  --zip-file fileb://deployment.zip \
  --timeout 90 \
  --memory-size 256

What do these flags mean?

--handler lambda_function.lambda_handler tells Lambda which file and function to call when triggered.
--timeout 90 gives the function 90 seconds to scrape, filter, and email before Lambda stops it.
--memory-size 256 allocates 256 MB of RAM, which also increases CPU. Web scraping and AI calls need more than the 128 MB default.

You should see a JSON response with FunctionName showing opportunity-agent and State showing Active.

Seeing a role not found error?

IAM role propagation can take up to 15 seconds. Wait a moment and retry the create-function command.

help me troubleshoot my Lambda create-function command

Configure environment variables and test

Your function reads secrets from environment variables. You need to set your Gemini API key, email address, and interests JSON on the Lambda configuration.

Set the environment variables on your Lambda function by running:

aws lambda update-function-configuration \
  --function-name opportunity-agent \
  --environment 'Variables={GEMINI_API_KEY=your-gemini-api-key-here,RECIPIENT_EMAIL=your-email@example.com,INTERESTS_JSON="{\"topics\":[\"AI\",\"machine learning\",\"web development\",\"beginner-friendly\"],\"format_preference\":\"online\",\"min_prize_usd\":0,\"exclude_keywords\":[\"blockchain\",\"crypto\",\"web3\"],\"max_results\":10}"}'

What values do I enter?

Replace your-gemini-api-key-here with the actual Gemini API key you saved earlier. Replace your-email@example.com with the email address you verified in SES. The INTERESTS_JSON value is your interests profile as a JSON string.

Before you run the next command, what do you think will happen? Will the Lambda successfully scrape, filter, and send you an email?

Invoke the Lambda manually to test it by running:

aws lambda invoke \
  --function-name opportunity-agent \
  --cli-read-timeout 120 \
  response.json

Check the response by running:

cat response.json

You should see a JSON response with statusCode of 200 and a body mentioning how many opportunities were sent.

Check your email inbox. You should find a new opportunity digest email from your agent.

No email arrived?

Check that your RECIPIENT_EMAIL matches the verified SES identity exactly. Also check your spam folder. If response.json shows an error, the Gemini API key or SES permissions may be incorrect.

help me debug why my Lambda function is not sending the email

Create the EventBridge schedule

The final piece is a schedule that triggers your Lambda every morning at 7 AM. EventBridge Scheduler supports timezone-aware cron expressions, so the agent fires at 7 AM in your local time regardless of UTC offset.

First, you need an IAM role that gives EventBridge Scheduler permission to invoke your Lambda.

Create a trust policy file for the scheduler by running:

cat > scheduler-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "scheduler.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the scheduler execution role by running:

aws iam create-role \
  --role-name opportunity-agent-scheduler-role \
  --assume-role-policy-document file://scheduler-trust-policy.json

Attach an inline policy granting Lambda invoke permission by running:

aws iam put-role-policy \
  --role-name opportunity-agent-scheduler-role \
  --policy-name InvokeLambda \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"lambda:InvokeFunction","Resource":"arn:aws:lambda:us-east-1:'[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]':function:opportunity-agent"}]}'

Now create the schedule itself. The cron expression cron(0 7 * * ? *) means "at minute 0 of hour 7, every day."

Create the daily schedule by running:

aws scheduler create-schedule \
  --name opportunity-agent-daily \
  --schedule-expression "cron(0 7 * * ? *)" \
  --schedule-expression-timezone "[[TIMEZONE="your timezone e.g. America/New_York"]]" \
  --target '{"RoleArn":"arn:aws:iam::'[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]':role/opportunity-agent-scheduler-role","Arn":"arn:aws:lambda:us-east-1:'[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]':function:opportunity-agent"}' \
  --flexible-time-window '{"Mode":"OFF"}'

What timezone should I use?

Use the IANA timezone identifier for your location. Examples: America/New_York, America/Los_Angeles, Europe/London, Asia/Tokyo. This ensures the schedule fires at 7 AM in your local time, adjusting for daylight savings automatically.

You should see a response containing ScheduleArn confirming the schedule was created.

Verify the schedule is enabled by opening the AWS Console.
Navigate to Amazon EventBridge in the search bar.
Click Schedules in the left sidebar.
Confirm opportunity-agent-daily appears with status Enabled.

Why EventBridge Scheduler instead of EventBridge Rules?

AWS recommends EventBridge Scheduler over EventBridge Rules for scheduled invocations. Scheduler supports timezone-aware cron expressions, offers higher scale, and provides a richer feature set for scheduling targets.

Schedule not showing in the console?

Make sure you are viewing the us-east-1 region in the console. EventBridge Scheduler schedules are regional. Also confirm the create-schedule command completed without errors.

help me troubleshoot my EventBridge Scheduler schedule not appearing

Your agent is now fully autonomous. Tomorrow morning at 7 AM, it will wake up on its own, scrape hackathon listings, filter them with AI, and deliver a personalized digest to your inbox. All within AWS Free Tier at zero cost.

Secret mission

Never Show the Same Opportunity Twice

Your agent works great, but it has no memory. Every morning it sends the same opportunities you already saw yesterday. Add persistent memory with S3 so the agent tracks what it has already sent and never repeats itself.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely within AWS Free Tier limits for a once-daily agent, so there are no ongoing costs while it stays within those limits.

Resources you used:

1 Lambda function (opportunity-agent) on the AWS Free Tier (1,000,000 requests/month, 400,000 GB-seconds/month).
1 EventBridge Scheduler schedule (opportunity-agent-daily) on the AWS Free Tier (14,000,000 invocations/month).
1 SES verified identity (your email address) on the AWS Free Tier (3,000 messages/month for 12 months).
1 S3 bucket (opportunity-agent-history-{account-id}) for history storage on the AWS Free Tier (5 GB).
Local project folder (opportunity-agent) on your Desktop.

Keep everything running

No action needed. Choose this if you want the agent to keep running daily. You will receive one opportunity digest email per day at zero cost within Free Tier limits.

Your Lambda function continues to invoke once per day via the EventBridge schedule.
Your S3 history bucket continues to grow slowly as the agent tracks sent URLs.
Your local opportunity-agent folder remains on your Desktop for future edits.

Pause - I'll come back to this later

Disable the schedule so the agent stops running daily. All resources stay in place for when you return.

Open the EventBridge Scheduler console at https://console.aws.amazon.com/scheduler/.
Select your opportunity-agent-daily schedule.
Click Disable to stop future invocations.
To resume later, return to the same page and click Enable.

Delete - I don't want to use this again

Remove all project resources and start fresh. Run each of the following commands in your terminal.

Delete the Lambda function by running:

aws lambda delete-function --function-name opportunity-agent

Delete the EventBridge Scheduler schedule by running:

aws scheduler delete-schedule --name opportunity-agent-daily

Delete the S3 history bucket and all its contents by running:

aws s3 rb s3://[[HISTORY_BUCKET="opportunity-agent-history-your-account-id"]] --force

Delete the SES verified identity by running:

aws ses delete-identity --identity [[YOUR_EMAIL="your-email@example.com"]]

Delete your local project folder by running:

macOS

rm -rf ~/Desktop/opportunity-agent

Windows

Remove-Item -Recurse -Force "$env:USERPROFILE\Desktop\opportunity-agent"

That's a Wrap!

Nice work. You just built a fully autonomous AI agent that discovers, filters, and delivers personalized student opportunities to your inbox every single morning. No manual checking. No forgotten deadlines. Just relevant opportunities arriving while you sleep.

You've learned how to:

Build a web scraper that pulls live hackathon data from Devpost and MLH, then pipe that raw data through Google Gemini AI to intelligently filter and rank opportunities based on your personal interests.
Construct a styled HTML email digest and deliver it to your inbox via Amazon SES, completing an end-to-end serverless pipeline from data source to inbox.
Deploy a fully autonomous event-driven agent using AWS Lambda and EventBridge Scheduler that runs daily at zero cost on the AWS Free Tier.
(Secret Mission) Added persistent memory with Amazon S3 so the agent tracks what it already sent and never shows the same opportunity twice.

Ready to quiz yourself?

Build an Autonomous Student Agent

Introduction

30 Second Summary

What You'll Build

Are there any prerequisites?

Before We Start

Set Up Your Tools and Project

In this step, get ready to:

Install and configure the AWS CLI

✔️ I see a version number

ⓧ Command not found

macOS

Windows

What will this command ask?

Find your access keys

Seeing an error instead of your account info?

Get your Gemini API key and install packages

Why is the Gemini API free?

Cannot create an API key?

What are these packages?

Seeing a ModuleNotFoundError?

What does this file do?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Scrape Live Opportunities

In this step, get ready to:

Build the Devpost scraper

What do these imports do?

What does this parsing loop do?

Seeing 0 hackathons or an error?

Build the MLH scraper

What does this function do?

Getting 0 events or a connection error?

Wire both scrapers together and run

What does this code do?

Seeing an empty list or an error?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Add AI-Powered Filtering

In this step, get ready to:

Create your interests profile

What does this file do?

Seeing an empty file or an error?

Build the AI filter function

What does this prompt do?

Why strip markdown fences?

Seeing a SyntaxError or IndentationError?

Test the full pipeline locally

What changed?

Getting an API error or empty results?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Build and Send the Email Digest

In this step, get ready to:

Build the opportunity card template

What does this code do?

What does the template do?

Seeing a SyntaxError?

Set up SES and verify your email

What happens next?

Still showing Pending?

Add the SES sending function and wire up the pipeline

What does this function do?

What does lambda_handler do?

Why change the __main__ block?

Email not arriving?

✔️ Awesome, I've got everything!

ⓧ I'd like to double check the full code

Deploy to AWS Lambda and Schedule It

In this step, get ready to:

Create the Lambda deployment package

Zip command not found?

What does this file do?

Why two separate policies?

What do these flags mean?

Seeing a role not found error?

Configure environment variables and test

What values do I enter?

No email arrived?

Create the EventBridge schedule

Why change the main block?