Build a Persistent Web-Surfing Agent

Build a local AI agent that surfs the web on a schedule and saves history.

Introduction

30 Second Summary

You check the same websites every day for updates. News sites, weather pages, project repositories. It takes time, and you always forget at least one.

In this project, you will build a persistent agent that surfs the web on autopilot, summarizes what it finds using a local AI model, and remembers everything across restarts.

What You'll Build

You start the agent, walk away, and come back to find timestamped AI-generated digests of the web pages you care about, ready to read in your terminal or query from a database.

By the end of this project, you'll have:

  • A daemon-style agent that fetches Hacker News headlines and produces AI summaries on a configurable timer, with zero API costs.
  • Full life-cycle control over the agent (pause, resume, check status, quit) without killing the process.
  • A SQLite-backed history of every routine run, persisting across restarts so the agent never loses its memory.
  • Secret Mission: A change detection routine that watches a page and alerts you when new content appears.

Are there any prerequisites?

You should be comfortable writing Python (functions, classes, dicts). No prior experience with Ollama, scheduling libraries, or agents is required.

This project runs on Linux with Python 3 and at least 32 GB of RAM (needed to run a local LLM).

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This step is your moment to commit to the goal before any hands-on work begins.

Set Up Your Environment

Your web-surfing agent needs two things to function: a local AI model for summarization, and Python libraries for web scraping and scheduling. Without these in place, the agent has nothing to think with and no way to reach the internet.

In this step, you'll install Ollama (the tool that runs AI models locally), pull the Qwen 3 model, and set up a Python project with all the dependencies your agent needs.

In this step, get ready to:
  • Install Ollama and pull the Qwen 3 model.
  • Create a project directory and virtual environment.
  • Install Python dependencies.
Install Ollama and pull the Qwen 3 model

Ollama lets you run large language models on your own machine with zero API costs. You'll install it, then download the Qwen 3 model that your agent will use for summarization.

  • Install Ollama by running this command:
curl -fsSL https://ollama.com/install.sh | sh

What does this command do?

The install script downloads the Ollama binary, places it in your system path, and sets up a systemd service that starts Ollama automatically. After it finishes, the Ollama server is already running in the background.

Seeing a permission error?

If the install fails with a permissions error, try prefixing with sudo: curl -fsSL https://ollama.com/install.sh | sudo sh.

help me fix Ollama installation permissions on Linux

  • Pull the Qwen 3 model by running:
ollama pull qwen3

This downloads approximately 5GB of model data. You'll see a progress bar as the layers download.

Seeing a connection refused error?

If you see connection refused, the Ollama server did not auto-start. Open a second terminal and run:

ollama serve

Leave that terminal running and retry ollama pull qwen3 in your original terminal.

help me get ollama serve running on Linux

  • Before you run the next command, what do you expect to see if Qwen 3 is working correctly?
  • Verify the model works by running:
ollama run qwen3 "Say hello in one word"

You should see a short response from the model (something like "Hello" or "Hi"). The exact wording will vary each time.

What is Qwen 3?

Qwen 3 is a locally-runnable language model with strong summarization and instruction-following abilities. Because it runs entirely on your machine, your agent can call it with zero API costs and no internet dependency.

Create your project directory and virtual environment
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir web-agent
cd web-agent
  • Create a Python virtual environment and activate it by running:
python3 -m venv venv
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning. This confirms you're working inside the virtual environment.

Why use a virtual environment?

A virtual environment isolates your project's packages from the rest of your system. This prevents version conflicts between projects and keeps your agent's dependencies self-contained.

Don't see (venv) in your prompt?

Make sure you ran source venv/bin/activate (not just the python3 -m venv venv part). If your shell is fish or csh, use source venv/bin/activate.fish or source venv/bin/activate.csh instead.

help me activate my Python virtual environment

Install Python dependencies

Your agent needs four packages: one for talking to Ollama, one for scheduling, one for HTTP requests, and one for parsing HTML.

  • Create a file called requirements.txt in your web-agent folder. Open it in your code editor and paste the following:
ollama==0.6.2
APScheduler==3.11.3
httpx==0.28.1
beautifulsoup4==4.15.0

What are these packages?

  • ollama is the Python SDK for communicating with your local Ollama server.
  • APScheduler handles time-based job scheduling so your agent can run routines automatically.
  • httpx is a modern HTTP client for fetching web pages.
  • beautifulsoup4 parses HTML so your agent can extract useful content from raw pages.
  • Save requirements.txt.
  • Install all the dependencies by running:
pip install -r requirements.txt

You'll see pip download and install each package along with their sub-dependencies.

Seeing install errors?

Make sure your virtual environment is active (you should see (venv) in your prompt). If you accidentally closed your terminal, navigate back to ~/Desktop/web-agent and run source venv/bin/activate again before retrying.

help me fix pip install errors in my virtual environment

  • Before you run the next command, what do you expect to see if all four packages installed correctly?
  • Verify everything is working by running:
python -c "import ollama, apscheduler, httpx, bs4; print('ready')"

You should see ready printed to your terminal. This confirms all four packages are importable from your virtual environment.

Seeing a ModuleNotFoundError?

Double-check that you ran pip install -r requirements.txt while (venv) was active. If you see an error for a specific package, try installing it individually: pip install ollama==0.6.2.

help me fix ModuleNotFoundError after pip install

✔️ Awesome, I've got everything!

Great. Double check you've saved requirements.txt and that your verify command printed ready.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file. Make sure yours matches exactly:

ollama==0.6.2
APScheduler==3.11.3
httpx==0.28.1
beautifulsoup4==4.15.0

Your environment is ready. Next up, you'll build the web-surfing toolkit that lets your agent fetch and parse live web pages.

Build the Web Surfer

Your environment is locked and loaded. Ollama is running, Qwen 3 is pulled, and your Python packages are installed.

Now the agent needs its first real capability: reaching out to the internet and bringing back structured data. In this step, you will build the web-fetching layer that grabs live pages and extracts useful content from them.

In this step, get ready to:
  • Build a page-fetching function that strips away HTML noise and returns clean text.
  • Build a headline-extraction function that pulls meaningful links from any page.
  • Fetch live Hacker News headlines and see them printed in your terminal.
Create web_tools.py with fetch functions

Your agent needs two tools to surf the web. One fetches a full page and cleans the HTML down to readable text. The other extracts just the headline links from a page.

  • Create a new file called web_tools.py in your web-agent folder.
  • Add the imports and the fetch_page function by pasting the following code:
import httpx
from bs4 import BeautifulSoup


def fetch_page(url: str) -> dict:
    """Fetch a web page and return cleaned text content."""
    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
    }
    response = httpx.get(url, headers=headers, timeout=15.0, follow_redirects=True)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    for element in soup(["script", "style", "nav", "footer", "header"]):
        element.decompose()

    title = soup.title.string if soup.title else "No title"
    text = soup.get_text(separator="\n", strip=True)
    text = text[:3000]

    return {"title": title, "url": url, "content": text}

What does this code do?

This function is your agent's "eyes" for reading web pages.

  • It sends a request with a browser-like User-Agent header so websites don't block it as a bot.
  • After fetching the HTML, it strips out noisy elements (scripts, styles, navigation) that would pollute a summary.
  • It truncates the remaining text to 3000 characters so the LLM gets a focused, manageable chunk.
  • Save web_tools.py.
  • Verify that fetch_page works by running this quick test:
python -c "from web_tools import fetch_page; print(fetch_page('https://wttr.in/?format=3')['content'][:100])"

You should see a short weather string printed to your terminal.

Seeing a connection error?

  • Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
  • Check that you have an internet connection. The function is reaching out to a live URL.
  • If you see ModuleNotFoundError, re-run pip install -r requirements.txt inside your activated venv.

Help me debug my fetch_page function

  • Now add the fetch_headlines function below fetch_page in web_tools.py:

def fetch_headlines(url: str) -> list[dict]:
    """Fetch a page and extract meaningful links with their text."""
    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
    }
    response = httpx.get(url, headers=headers, timeout=15.0, follow_redirects=True)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    headlines = []
    for link in soup.find_all("a", href=True):
        text = link.get_text(strip=True)
        if len(text) > 10:
            headlines.append({"text": text, "href": link["href"]})

    return headlines[:20]

What does this code do?

  • Instead of grabbing all text, this function focuses on extracting links with their anchor text.
  • The len(text) > 10 filter skips tiny navigation links (like "More" or "Login") and keeps only meaningful headlines.
  • It caps at 20 results so the LLM receives a focused list rather than hundreds of links.
  • Save web_tools.py.
Add a test block that fetches Hacker News headlines

Your two fetch functions are ready. Now add a test harness that proves they work against a real website.

  • Add the following __main__ block at the bottom of web_tools.py:

if __name__ == "__main__":
    print("Fetching Hacker News headlines...\n")
    results = fetch_headlines("https://news.ycombinator.com")
    for item in results:
        print(f"  - {item['text']}")
    print(f"\nFetched {len(results)} headlines.")

What does this code do?

The if __name__ == "__main__" guard means this block only runs when you execute the file directly. When other files import web_tools later, the test block stays silent.

  • Save web_tools.py.

Before you run this, what do you expect to see? Think about what Hacker News looks like and how many links the function should return.

  • Run the script by executing:
python web_tools.py

You should see 15-20 real Hacker News headlines printed to your terminal, each prefixed with a bullet point.

Not seeing any headlines?

  • If you see 0 headlines, Hacker News may be temporarily unreachable. Try again in a few seconds.
  • If you see a NameError on fetch_headlines, make sure your __main__ block is below the function definition, not above it.
  • If you see an IndentationError, check that the if __name__ line starts at column 0, not indented inside the function.

Help me debug my web_tools.py script

✔️ Awesome, I've got everything!

Great. Double check you have saved web_tools.py and confirmed the headlines printed.

ⓧ I'd like to double check the full code

Scroll up and confirm your web_tools.py contains these sections in order:

  • The import httpx and from bs4 import BeautifulSoup lines at the top.
  • The fetch_page function that strips noisy HTML elements and returns a dict.
  • The fetch_headlines function that extracts links with text longer than 10 characters.
  • The if __name__ == "__main__" block that fetches and prints Hacker News headlines.

Your agent can now reach out to any URL and bring back structured, readable content. Next up, you will connect these web tools to the LLM so the agent can summarize what it finds.

Add LLM Summarization

In the last step you proved your agent can reach the internet and bring back structured data. But raw headlines are just a list of links.

The agent becomes genuinely useful when it can read that content and produce a human-friendly summary. In this step you will create a "routines" system that pairs web fetching with LLM summarization, so one command produces a digest of what is happening on Hacker News right now.

In this step, get ready to:
  • Define reusable routines that pair a URL with a prompt template.
  • Wire up the Ollama Python SDK to send fetched content to Qwen 3 for summarization.
  • Run the full fetch-and-summarize pipeline end to end.
Create routines.py with the first routine

A routine is a reusable recipe that tells the agent what URL to fetch, which tool to use, and what prompt to send to the LLM. You will store these as a dictionary so the agent can look them up by ID later.

  • Create a new file called routines.py inside your web-agent folder.
  • Add the imports and the first routine definition by pasting this code:
from ollama import chat
from web_tools import fetch_page, fetch_headlines


ROUTINES = {
    "hn_digest": {
        "name": "Hacker News Digest",
        "description": "Fetch top stories from Hacker News and summarize them",
        "url": "https://news.ycombinator.com",
        "interval_seconds": 60,
        "tool": "fetch_headlines",
        "prompt": (
            "Here are the current top headlines from Hacker News:\n\n"
            "{content}\n\n"
            "Summarize the top 5 most interesting stories in 2-3 sentences each. "
            "Focus on tech trends."
        ),
    },
}

What does this code do?

The ROUTINES dictionary stores everything the agent needs to execute a task. Each entry has:

  • A target url to fetch content from.
  • A tool field that picks which web function to call (fetch_headlines or fetch_page).
  • A prompt template with a {content} placeholder where fetched data gets inserted before sending to the LLM.
  • An interval_seconds value the scheduler will use in a later step.
  • Save routines.py.
Add the second routine and the execution function

One routine is a start, but the system is designed to hold many. You will add a weather routine and then build the function that actually runs any routine by its ID.

  • In routines.py, find the closing } of the ROUTINES dictionary (the last line of the file).
  • Replace that closing } with the following code to add a second routine and the execution function:
    "weather_check": {
        "name": "Weather Summary",
        "description": "Check current weather conditions",
        "url": "https://wttr.in/?format=3",
        "interval_seconds": 120,
        "tool": "fetch_page",
        "prompt": (
            "Here is the current weather information:\n\n"
            "{content}\n\n"
            "Give a brief, friendly weather summary in one sentence."
        ),
    },
}


def execute_routine(routine_id: str) -> str:
    """Execute a routine's web fetching step and return raw content."""
    routine = ROUTINES[routine_id]

    if routine["tool"] == "fetch_headlines":
        results = fetch_headlines(routine["url"])
        content = "\n".join(f"- {item['text']}" for item in results)
    else:
        result = fetch_page(routine["url"])
        content = result["content"]

    return content

What does this code do?

  • The weather_check routine targets wttr.in, a terminal-friendly weather service, and uses fetch_page instead of fetch_headlines since the content is plain text, not a list of links.
  • execute_routine(routine_id) looks up a routine by ID, picks the right fetch tool based on the tool field, and returns the raw content as a string ready for the LLM.
  • Save routines.py.
  • Confirm the routine execution works by running this command:
python -c "from routines import execute_routine; content = execute_routine('hn_digest'); print(f'Fetched {len(content)} characters')"

You should see a message like Fetched 487 characters (the exact number will vary depending on the current headlines).

Seeing an ImportError?

  • Make sure you are running the command from inside the web-agent folder where both routines.py and web_tools.py live.
  • Confirm your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.

Help me debug my ImportError in routines.py

Add LLM summarization and test the full pipeline

The fetch step works. Now you will wire it into Ollama so the agent sends that content to Qwen 3 and gets back a human-readable summary.

  • At the bottom of routines.py, add the following test block that runs the full pipeline:
if __name__ == "__main__":
    print("Testing full pipeline: fetch + summarize\n")
    content = execute_routine("hn_digest")
    prompt = ROUTINES["hn_digest"]["prompt"].format(content=content)

    print("Sending to Qwen 3 for summarization...\n")
    response = chat(
        model="qwen3",
        messages=[{"role": "user", "content": prompt}],
        options={"temperature": 0.3},
    )
    print(response.message.content)

What does this code do?

  • execute_routine("hn_digest") fetches the current Hacker News headlines using the web tools you built in the previous step.
  • The .format(content=content) call injects the fetched headlines into the prompt template's {content} placeholder.
  • chat() from the ollama package sends the prompt to your local Qwen 3 model. The temperature: 0.3 keeps the summary focused and factual.
  • Save routines.py.

Before you run this, what do you expect to see? The script fetches live headlines and sends them to an AI model. Think about what a "summary of Hacker News" might look like.

  • Run the full pipeline by executing:
python routines.py

You should see Qwen 3 produce a 3-5 sentence summary of the current Hacker News top stories. The output will describe real headlines and tech trends from the live page.

Why does it take a few seconds?

Ollama loads the model into memory on the first call, which can take 5-15 seconds. Subsequent calls in the same session are faster because the model stays loaded.

Seeing a connection error?

  • Make sure Ollama is still running. Open another terminal and run ollama list to confirm it responds.
  • If Ollama stopped, restart it with ollama serve in a separate terminal, then try again.
  • If you see a model-not-found error, re-pull the model with ollama pull qwen3.

Help me fix my Ollama connection error when running routines.py

✔️ Awesome, I've got everything!

Great. Double check you have saved routines.py and move on to the next step.

ⓧ I'd like to double check the full code

Your routines.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports: from ollama import chat and from web_tools import fetch_page, fetch_headlines.
  • ROUTINES dict: two entries, "hn_digest" and "weather_check", each with name, description, url, interval_seconds, tool, and prompt fields.
  • execute_routine function: takes a routine_id string, calls the right fetch tool, and returns raw content.
  • __main__ block: calls execute_routine("hn_digest"), formats the prompt, sends it to chat() with model "qwen3", and prints the response.

Your agent can now fetch live web content and produce AI-powered summaries. Next up, you will add a scheduler so routines fire automatically on a timer, turning your script into a proper autonomous agent.

Add the Scheduler

Your fetch-and-summarize pipeline works, but you have to run the script manually every single time you want a digest. That defeats the purpose of an autonomous agent.

APScheduler lets you define time intervals for each routine, and the agent fires them automatically in the background. In this step, you will build the agent's scheduling engine and watch it run on its own.

In this step, get ready to:
  • Create an Agent class with a BackgroundScheduler that fires routines on intervals.
  • Add a main loop that keeps the agent alive and runs routines automatically.
  • Observe the blank-slate problem when you kill and restart the agent.
Create agent.py with the Agent class

The agent needs a central orchestrator that owns the scheduler and knows how to execute routines. You will build this as a class so all state lives in one place.

  • Create a new file called agent.py in your web-agent folder.
  • Add the imports and the Agent class with its __init__ method by pasting this code:
from datetime import datetime

from apscheduler.schedulers.background import BackgroundScheduler
from ollama import chat

from routines import ROUTINES, execute_routine


class Agent:
    def __init__(self):
        self.scheduler = BackgroundScheduler()
        self.paused = False
        self.running = False

What does this code do?

  • BackgroundScheduler runs jobs in a background thread, leaving the main thread free for user interaction.
  • self.paused and self.running are flags that control whether routines execute and whether the main loop continues.
  • The imports pull in ROUTINES and execute_routine from the module you built in the previous step.
  • Now add the _run_routine method below the __init__ method. This is the function the scheduler calls each time a routine is due:
    def _run_routine(self, routine_id: str):
        """Execute a single routine: fetch web content, summarize with LLM, print."""
        if self.paused:
            return

        routine = ROUTINES[routine_id]
        timestamp = datetime.now().strftime("%H:%M:%S")
        print(f"\n  [{timestamp}] Running: {routine['name']}")

        try:
            raw_content = execute_routine(routine_id)
            print(f"  Fetched {len(raw_content)} chars from {routine['url']}")

            prompt = routine["prompt"].format(content=raw_content)
            response = chat(
                model="qwen3",
                messages=[{"role": "user", "content": prompt}],
                options={"temperature": 0.3},
            )
            summary = response.message.content
            print(f"  Summary: {summary[:200]}...")

        except Exception as e:
            print(f"  [ERROR] {routine_id}: {e}")

What does this method do?

  • It checks the paused flag first. If the agent is paused, it returns immediately without doing anything.
  • It calls execute_routine to fetch web content, then formats the routine's prompt template with that content.
  • It sends the prompt to Qwen 3 via ollama.chat and prints the first 200 characters of the summary.
  • The try/except block catches network errors or LLM failures so one bad routine does not crash the entire agent.
  • Save agent.py.

Seeing a syntax error on save?

  • Make sure _run_routine is indented one level inside the Agent class (4 spaces from the left margin).
  • Check that the try: block and its except are at the same indentation level.
  • Help me fix a syntax error in my Agent class.
Add the main loop and run the agent

The scheduler runs in a background thread, but the main thread needs to stay alive. Without a loop, the script would exit immediately and the scheduler would die with it.

  • Add the start() method below _run_routine. This registers each routine with the scheduler and starts it:
    def start(self):
        """Start the agent and schedule all routines."""
        print("Starting agent...")
        self.running = True

        for routine_id, routine in ROUTINES.items():
            self.scheduler.add_job(
                self._run_routine,
                "interval",
                seconds=routine["interval_seconds"],
                args=[routine_id],
                id=routine_id,
                name=routine["name"],
            )
            print(f"  Scheduled: {routine['name']} (every {routine['interval_seconds']}s)")

        self.scheduler.start()

What does this code do?

  • scheduler.add_job registers each routine with an "interval" trigger. The scheduler will call _run_routine every interval_seconds seconds (60s for Hacker News, 120s for weather).
  • scheduler.start() launches the background thread that watches the clock and fires jobs when they are due.
  • Now add the initial pass, the keep-alive loop, and the entry point directly below the self.scheduler.start() line (still inside the start() method), followed by the __main__ block outside the class:
        print("\nRunning initial pass...")
        for routine_id in ROUTINES:
            self._run_routine(routine_id)

        print("\nAgent running. Press Ctrl+C to stop.")

        try:
            while self.running:
                import time
                time.sleep(1)
        except KeyboardInterrupt:
            print("\n  Shutting down...")
            self.scheduler.shutdown(wait=False)
            print("  Agent stopped.")


if __name__ == "__main__":
    agent = Agent()
    agent.start()

What does this code do?

  • The initial pass runs every routine once immediately so you see output right away instead of waiting for the first interval to elapse.
  • The while self.running loop with time.sleep(1) keeps the main thread alive. Without it, the script exits and the background scheduler dies.
  • KeyboardInterrupt catches Ctrl+C and calls scheduler.shutdown(wait=False) for a clean exit.
  • Save agent.py.

Before you run the agent, what do you expect to see in the terminal? Think about what the initial pass does and what the scheduler prints.

  • Run the agent by executing:
python agent.py

You should see "Starting agent..." followed by scheduling confirmations for both routines, then the initial pass fetches Hacker News and weather data and prints AI summaries. The agent then sits idle, printing "Agent running. Press Ctrl+C to stop."

Why does it take a moment?

Each routine fetches a live web page and then sends the content to Qwen 3 for summarization. The LLM inference takes a few seconds per routine depending on your hardware.

  • Wait 60 seconds. You should see the scheduler fire the hn_digest routine again automatically, printing a fresh summary without any action from you.

Agent exits immediately or no output appears?

  • Make sure your virtual environment is activated. Run source venv/bin/activate if you see a ModuleNotFoundError.
  • Confirm Ollama is still running in another terminal. Run ollama list to check.
  • Check that the while self.running: loop is inside the start() method (indented 8 spaces from the left margin).
  • Help me debug why my agent exits immediately.

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and the agent is running.

ⓧ I'd like to double check the full code

Scroll up and confirm your agent.py contains these sections in order:

  • Imports (datetime, BackgroundScheduler, chat, ROUTINES, execute_routine).
  • Agent class with __init__ setting up scheduler, paused, and running.
  • _run_routine method that fetches content, calls Ollama, and prints the summary.
  • start() method that registers jobs, starts the scheduler, runs the initial pass, and enters the while loop.
  • if __name__ == "__main__": block that creates and starts the agent.
Observe the blank-slate problem

The agent runs and fires routines on schedule. But what happens to all that work if you stop it?

  • Let the agent run for at least one scheduled cycle (you saw it fire automatically after 60 seconds).
  • Press Ctrl+C to stop the agent. You should see "Shutting down..." and "Agent stopped."

Before you restart, think about this: does the agent have any way to remember what it just did? Any file, database, or log that records the summaries it produced?

  • Restart the agent by running:
python agent.py

The agent starts completely fresh. There is no record of the previous run. No history, no stored summaries, no indication it ever ran before. Every restart is a blank slate.

Why is this a problem?

An agent that forgets everything on restart is unreliable. You cannot review past summaries, track trends over time, or even confirm it ran while you were away.

A proper long-running agent needs persistence. It needs a database that survives process restarts and life-cycle controls that let you pause, resume, and check history without killing the process.

  • Press Ctrl+C to stop the agent again.

You have proved the agent can fetch, summarize, and schedule autonomously. The blank-slate problem is the gap that makes it unreliable. Next up, you will add SQLite persistence and interactive life-cycle controls so the agent remembers everything across restarts.

Add Persistence and Life-Cycle Control

In the last step you saw the blank-slate problem firsthand: killing and restarting the agent wiped all memory of previous runs. That makes the agent unreliable as a long-running service.

In this step you will solve both problems at once. SQLite gives the agent persistent memory across restarts, and an interactive command loop gives you life-cycle control (pause, resume, quit) without killing the process.

In this step, get ready to:
  • Create a persistence layer that stores every routine run in a local database.
  • Wire the agent to save results and state after each execution.
  • Replace the sleep loop with interactive commands so you can pause, resume, and inspect history without killing the process.
Create persistence.py with SQLite storage

Python ships with sqlite3 in the standard library, so there is nothing extra to install. You will create a new file that handles all database operations for the agent.

  • Create a new file called persistence.py in your web-agent folder and add the database setup code:
import sqlite3
from datetime import datetime
from pathlib import Path

DB_PATH = Path("agent_state.db")

def init_db():
    """Initialize the SQLite database with required tables."""
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()

    cursor.execute("""
        CREATE TABLE IF NOT EXISTS run_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            routine_id TEXT NOT NULL,
            timestamp TEXT NOT NULL,
            raw_content TEXT,
            summary TEXT,
            status TEXT NOT NULL
        )
    """)

    cursor.execute("""
        CREATE TABLE IF NOT EXISTS agent_state (
            key TEXT PRIMARY KEY,
            value TEXT NOT NULL
        )
    """)
    conn.commit()
    conn.close()

What does this code do?

DB_PATH points to a file called agent_state.db that SQLite creates automatically when you first connect to it.

init_db() creates two tables: run_history stores every routine execution (what was fetched, what the LLM said, whether it succeeded), and agent_state stores key-value pairs like the agent's current status.

  • Below init_db(), add the function that writes a routine run to the database:
def save_run(routine_id: str, raw_content: str, summary: str, status: str):
    """Save a routine run to the database."""
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()

    cursor.execute(
        "INSERT INTO run_history (routine_id, timestamp, raw_content, summary, status) "
        "VALUES (?, ?, ?, ?, ?)",
        (routine_id, datetime.now().isoformat(), raw_content, summary, status),
    )

    conn.commit()
    conn.close()

What does this code do?

save_run() inserts one row into run_history each time a routine executes. It records the routine ID, a timestamp, the raw fetched content, the LLM summary, and whether it succeeded or errored.

  • Verify the database setup works by running this quick test:
python -c "from persistence import init_db, save_run; init_db(); save_run('test', 'hello', 'world', 'success'); print('DB created and write confirmed')"

You should see DB created and write confirmed printed, and a new file agent_state.db in your folder.

  • Confirm the file exists by running:
ls agent_state.db

You should see agent_state.db listed.

Don't see the file?

Make sure your terminal is in the web-agent directory and your virtual environment is activated. If you see a ModuleNotFoundError, confirm persistence.py is saved in the same folder where you ran the command.

Why can't I import persistence?

  • Delete the test database so you start fresh (the agent will recreate it):
rm agent_state.db
  • Back in persistence.py, add the function to read run history below save_run():
def get_history(routine_id: str = None, limit: int = 10) -> list[dict]:
    """Get run history, optionally filtered by routine."""
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    cursor = conn.cursor()

    if routine_id:
        cursor.execute(
            "SELECT * FROM run_history WHERE routine_id = ? ORDER BY timestamp DESC LIMIT ?",
            (routine_id, limit),
        )
    else:
        cursor.execute(
            "SELECT * FROM run_history ORDER BY timestamp DESC LIMIT ?",
            (limit,),
        )

    rows = [dict(row) for row in cursor.fetchall()]
    conn.close()
    return rows

What does this code do?

get_history() retrieves past runs ordered by most recent first. You can filter by routine or grab everything. Setting conn.row_factory = sqlite3.Row lets you access columns by name instead of index.

  • Below get_history(), add the function to retrieve the last successful content for a routine:
def get_last_content(routine_id: str) -> str | None:
    """Get the raw content from the last successful run of a routine."""
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()
    cursor.execute(
        "SELECT raw_content FROM run_history WHERE routine_id = ? AND status = 'success' "
        "ORDER BY timestamp DESC LIMIT 1",
        (routine_id,),
    )
    row = cursor.fetchone()
    conn.close()
    return row[0] if row else None

What does this code do?

get_last_content() retrieves the raw content from the most recent successful run of a routine. This is useful for comparing current content against past content (which you will use in the Secret Mission).

  • Finally, add the state management functions below get_last_content():
def save_state(key: str, value: str):
    """Save agent state as key-value pair."""
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()
    cursor.execute(
        "INSERT OR REPLACE INTO agent_state (key, value) VALUES (?, ?)",
        (key, value),
    )
    conn.commit()
    conn.close()

def get_state(key: str) -> str | None:
    """Get an agent state value by key."""
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()
    cursor.execute("SELECT value FROM agent_state WHERE key = ?", (key,))
    row = cursor.fetchone()
    conn.close()
    return row[0] if row else None

What do these functions do?

  • save_state() uses INSERT OR REPLACE to upsert a key-value pair into agent_state. This stores the agent's current lifecycle status.
  • get_state() reads a state value back by key.
  • Save persistence.py.

✔️ Awesome, I've got everything!

Great. Confirm you have saved persistence.py with all six functions: init_db, save_run, get_history, get_last_content, save_state, and get_state.

ⓧ I'd like to double check the full code

Scroll up through your file and confirm each section is present in this order:

  • Imports (sqlite3, datetime, Path) and DB_PATH constant.
  • init_db() creating both tables.
  • save_run() inserting into run_history.
  • get_history() with the optional routine_id filter.
  • get_last_content() querying the last successful run.
  • save_state() and get_state() for key-value state.
Update agent.py with persistence and interactive commands

Now you will wire the persistence layer into the agent so every execution is recorded and the agent state survives restarts.

  • In agent.py, find this import line:
from routines import ROUTINES, execute_routine
  • Add a new import line directly above it, so the two lines look like this:
from persistence import init_db, save_run, get_history, save_state
from routines import ROUTINES, execute_routine
  • In the __init__ method, find this line:
        self.running = False
  • Add init_db() on a new line directly below it, so __init__ now looks like this:
    def __init__(self):
        self.scheduler = BackgroundScheduler()
        self.paused = False
        self.running = False
        init_db()
  • Next, update _run_routine to save results after each execution. Replace the entire _run_routine method with this version:
    def _run_routine(self, routine_id: str):
        """Execute a single routine: fetch web content, summarize with LLM, save."""
        if self.paused:
            return

        routine = ROUTINES[routine_id]
        timestamp = datetime.now().strftime("%H:%M:%S")
        print(f"\n  [{timestamp}] Running: {routine['name']}")

        try:
            raw_content = execute_routine(routine_id)
            print(f"  Fetched {len(raw_content)} chars from {routine['url']}")

            prompt = routine["prompt"].format(content=raw_content)
            response = chat(
                model="qwen3",
                messages=[{"role": "user", "content": prompt}],
                options={"temperature": 0.3},
            )
            summary = response.message.content
            print(f"  Summary: {summary[:200]}...")

            save_run(routine_id, raw_content, summary, "success")

        except Exception as e:
            print(f"  [ERROR] {routine_id}: {e}")
            save_run(routine_id, "", str(e), "error")

What changed?

Two save_run() calls were added. The first stores successful runs with the raw content and summary. The second stores errors so you have a record of failures too.

  • Save agent.py and run a quick test to confirm persistence is working:
python agent.py

Let the initial pass complete (you will see summaries print), then press Ctrl+C to stop.

  • Query the database to confirm runs were saved:
python -c "from persistence import get_history; rows = get_history(limit=3); [print(f\"  {r['routine_id']}: {r['status']}\") for r in rows]"

You should see your recent routine runs listed with their status.

See an empty list or import error?

If the list is empty, make sure you let the agent complete at least one routine before pressing Ctrl+C. If you see an import error, confirm persistence.py is in the same directory as agent.py.

Why is my run history empty?

Now you will replace the simple sleep loop with an interactive command system.

  • In agent.py, find the start() method. Locate the line that says:
        self.running = True
  • Add these two lines directly below it:
        save_state("status", "running")
        save_state("started_at", datetime.now().isoformat())
  • Now find and delete the old loop at the bottom of start(). It currently looks like this:
        print("\nAgent running. Press Ctrl+C to stop.")

        try:
            while self.running:
                import time
                time.sleep(1)
        except KeyboardInterrupt:
            print("\n  Shutting down...")
            self.scheduler.shutdown(wait=False)
            print("  Agent stopped.")
  • Replace it with the interactive command loop:
        print("\nAgent running. Commands: [p]ause  [r]esume  [h]istory  [s]tatus  [q]uit")

        try:
            while self.running:
                cmd = input("\n> ").strip().lower()

                if cmd == "p":
                    self.pause()
                elif cmd == "r":
                    self.resume()
                elif cmd == "h":
                    self.show_history()
                elif cmd == "s":
                    self.show_status()
                elif cmd == "q":
                    self.stop()
                else:
                    print("  Commands: [p]ause  [r]esume  [h]istory  [s]tatus  [q]uit")
        except KeyboardInterrupt:
            self.stop()

What changed here?

Instead of sleeping forever and relying on Ctrl+C, the agent now waits for your input at a > prompt. Each single-letter command maps to a lifecycle method you will add next.

  • Below the start() method, add the three core lifecycle methods:
    def pause(self):
        """Pause all routines without stopping the agent."""
        self.paused = True
        self.scheduler.pause()
        save_state("status", "paused")
        print("  Agent PAUSED. Routines will not execute until resumed.")

    def resume(self):
        """Resume paused routines."""
        self.paused = False
        self.scheduler.resume()
        save_state("status", "running")
        print("  Agent RESUMED. Routines will execute on schedule.")

    def stop(self):
        """Stop the agent gracefully."""
        print("  Shutting down...")
        self.running = False
        self.scheduler.shutdown(wait=False)
        save_state("status", "stopped")
        save_state("stopped_at", datetime.now().isoformat())
        print("  Agent stopped. State saved to agent_state.db")

What do these methods do?

  • pause() sets a flag so _run_routine returns early, and calls scheduler.pause() to stop APScheduler from waking up.
  • resume() clears the flag and calls scheduler.resume() to restart job processing.
  • stop() shuts down the scheduler, saves the final state, and exits the command loop by setting self.running = False.
  • Below stop(), add the two display methods:
    def show_history(self):
        """Display recent run history."""
        history = get_history(limit=5)
        if not history:
            print("  No history yet.")
            return

        print("  Recent runs:")
        for run in history:
            timestamp = run["timestamp"][:19]
            status = run["status"]
            summary_preview = (run["summary"] or "")[:80]
            print(f"    [{timestamp}] {run['routine_id']} ({status}): {summary_preview}...")

    def show_status(self):
        """Display current agent status."""
        status = "PAUSED" if self.paused else "RUNNING"
        jobs = self.scheduler.get_jobs()
        print(f"  Status: {status}")
        print(f"  Active routines: {len(jobs)}")
        for job in jobs:
            next_run = job.next_run_time
            next_str = next_run.strftime("%H:%M:%S") if next_run else "paused"
            print(f"    - {job.name}: next run at {next_str}")

What do these methods do?

  • show_history() pulls the 5 most recent runs from SQLite and prints a compact summary of each.
  • show_status() reports whether the agent is running or paused, and lists each scheduled routine with its next fire time.
  • Save agent.py.

✔️ Awesome, I've got everything!

Great. Confirm you have saved agent.py with the updated imports, init_db() in __init__, save_run calls in _run_routine, the interactive command loop in start(), and all five new methods.

ⓧ I'd like to double check the full code

Scroll through agent.py and confirm these sections are present in order:

  • Imports include from persistence import init_db, save_run, get_history, save_state.
  • __init__ calls init_db() at the end.
  • _run_routine has save_run(...) in both the try and except blocks.
  • start() calls save_state at the top and uses input() for the command loop.
  • pause(), resume(), stop(), show_history(), and show_status() are all present below start().
  • The if __name__ == "__main__" block at the bottom is unchanged.
Test the full life-cycle

Everything is wired up. Before you run the agent, think about what you expect to happen differently from last time.

When you quit and restart, do you think the history from the first run will still be there? Why or why not?

  • Delete any leftover database from earlier testing:
rm -f agent_state.db
  • Start the agent:
python agent.py

You will see the initial pass run both routines and print summaries. Then a > prompt appears.

  • Type h and press Enter to view run history.
  • Type p and press Enter to pause the agent.
  • Type s and press Enter to check status. You should see Status: PAUSED and the next-run times showing paused.
  • Type r and press Enter to resume.
  • Type q and press Enter to quit gracefully.

You should see Agent stopped. State saved to agent_state.db printed.

  • Now restart the agent to prove persistence works:
python agent.py
  • After the initial pass completes, type h and press Enter.

You should see runs from BOTH the first session and the current session listed together. The blank-slate problem is solved.

  • Type q to quit.

History shows only the current session?

Make sure you did not accidentally delete agent_state.db between sessions. The database file must persist on disk between runs. Also confirm your init_db() uses CREATE TABLE IF NOT EXISTS so it does not wipe existing data.

Why isn't my agent remembering history across restarts?

Your agent is now a proper long-running service with full lifecycle control and persistent memory. In the Secret Mission, you will push this further by building a change-detection routine that compares content across runs.

Secret mission

Build a Change Detector Routine

Your agent can summarize pages, but what if you want to know when something changes? Build a routine that watches a page over time and alerts you the moment new content appears.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally, so there are no ongoing costs. The only resource consumption is disk space for the Ollama model and the SQLite database.

Resources you used:

  • SQLite database file: agent_state.db (stores all run history and agent state, no cost).
  • Ollama model: qwen3 stored locally (~5GB disk space, no ongoing cost).
  • Project directory: web-agent on your Desktop (contains all Python files and virtual environment).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your agent or building new routines.

  • Your agent runs locally with zero ongoing cost.
  • The qwen3 model stays available for other Ollama projects.
  • Your SQLite database preserves all run history for future queries.
  • Ollama itself remains installed on your system. You can use it with any other local AI project.

Pause - I'll come back to this later

Stop the agent process to free up memory. Your history persists in SQLite and the model stays on disk for when you return.

  • If the agent is running, type q in the agent terminal to quit gracefully.
  • All run history remains in agent_state.db. When you restart with python agent.py, past runs are still there.
  • The qwen3 model stays downloaded. No re-pull needed next time.

Delete - I don't want to use this again

Remove all project resources and free up disk space.

  • If the agent is running, type q to stop it first.
  • Remove the Ollama model to free ~5GB of disk space by running:
ollama rm qwen3
  • Remove the entire project directory (including the database, virtual environment, and all Python files) by running:
cd ~/Desktop
rm -rf web-agent

What about Ollama itself?

The commands above remove the model and your project files. Ollama itself remains installed on your system. If you want to keep using local AI models for other projects, leave it installed. If you want to fully remove Ollama, follow the official uninstall guide at docs.ollama.com/linux.

That's a Wrap!

That's a Wrap!

Nice work! You just built a fully autonomous agent that lives on your machine, surfs the web on a schedule, and remembers everything it has ever done.

You've learned how to:

  • Build a daemon-style agent that fetches live web pages with httpx and BeautifulSoup, then summarizes content using a local LLM (Ollama + Qwen 3) with zero API costs.
  • Implement time-based scheduling with APScheduler so routines fire automatically, and add interactive life-cycle controls (pause, resume, status, history, quit) to manage the agent without killing the process.
  • Add SQLite persistence so the agent remembers its entire run history across restarts, turning a blank-slate script into a reliable long-running service.
  • Secret Mission: Built a change detection routine that compares content across runs and alerts you when a monitored page has new content.

Ready to quiz yourself?