Agentic Scene Analyzer with YOLO

Build an AI agent that detects objects and generates safety reports.

Introduction

30 Second Summary

Security cameras, traffic monitors, and warehouse sensors capture thousands of images every day. Seeing what is in those images is the easy part. Understanding what it all means and deciding what to do about it is where the real value lies.

In this project, you will build an agentic scene safety analyzer. You upload an image, YOLOv8 detects every object in it, and a LangGraph agent reasons about the scene to produce a safety report with risk levels and recommendations.

What You'll Build

Upload any photo to your running API and watch it identify every object in the scene, assess the risk level, and return specific safety recommendations, all at zero cost.

By the end of this project, you'll have:

  • A working object detection pipeline where you upload any photo and see every identified object with confidence scores and bounding box coordinates.
  • A LangGraph agent that autonomously calls the YOLO detector as a tool, interprets the raw detections, and produces a natural-language safety analysis with risk levels and actionable recommendations.
  • A running FastAPI server with a single /analyze endpoint where you upload an image and receive the full analysis as JSON, with interactive documentation at /docs.
  • Secret Mission: Add a second vision tool so the agent dynamically chooses between detailed detection and category counting based on your question.

Are there any prerequisites?

This project assumes you have basic Python knowledge and some familiarity with agentic AI frameworks like LangGraph, CrewAI, or AutoGen. No computer vision experience is needed.

Before We Start

Before diving in, let's lock in what you're about to build and why it matters. This project connects computer vision with agentic AI to create a system where the model sees and the agent decides what to do about it.

Set Up Your Environment

Your scene analyzer needs four layers to work: a computer vision model that spots objects, an agent framework that reasons about them, an LLM connector that talks to the reasoning model, and an API server that ties it all together.

In this step, you will create a clean project workspace, install all four packages, and grab a free OpenRouter API key so the agent has an LLM to think with.

In this step, get ready to:
  • Create a project folder with a Python virtual environment.
  • Install the computer vision, agent, and API dependencies.
  • Configure your OpenRouter API key and verify the YOLO model loads.
Create a project folder and virtual environment

A virtual environment gives your project its own isolated set of Python packages. This prevents version conflicts with other projects on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir scene-analyzer
cd scene-analyzer

Your terminal prompt now shows scene-analyzer as the current directory.

  • Create a virtual environment and activate it by running the commands for your operating system:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

Your terminal prompt now starts with (venv). This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activate command, not just the python -m venv venv line. The first command creates the environment. The second activates it.

On Windows, if PowerShell blocks the activation script with an error about execution policies, run this command first, then try activating again:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

help me activate my Python virtual environment

Install dependencies

The project uses four core packages. You will list them in a requirements.txt file with pinned versions so everyone gets the same setup.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code (or Cursor) and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Navigate to Desktop and select the scene-analyzer folder, then click Open.
  • Click the new file icon in the Explorer sidebar.
  • Name the file requirements.txt and paste the following content:
ultralytics==8.4.126
langgraph==1.2.11
langchain-openai==1.6.0
fastapi[standard]==0.141.1

What does each package do?

  • ultralytics provides YOLOv8 object detection. It handles model loading, inference, and result parsing.
  • langgraph is the LangGraph agent framework. It orchestrates the reasoning loop between the LLM and tools.
  • langchain-openai connects LangGraph to any OpenAI-compatible API, including OpenRouter.
  • fastapi[standard] is the FastAPI web framework with its standard server dependencies included.
  • Save requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, same on both macOS and Windows).

Is your virtual environment active in this terminal?

Check that your terminal prompt starts with (venv). If VS Code opened a fresh terminal without the virtual environment, activate it again using the same command from the previous substep.

  • Install all four packages by running:
pip install -r requirements.txt

You will see download progress bars for each package and its dependencies. The install finishes with a Successfully installed line listing all packages.

Install failed or timed out?

Check that (venv) appears in your prompt. Installing without the virtual environment active can cause permission errors.

If you see a network timeout, try running the install command again. Large packages like ultralytics pull in many dependencies and may take a minute on slower connections.

help me fix my pip install error

✔️ Awesome, I've got everything!

Great. Double check you have saved requirements.txt and move on to the next substep.

ⓧ I'd like to double check the full code

ultralytics==8.4.126
langgraph==1.2.11
langchain-openai==1.6.0
fastapi[standard]==0.141.1
Get your OpenRouter API key and verify YOLO

OpenRouter gives you access to LLM models through a single API key. The free tier includes 50 requests per day with no credit card required.

  • Open your browser and go to openrouter.ai.
  • Create a free account. No credit card is needed.
  • After signing in, navigate to your API keys page in your account settings.
  • Create a new key and give it a name (e.g. scene-analyzer).
  • Copy the key immediately. It is only shown once.

Why OpenRouter?

OpenRouter provides an OpenAI-compatible API, which means you can use the battle-tested langchain-openai package to connect. The free tier gives 50 requests per day at zero cost.

  • Switch back to the VS Code terminal and set your API key as an environment variable using the command for your operating system:

macOS

export OPENROUTER_API_KEY=your-api-key-here

Windows

$env:OPENROUTER_API_KEY="your-api-key-here"
  • Replace your-api-key-here with the key you just copied from OpenRouter.

This variable is session-only

The environment variable lasts only as long as this terminal session. If you close and reopen the terminal, you will need to set it again.

Now verify that the YOLOv8 model downloads and loads correctly. Before you run the next command, what do you think will happen? Will it print output immediately, or will something need to download first?

  • Test that YOLO loads by running:
python -c "from ultralytics import YOLO; print(YOLO('yolov8n.pt'))"

The first time you run this, ultralytics downloads the yolov8n.pt weights file (about 6 MB). After the download, you see a model summary with the architecture layers, parameters, and GFLOPs printed to the terminal.

Seeing a ModuleNotFoundError?

If Python cannot find ultralytics, your virtual environment may not be active. Check for (venv) in your prompt and activate it again if needed.

If you see a download error for yolov8n.pt, check your internet connection and try the command again.

help me troubleshoot my YOLO model loading error

Your workspace is ready with all the tools installed and a working YOLO model. Next up, you will point YOLO at a real image and see exactly what it detects.

Run YOLO Object Detection

Your environment is locked in. Python packages are installed and the YOLOv8 model is ready to go. Now it is time to put it to work.

In this step, you will feed an image into YOLO and see exactly what comes back. The output is raw detection data: class names, confidence scores, and bounding box coordinates. It tells you what is in the image, but nothing about what it means. That gap is what your agent will close in the next step.

In this step, get ready to:
  • Download a sample image and write a detection script.
  • Run the detector and read YOLO's raw output.
Get a sample image and create detector.py

YOLO needs an image to analyze. The Ultralytics project provides a standard test photo of a city bus scene with people and vehicles.

  • Download the sample image by running this command:
curl -L -o sample.jpg https://github.com/ultralytics/assets/releases/download/v0.0.0/bus.jpg

Use your own photo

You can use any image with people, vehicles, or objects instead. Save it as sample.jpg in your project folder.

  • Confirm the file downloaded by running:
ls sample.jpg

You should see sample.jpg listed in the output.

File not found?

  • Make sure you are in your project folder. Run pwd to check your current directory.
  • If curl is not available on Windows, download the image directly from your browser at the same URL and save it as sample.jpg in your project folder.
  • help me download the sample image

Now you have an image to work with. Next, create the detection script that will analyze it.

  • Create a new file called detector.py in your project folder.
  • Add the YOLO model loader and detection function by pasting this code:
from ultralytics import YOLO

# Load the lightweight YOLOv8 nano model
model = YOLO("yolov8n.pt")


def run_detection(image_path: str) -> list[dict]:
    """Run YOLOv8 object detection on an image."""
    # Run inference with verbose=False to suppress model logs
    results = model(image_path, verbose=False)
    detections = []
    # Extract class name, confidence, and bounding box for each detected object
    for box in results[0].boxes:
        detections.append({
            "class_name": model.names[int(box.cls)],
            "confidence": round(float(box.conf), 3),
            "bbox": [round(float(v)) for v in box.xyxy[0]],
        })
    return detections

What does this code do?

  • model = YOLO("yolov8n.pt") loads the YOLOv8 nano model. The weights file was already downloaded in Step 1, so this loads instantly.
  • run_detection() takes an image path, runs YOLO inference, and returns a list of dictionaries. Each dictionary holds three pieces of information about a detected object.
  • class_name is the human-readable label (e.g. "person", "bus"). confidence is a float between 0 and 1 representing how sure the model is. bbox is the bounding box in pixel coordinates.

The detection function is ready, but there is no way to run it from the terminal yet. The next chunk adds a command-line entry point.

  • Add the command-line runner below your run_detection function in detector.py:
if __name__ == "__main__":
    import sys

    # Use the filename from the command line, or default to sample.jpg
    image_path = sys.argv[1] if len(sys.argv) > 1 else "sample.jpg"
    detections = run_detection(image_path)

    # Print each detected object with its details
    print(f"\nDetected {len(detections)} objects:\n")
    for d in detections:
        print(
            f"  {d['class_name']:15s}  confidence: {d['confidence']:.3f}"
            f"  bbox: {d['bbox']}"
        )

What does this code do?

  • if __name__ == "__main__" makes this block run only when you execute the file directly, not when another script imports it. This is important because agent.py will import run_detection later.
  • sys.argv[1] reads the image filename from the command line. If you do not provide one, it defaults to sample.jpg.
  • The print loop formats each detection on its own line with the class name padded to 15 characters for readability.
  • Save detector.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved detector.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete detector.py file. Compare it against yours:

from ultralytics import YOLO

model = YOLO("yolov8n.pt")


def run_detection(image_path: str) -> list[dict]:
    """Run YOLOv8 object detection on an image."""
    results = model(image_path, verbose=False)
    detections = []
    for box in results[0].boxes:
        detections.append({
            "class_name": model.names[int(box.cls)],
            "confidence": round(float(box.conf), 3),
            "bbox": [round(float(v)) for v in box.xyxy[0]],
        })
    return detections


if __name__ == "__main__":
    import sys

    image_path = sys.argv[1] if len(sys.argv) > 1 else "sample.jpg"
    detections = run_detection(image_path)

    print(f"\nDetected {len(detections)} objects:\n")
    for d in detections:
        print(
            f"  {d['class_name']:15s}  confidence: {d['confidence']:.3f}"
            f"  bbox: {d['bbox']}"
        )
Run the detector and read the output

Your script is ready. Before running it, think about what YOLO will actually give you.

What do you think the output will contain? Just object names? Locations? Both? Make a prediction before you run it.

  • Run the detector on your sample image:
python detector.py sample.jpg

You should see a list of detected objects printed to the terminal, each with a class name, a confidence score between 0 and 1, and bounding box coordinates in pixels.

What am I looking at?

Each line represents one object YOLO found in the image. The class_name is YOLO's label for the object (like "person" or "bus"). The confidence score tells you how certain the model is about that detection. Higher means more confident.

The bbox is a list of four numbers representing the top-left and bottom-right corners of a rectangle around the object, in pixel coordinates. This tells you where in the image the object was found.

Notice what is missing. YOLO tells you a person is at coordinates [669, 392, 810, 878], but it cannot tell you that the person is standing dangerously close to a moving vehicle. That kind of reasoning requires an agent.

Seeing an error when running the script?

  • If you see ModuleNotFoundError: No module named 'ultralytics', make sure your virtual environment is activated.
  • If you see FileNotFoundError, check that sample.jpg exists in the same folder as detector.py. Run ls to verify.
  • If you see SyntaxError, compare your code against the full file in the double-check tab above.
  • help me debug my detector.py script

Your detection pipeline works. You can feed in any image and get back a structured list of objects with locations and confidence scores. But the output is just raw data. It cannot reason about relationships between objects, assess risk, or recommend actions. That is exactly the job for a LangGraph agent, which you will build next.

Build the LangGraph Scene Analyzer

In the last step, you ran detector.py and saw what YOLO actually produces: class names, confidence scores, and bounding box coordinates. Useful data, but just data.

Raw detections cannot tell you that a person standing next to a forklift is a safety concern. That requires reasoning. In this step, you will build a LangGraph agent that calls the YOLO detector as a tool, interprets the results, and produces an actionable safety analysis.

In this step, get ready to:
  • Wrap your detection function as a LangGraph tool with a system prompt for safety analysis.
  • Build a StateGraph with agent and tools nodes connected by conditional edges.
  • Run the agent and observe it autonomously calling the detector to produce a safety report.
Wrap detection as a LangGraph tool

A LangGraph agent does not call Python functions directly. It uses tools: functions wrapped with a special decorator so the agent can discover and invoke them.

  • Create a new file called agent.py in your project folder. Add the imports and system prompt by pasting this code:
import os
import sys
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, MessagesState, END
from langgraph.prebuilt import ToolNode, tools_condition
from detector import run_detection

SYSTEM_PROMPT = """You are a scene safety analyzer. When asked to analyze an image, \
use the detect_objects tool to identify objects in the scene. Then provide:

1. A brief summary of the scene
2. Any safety concerns (people near vehicles, overcrowding, unattended objects)
3. A risk level: low, medium, or high
4. Specific recommendations

Always call the detect_objects tool before making your assessment. Be concise \
and actionable."""

What does this code do?

The imports pull in the pieces you need: ChatOpenAI to talk to OpenRouter, tool to create an agent-callable function, and StateGraph plus ToolNode to wire the agent loop together.

The SYSTEM_PROMPT tells the LLM exactly what to do: call the detection tool first, then reason about the scene in a structured format with a risk level and recommendations.

  • Below the SYSTEM_PROMPT, add the detect_objects tool function by pasting this code:
@tool
def detect_objects(image_path: str) -> str:
    """Detect objects in an image using YOLOv8 object detection.

    Args:
        image_path: File path to the image to analyze.
    """
    detections = run_detection(image_path)
    if not detections:
        return "No objects detected in the image."

    lines = [f"Detected {len(detections)} objects:"]
    for d in detections:
        lines.append(
            f"- {d['class_name']} (confidence: {d['confidence']}) "
            f"at bbox [{d['bbox'][0]}, {d['bbox'][1]}, "
            f"{d['bbox'][2]}, {d['bbox'][3]}]"
        )
    return "\n".join(lines)

What does this code do?

The @tool decorator turns your plain function into a LangGraph tool. The docstring becomes the tool's description that the LLM reads when deciding which tool to call.

Inside the function, run_detection calls the YOLO model. The results are formatted into a human-readable string so the LLM can reason about the detections instead of parsing raw Python dicts.

  • Save agent.py.
  • Verify that the tool wraps correctly by running this command:
python -c "from agent import detect_objects; print('Tool ready:', detect_objects.name)"

You should see Tool ready: detect_objects printed in the terminal.

Seeing an ImportError?

  • Make sure your virtual environment is activated. The langchain-openai and langgraph packages must be installed in the active environment.
  • Check that detector.py is in the same folder as agent.py. The import from detector import run_detection needs both files side by side.
  • Help me debug this ImportError.
Build the StateGraph with agent and tools nodes

The ReAct pattern is a loop: the agent thinks, optionally calls a tool, reads the tool's output, then thinks again. In LangGraph, you model this loop as a graph with two nodes and conditional edges that route between them.

  • Below the detect_objects function in agent.py, add the create_agent function by pasting this code:
def create_agent():
    """Build and return the compiled LangGraph agent."""
    api_key = os.getenv("OPENROUTER_API_KEY")
    if not api_key:
        raise ValueError("Set the OPENROUTER_API_KEY environment variable.")

    llm = ChatOpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=api_key,
        model="openai/gpt-oss-20b:free",
        temperature=0,
    )

    tools = [detect_objects]
    llm_with_tools = llm.bind_tools(tools)

    def call_agent(state: MessagesState):
        messages = [SystemMessage(content=SYSTEM_PROMPT)] + state["messages"]
        response = llm_with_tools.invoke(messages)
        return {"messages": [response]}

    tool_node = ToolNode(tools)

    graph = StateGraph(MessagesState)
    graph.add_node("agent", call_agent)
    graph.add_node("tools", tool_node)
    graph.set_entry_point("agent")
    graph.add_conditional_edges("agent", tools_condition)
    graph.add_edge("tools", "agent")
    return graph.compile()

What does this code do?

  • ChatOpenAI connects to OpenRouter using the free-tier model openai/gpt-oss-20b:free. The base_url points the standard langchain-openai client at OpenRouter instead of OpenAI directly.
  • bind_tools attaches the detect_objects tool schema to the LLM so it knows what tools are available and how to call them.
  • call_agent prepends the system prompt to every conversation, then invokes the LLM. The response may contain a tool call or a final answer.
  • tools_condition checks the LLM's response. If it contains tool calls, the graph routes to the tools node. If not, the graph ends.
  • The edge from tools back to agent creates the loop. After a tool executes, the agent sees the result and decides whether to call another tool or produce its final answer.
  • Below the create_agent function, add the analyze_scene function and the __main__ block by pasting this code:
def analyze_scene(image_path: str) -> str:
    """Run the agent on an image and return the analysis."""
    agent = create_agent()
    result = agent.invoke({
        "messages": [
            HumanMessage(
                content=f"Analyze this image for safety concerns: {image_path}"
            )
        ]
    })
    return result["messages"][-1].content


if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "sample.jpg"
    print("\n--- Scene Safety Analysis ---\n")
    print(analyze_scene(path))

What does this code do?

analyze_scene creates the agent, sends it a HumanMessage asking for a safety analysis of the given image, and extracts the final text response from the conversation.

The __main__ block lets you run the file directly with python agent.py sample.jpg. If no image path is given, it defaults to sample.jpg.

  • Save agent.py. You will run the full agent in the next substep.
Run the agent and observe the ReAct loop

Everything is wired up. The agent has a tool, a system prompt, and a graph that loops between thinking and acting. Time to see it in action.

Before you run this, think about what you expect. Will the agent just dump the raw detection output, or will it do something more with the data?

  • Run the agent on your sample image by running this command:
python agent.py sample.jpg

The agent sends your request to the LLM. The LLM reads the system prompt, sees the detect_objects tool available, and decides to call it.

After receiving the detection results, it reasons about the scene and produces a structured safety analysis with a summary, safety concerns, risk level, and recommendations.

What should I see?

You should see --- Scene Safety Analysis --- followed by a natural language analysis. The exact wording depends on your image and the LLM's response, but it should include a scene summary, identified safety concerns, a risk level (low, medium, or high), and specific recommendations.

Compare this to the raw YOLO output from Step 2. The same image produced only class names and numbers. Now the agent has turned those raw detections into actionable insights.

Seeing an API error or empty response?

  • Check that your OPENROUTER_API_KEY environment variable is still set. Run echo $OPENROUTER_API_KEY (macOS/Linux) or echo %OPENROUTER_API_KEY% (Windows) to verify.
  • OpenRouter's free tier allows 50 requests per day. If you have exceeded this limit, wait until the next day or check your usage at openrouter.ai.
  • If the output is empty or the agent does not call the tool, the free model may be experiencing high traffic. Try running the command again after a few seconds.
  • Help me debug this API error.

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and that the safety analysis printed successfully.

ⓧ I'd like to double check the full code

Since agent.py is long, confirm that each of these sections is present in your file, in this order:

  • Imports (8 lines starting with import os through from detector import run_detection).
  • SYSTEM_PROMPT triple-quoted string describing the safety analyzer role.
  • detect_objects function with the @tool decorator.
  • create_agent() function that builds the ChatOpenAI client, binds tools, creates the StateGraph, and compiles it.
  • analyze_scene() function that invokes the agent with a HumanMessage.
  • if __name__ == "__main__": block that parses the command-line argument and prints the analysis.

Scroll up and compare each chunk against what you pasted. Make sure there are no missing functions or out-of-order sections.

Your agent can see and reason. It takes raw pixel-level detections and transforms them into a structured safety report. Next up, you will wrap this entire pipeline in a FastAPI endpoint so anyone can upload an image and get an analysis back as JSON.

Serve It via FastAPI

Your agent produces a full safety analysis from the terminal, but anyone who wants to use it needs Python installed, your source files, and a working environment.

Wrapping the agent in an API endpoint turns it into a service that anything can call. In this step, you'll create a FastAPI server that accepts image uploads and returns the agent's safety analysis as JSON.

In this step, get ready to:
  • Create a FastAPI server with a /analyze endpoint that accepts image uploads.
  • Test the full pipeline through FastAPI's interactive docs in your browser.
Create server.py with the /analyze endpoint

The server file needs to accept an uploaded image, hand it to the agent, and return the analysis as JSON.

  • Create a new file called server.py in the same folder as agent.py and detector.py.
  • Set up the FastAPI app by pasting this code into server.py:
import os
import tempfile
from fastapi import FastAPI, File, UploadFile, HTTPException
from agent import analyze_scene

# Create the FastAPI app with metadata for the docs page
app = FastAPI(
    title="Agentic Scene Analyzer",
    description="Upload an image for AI-powered safety analysis "
    "using YOLO + LangGraph.",
)

What does this code do?

  • The imports pull in FastAPI's core pieces: File and UploadFile handle incoming file uploads, and HTTPException lets you return proper error responses.
  • The analyze_scene import brings in the agent function you built in the previous step.
  • The title and description appear on the auto-generated docs page so you know which API you are looking at.
  • Now add the /analyze endpoint below the app definition in server.py:
@app.post("/analyze")
async def analyze_image(file: UploadFile = File(...)):
    """Accept an image upload and return a scene safety analysis."""
    # Only allow JPEG and PNG images
    if file.content_type not in {"image/jpeg", "image/png"}:
        raise HTTPException(
            status_code=400,
            detail="Only JPEG and PNG images are accepted.",
        )

    # Read the uploaded file contents
    contents = await file.read()

    # Write to a temp file so YOLO can read it from disk
    with tempfile.NamedTemporaryFile(suffix=".jpg", delete=False) as tmp:
        tmp.write(contents)
        tmp_path = tmp.name

    # Run the agent and clean up the temp file afterwards
    try:
        analysis = analyze_scene(tmp_path)
        return {"analysis": analysis, "filename": file.filename}
    finally:
        os.unlink(tmp_path)

What does this code do?

  • The @app.post("/analyze") decorator registers a POST endpoint at /analyze.
  • The content_type check rejects anything that is not a JPEG or PNG before doing any processing.
  • YOLO reads images from disk, so the endpoint writes the upload to a temporary file, passes the path to analyze_scene(), and then deletes the temp file in the finally block so nothing piles up.
  • The response is a JSON object with two keys: analysis (the agent's safety report) and filename (the name of the uploaded file).
  • Save server.py.

Seeing a red underline on the import?

  • If your editor highlights from agent import analyze_scene, make sure your virtual environment is selected as the Python interpreter.
  • Confirm that agent.py is in the same folder as server.py.
  • help me fix an import error for analyze_scene in server.py

✔️ Awesome, I've got everything!

Great. Double check you have saved server.py and let's start the server.

ⓧ I'd like to double check the full code

Your server.py has two sections. Scroll up and confirm each one is present, in order:

  • The imports and app = FastAPI(...) definition at the top.
  • The @app.post("/analyze") endpoint function with the content-type check, temp file write, analyze_scene() call, and cleanup.
Start the server and test with the interactive docs

FastAPI includes a built-in Swagger UI that auto-generates interactive docs for every endpoint you define. You can test your /analyze endpoint directly in the browser without writing any client code.

  • Start the development server by running this command:
fastapi dev server.py

What happens when you run this?

The fastapi dev command starts a local development server with auto-reload enabled. It finds the app object in server.py and serves it at http://127.0.0.1:8000.

This command takes over your terminal. You will not be able to type other commands in this terminal while the server is running. Press Ctrl+C when you want to stop it.

  • You should see log output ending with a line like Application startup complete.
  • Open your browser and go to http://localhost:8000/docs.

You should see a Swagger UI page titled Agentic Scene Analyzer with a green POST /analyze card.

Before you upload an image, take a moment to predict: what keys do you expect to see in the JSON response?

  • Click the POST /analyze card to expand it.
  • Click Try it out.
  • Click Choose File and select a JPEG or PNG image from your computer.
  • Click Execute.

The response takes a few seconds while YOLO runs detection and the agent reasons about the scene. You should see a JSON response with two keys: analysis containing the agent's safety report and filename containing your uploaded file's name.

Seeing an error in the response?

  • A 422 Unprocessable Entity usually means you clicked Execute without selecting a file first. Choose a file and try again.
  • A 400 Bad Request means the file is not JPEG or PNG. Make sure you selected an image file.
  • A 500 Internal Server Error usually means the OPENROUTER_API_KEY environment variable is not set in the terminal where you ran fastapi dev. Stop the server with Ctrl+C, set the key, and restart.
  • help me debug my FastAPI /analyze endpoint returning an error

Why FastAPI over Flask?

FastAPI auto-generates interactive API docs (the Swagger UI you just used) from your endpoint type hints. It also handles async functions natively, which matters when the agent makes network calls to OpenRouter.

Flask would work too, but you would need to add Swagger docs and async support yourself.

Secret mission

Add a Second Vision Tool

Your agent has one vision tool right now. Real-world vision systems need to match the right tool to the right question. Can you add a count_by_category tool and teach the agent to choose which tool to call based on the query?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine. The only external service is your free OpenRouter account, which has no ongoing costs and does not require a credit card. There is nothing here that will charge you if left running.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Your project folder on the Desktop (containing detector.py, agent.py, server.py, requirements.txt, sample.jpg, yolov8n.pt, and the Python virtual environment).
  • The OPENROUTER_API_KEY environment variable set in your terminal session.
  • The FastAPI dev server (if still running in your terminal).
  • Your free OpenRouter account.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your scene analyzer.

  • Your project files stay on your Desktop.
  • Your OpenRouter account stays active with its free tier (50 requests per day, no charges).
  • If you close your terminal, the OPENROUTER_API_KEY variable will be cleared. You will need to export it again next time you run the project.

Pause - I'll come back to this later

Shut down running processes to free up system resources while keeping your files intact.

  • If the FastAPI dev server is still running, switch to that terminal and press Ctrl+C to stop it.
  • Deactivate your virtual environment by running:
deactivate
  • Your project folder and all files remain on your Desktop for next time.
  • When you come back, re-activate the virtual environment, export OPENROUTER_API_KEY again, and you are good to go.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and optionally your OpenRouter account.

  • If the FastAPI dev server is still running, switch to that terminal and press Ctrl+C to stop it.
  • If your virtual environment is still active, deactivate it by running:
deactivate
  • Delete your project folder from the Desktop by running:
cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

Windows users

On Windows, use this command instead in PowerShell:

cd ~\Desktop
Remove-Item -Recurse -Force [[PROJECT_FOLDER="your project folder name"]]
  • Clear the OPENROUTER_API_KEY environment variable from your current terminal session. On macOS or Linux, run:
unset OPENROUTER_API_KEY
  • If you added the key to a shell profile file (like .zshrc or .bashrc), open that file and remove the export OPENROUTER_API_KEY line.
  • To delete your OpenRouter account, go to openrouter.ai/settings/preferences, click Manage Account, select the Security tab, and follow the prompts to delete your account. This is permanent and cannot be undone.

Nice Work!

Nice Work!

You just built a full agentic scene analyzer that sees objects in images and reasons about what they mean. From raw pixel data to intelligent safety reports, your system connects computer vision with autonomous decision-making.

You've learned how to:

  • Run YOLOv8 object detection on any image and extract class names, confidence scores, and bounding box coordinates.
  • Wrap a detection function as a LangGraph tool and build a ReAct agent that autonomously calls the detector and produces natural language safety analyses via OpenRouter's free tier.
  • Serve the entire pipeline as a FastAPI endpoint with interactive documentation at /docs, turning a local script into a service anyone can call.
  • Secret Mission: Added a count_by_category tool and verified the agent dynamically selects the right vision tool based on the query, demonstrating multi-tool orchestration.

Ready to quiz yourself?