Multi-Agent AI on Google Cloud Run

Build a RAG-powered research assistant with Google ADK and deploy it to Cloud Run.

Introduction

30 Second Summary

When you ask a question and get a vague answer, you probably wish you could send a team of researchers to dig through files, find the right sources, and come back with a clear, cited response. That's exactly what multi-agent AI systems do.

In this project, you will build a multi-agent research assistant using Google's Agent Development Kit (ADK) that searches a knowledge base, retrieves relevant documents, and synthesizes grounded answers with citations. You'll deploy the entire system to Google Cloud Run with production-grade observability.

What You'll Build

A three-agent system where a Coordinator delegates questions to a Retriever (powered by ChromaDB vector search) and a Synthesizer (powered by Gemini), deployed to Google Cloud Run with OpenTelemetry tracing.

By the end of this project, you'll have:

  • A fully functional multi-agent ADK application with hierarchical delegation between a Coordinator, Retriever, and Synthesizer agent.
  • A RAG pipeline powered by a ChromaDB vector database and Gemini embeddings that retrieves relevant documents via semantic search.
  • A Cloud Run deployment accessible via HTTP endpoints, with OpenTelemetry traces visible in Google Cloud Trace.
  • Secret Mission: Add a Critic agent that implements the self-reflection pattern to fact-check the Synthesizer's answers against source documents.

Are there any prerequisites?

You should have basic Python knowledge and be comfortable running terminal commands. A conceptual understanding of RAG (Retrieval-Augmented Generation) and agentic AI is helpful but not required.

Set Up Your GCP Account and Project

Before you can build anything with Google's Agent Development Kit, you need a Google Cloud project to host your services.

In this step, you will create a free GCP account, set up a new project with billing, enable the APIs your multi-agent system depends on, and configure your local gcloud CLI to point at the project.

In this step, get ready to:
  • Create a Google Cloud account with $300 in free trial credits.
  • Enable the Vertex AI, Cloud Run Admin, and Cloud Build APIs.
  • Configure your gcloud CLI with the correct project and region.
Create your Google Cloud account and project

Google Cloud offers a 90-day free trial preloaded with $300 in credits. You will not be charged unless you manually upgrade to a paid account after the trial ends.

  • Open your browser and navigate to the Google Cloud free trial page.
  • Click Get started for free.
  • Sign in with your Google account (or create one if needed).
  • Follow the prompts to enter your country and payment method for identity verification.

Will I be charged?

No. Google requires a credit card only for identity verification. Your free trial billing account auto-closes after 90 days or when the $300 credit is spent. You will not be charged unless you manually upgrade to a paid account.

Once your account is ready, you need to create a new project. Every resource in GCP lives inside a project.

  • In the Google Cloud Console, click the project dropdown at the top of the page.
  • Click New Project.
  • Enter a project name such as research-assistant.
  • Click Create.
  • Wait a few seconds for the project to be created, then select it from the project dropdown.

What is a GCP project?

A project is the top-level container that holds all of your cloud resources. It has a unique project ID (like research-assistant-442210) that you will reference throughout this project.

Note your project ID now. You can always find it in the project dropdown or on the Dashboard page.

Billing is automatically enabled for new free trial accounts. If you used an existing Google Cloud account, verify billing is active.

  • Open the Navigation menu (three horizontal lines at the top left).
  • Select Billing.
  • Confirm your project is linked to your billing account. If it shows Billing is not enabled, click Link a billing account and follow the prompts.

Getting an error during signup?

Make sure your Google account has an email address associated with it. Accounts with only a phone number are not supported. If you see error codes like OR-CCSEH-26, try a different payment method.

help me troubleshoot my Google Cloud signup error

Enable the required APIs

Your project needs three APIs enabled: Vertex AI for Gemini model access, Cloud Run Admin for serverless deployment, and Cloud Build for building container images from source.

  • Open your terminal.
  • Enable all three APIs at once by running this command:
gcloud services enable aiplatform.googleapis.com \
    run.googleapis.com \
    cloudbuild.googleapis.com

What does this command do?

  • `gcloud services enable` activates APIs for the currently configured project.
  • `aiplatform.googleapis.com` is the Vertex AI API, which gives your ADK agents access to Gemini models.
  • `run.googleapis.com` is the Cloud Run Admin API for deploying serverless containers.
  • `cloudbuild.googleapis.com` is the Cloud Build API that compiles your source code into container images during deployment.

The command takes about 30 seconds to complete. You should see output similar to:

Operation "operations/acf.p2-123456789-abc123" finished successfully.

Seeing a permission denied error?

Make sure you selected the correct project before running the command. Run gcloud config get-value project to check which project is active. If it returns the wrong project, set it with the command in the next substep.

help me fix permission denied when enabling GCP APIs

Configure your gcloud CLI

Your gcloud CLI needs to know which project to target and which region to deploy to. Setting these as defaults means you will not have to specify them on every command.

  • Set your project ID by running this command:
gcloud config set project [[GCP_PROJECT_ID="your-project-id"]]
  • Set your default region to us-central1 by running this command:
gcloud config set run/region us-central1

Why us-central1?

This region has strong availability for Vertex AI models and Cloud Run. It also tends to have the lowest latency for most learners. You can choose a different region if needed, but the rest of this project assumes us-central1.

  • Verify your configuration is correct by running:
gcloud config list

You should see output that includes your project ID and region:

[core]
project = your-project-id

[run]
region = us-central1

Project ID not showing correctly?

Run gcloud auth login to re-authenticate your CLI. This opens a browser window where you sign in with the same Google account you used to create the project.

help me connect my gcloud CLI to the right project

Your Google Cloud project is fully configured. Next up, you will build your first ADK agent and test it locally.

Build Your First ADK Agent

Your GCP account is set up and your APIs are enabled. Now it's time to build something on top of that infrastructure.

Google's Agent Development Kit (ADK) is a Python framework for building multi-agent AI systems. In this step, you'll install ADK, create your first agent project, and see it respond to messages in the browser.

In this step, get ready to:
  • Install ADK and set up the project structure.
  • Build a greeting agent with a custom tool.
  • Run and test the agent locally through ADK's web interface.
Set up your ADK project

Before writing any agent code, you need a Python virtual environment to isolate your project's dependencies. You'll also need Application Default Credentials (ADC) so ADK can authenticate to Vertex AI on your behalf.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create and activate a Python virtual environment by running these commands:
# Create a virtual environment called .venv
python3 -m venv .venv

# Activate it
source .venv/bin/activate

What does this do?

A virtual environment is an isolated Python installation. Packages you install here won't interfere with your system Python or other projects.

After activation, your terminal prompt shows (.venv) at the beginning, confirming you're inside the virtual environment.

  • Install the core packages by running:
pip install google-adk chromadb

What does this install?

google-adk is Google's Agent Development Kit for building multi-agent systems. chromadb is a vector database you'll use later for the RAG pipeline.

You should see a stream of download messages ending with Successfully installed followed by a list of packages.

Installation failing?

Make sure you see (.venv) in your terminal prompt. If not, rerun source .venv/bin/activate and try the install again.

help me fix my pip install error for google-adk

  • Set up Application Default Credentials so ADK can access Vertex AI locally by running:
gcloud auth application-default login

What is Application Default Credentials?

ADC is how Google Cloud client libraries find credentials automatically. This command saves your user credentials locally so ADK can authenticate to Vertex AI without you pasting API keys into code.

A browser window opens asking you to sign in with your Google account. After signing in, you'll see a confirmation message in your terminal.

  • Sign in with the same Google account you used to create your GCP project.
  • Click Allow when prompted to grant access.

You should see Credentials saved to file in your terminal output.

  • Create the ADK project structure by running:
adk create research_assistant --project your-project-id-here --region us-central1

What does adk create do?

This command scaffolds a new ADK agent project. It creates a research_assistant/ folder containing three files: agent.py, .env, and __init__.py.

The --project and --region flags tell ADK to configure the project for Vertex AI, skipping the interactive backend selection prompt.

The CLI will prompt you to select a model and an agent type.

  • Select gemini-2.5-flash when prompted for the model.
  • Select code when prompted for the agent type.
  • Confirm the files were created by running:
ls research_assistant/

You should see agent.py, .env, and __init__.py listed.

  • Open the .env file inside research_assistant/ in VS Code to verify it contains your Vertex AI configuration:
GOOGLE_GENAI_USE_VERTEXAI=TRUE
GOOGLE_CLOUD_PROJECT=your-project-id-here
GOOGLE_CLOUD_LOCATION=us-central1

What is in the .env file?

GOOGLE_GENAI_USE_VERTEXAI=TRUE tells ADK to route all model calls through Vertex AI instead of a standalone API key.

GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION tell ADK which GCP project and region to use. These should match what you configured in Step 1.

  • If the project ID or region do not match your GCP project, update them now and save the file.

Not sure what your project ID is?

Run gcloud config get-value project in your terminal. It prints the project ID you configured in Step 1.

Build a greeting agent with a custom tool

The scaffolded agent.py contains a placeholder agent. You'll replace it with a custom agent that has a greeting tool to verify everything works end to end.

  • Open research_assistant/agent.py in VS Code.
  • Delete all existing content and replace it with the following code:
from google.adk.agents import Agent

def greet(name: str) -> str:
    """Greets the user by name."""
    return f"Hello, {name}! I am your research assistant."

root_agent = Agent(
    name="research_coordinator",
    model="gemini-2.5-flash",
    description="A research assistant agent.",
    instruction="You are a helpful research assistant. Use the greet tool to greet the user.",
    tools=[greet],
)

What does this code do?

The greet function is a tool that the agent can call. ADK tools are plain Python functions with type hints and docstrings. The docstring tells the LLM when to use the tool.

root_agent is the required entry point for any ADK agent. It defines the model (gemini-2.5-flash), the system instruction, and which tools the agent has access to.

  • Save agent.py.
  • Verify that research_assistant/__init__.py contains this line:
from . import agent

This import makes the agent discoverable by ADK's CLI tools. It should already be there from the scaffold.

  • Launch the ADK development server by running this command from the Desktop directory (the parent of research_assistant/):
adk web

Why run from the parent directory?

ADK discovers agents by scanning subdirectories. Running adk web from the parent of research_assistant/ lets ADK find your agent automatically.

You should see output indicating the server is running, including a line with http://localhost:8000.

  • Open your browser and navigate to http://localhost:8000.
  • Select research_coordinator from the agent dropdown if it's not already selected.
  • Type a message like Hello, my name is Alex and press Enter.

The agent should respond with something like "Hello, Alex! I am your research assistant." This confirms your agent is working, the Vertex AI authentication is successful, and the tool is being called correctly.

Agent not responding?

If you see an authentication error, make sure you ran gcloud auth application-default login earlier in this step.

If the agent doesn't appear in the dropdown, verify that __init__.py contains from . import agent and that agent.py defines root_agent.

help me debug my ADK agent that is not responding in the web interface

Understand the ADK project structure

You now have a working agent. Before moving on, take a moment to understand the three pieces that make it work.

How does ADK discover and run an agent?

  • __init__.py makes the folder a Python package and imports the agent module so ADK's CLI can find it.
  • agent.py must define a variable called root_agent. This is the entry point ADK uses to start your agent.
  • .env stores environment configuration. ADK reads it automatically to determine which model backend and project to use.

Every ADK agent you build follows this same three-file pattern. In later steps, you'll expand agent.py with multiple agents and add a tools.py file for the vector search tool.

What makes a function a tool?

ADK tools are regular Python functions that follow two rules: they have type hints on all parameters, and they have a docstring describing what the function does.

The LLM reads the function name and docstring to decide when to call it. Clear, descriptive names and docstrings lead to better tool-use behavior from the model.

✔️ Awesome, I've got everything!

Great. Double check you've saved all your files, then press Ctrl+C in your terminal to stop the ADK dev server before moving on.

ⓧ I'd like to double check the full code

Here's what each file should look like at this point:

from . import agent
from google.adk.agents import Agent

def greet(name: str) -> str:
    """Greets the user by name."""
    return f"Hello, {name}! I am your research assistant."

root_agent = Agent(
    name="research_coordinator",
    model="gemini-2.5-flash",
    description="A research assistant agent.",
    instruction="You are a helpful research assistant. Use the greet tool to greet the user.",
    tools=[greet],
)
GOOGLE_GENAI_USE_VERTEXAI=TRUE
GOOGLE_CLOUD_PROJECT=your-project-id-here
GOOGLE_CLOUD_LOCATION=us-central1

Your first agent is alive and responding. Next up, you'll create a vector database and fill it with documents so your agent has a knowledge base to search.

Create and Populate Your Vector Database

Your agent can greet users, but it can't answer research questions yet. For that, it needs access to a knowledge base it can search for relevant information.

In this step, you'll build that knowledge base using ChromaDB, a vector database that stores documents as high-dimensional embeddings. Instead of matching keywords, vector search finds documents based on meaning. This is the foundation of the RAG pattern your multi-agent system will use.

In this step, get ready to:
  • Install the google-genai package for embedding support.
  • Create and populate a ChromaDB vector database with sample documents.
  • Build a search tool that queries the database and returns relevant results with relevance scores.
Install google-genai and create the population script

ChromaDB can generate embeddings automatically, but it needs a connection to an embedding model. The google-genai package provides the GoogleGeminiEmbeddingFunction that ChromaDB uses to turn documents into vectors.

  • Install the google-genai package by running this command:
pip install google-genai

Seeing a permissions error?

Make sure your virtual environment is still active. You should see the environment name in parentheses at the start of your terminal prompt. If not, reactivate it before running pip.

Help me fix my pip install error

Now you'll create the script that loads documents into your vector database. This script defines sample documents and uses ChromaDB's embedding integration to store them as vectors.

  • Create a new file called populate_db.py inside your research_assistant/ folder.
  • Add the imports and sample documents by pasting the following code into populate_db.py:
"""Script to populate the ChromaDB vector database with sample documents."""

import chromadb
from chromadb.utils.embedding_functions import GoogleGeminiEmbeddingFunction


SAMPLE_DOCUMENTS = [
    {
        "id": "doc_cloud_run_001",
        "content": "Cloud Run is a fully managed serverless platform on Google Cloud that automatically scales stateless containers. It charges only for the resources used during request handling, making it cost-effective for variable workloads. Cloud Run supports deploying from source code or container images.",
        "metadata": {"source": "cloud_run_overview", "topic": "deployment"},
    },
    {
        "id": "doc_vertex_ai_001",
        "content": "Vertex AI is Google Cloud's unified machine learning platform for building, deploying, and scaling ML models. It provides access to Gemini models, Model Garden with 200+ models, and tools for training, tuning, and evaluation. New customers receive $300 in free credits.",
        "metadata": {"source": "vertex_ai_overview", "topic": "ml_platform"},
    },
    {
        "id": "doc_rag_001",
        "content": "Retrieval-Augmented Generation (RAG) combines information retrieval with text generation. The system first retrieves relevant documents from a knowledge base using semantic search, then passes them as context to a language model for generating grounded responses. This reduces hallucinations compared to pure generation.",
        "metadata": {"source": "rag_architecture", "topic": "ai_patterns"},
    },
    {
        "id": "doc_adk_001",
        "content": "Google Agent Development Kit (ADK) is an open-source framework for building multi-agent AI systems. It supports hierarchical delegation with SequentialAgent, ParallelAgent, and LoopAgent orchestrators. ADK uses sessions and state management to maintain context across conversations.",
        "metadata": {"source": "adk_overview", "topic": "agent_frameworks"},
    },
    {
        "id": "doc_vector_db_001",
        "content": "Vector databases store high-dimensional embeddings and enable similarity search using distance metrics like cosine similarity, dot product, or Euclidean distance. They are essential for RAG pipelines, powering semantic retrieval of relevant documents based on query meaning rather than keyword matching.",
        "metadata": {"source": "vector_db_concepts", "topic": "data_infrastructure"},
    },
    {
        "id": "doc_otel_001",
        "content": "OpenTelemetry is an open-source observability framework that provides APIs for collecting traces, metrics, and logs. When integrated with ADK, it captures spans for each agent invocation, tool call, and LLM interaction, enabling granular performance analysis and debugging of multi-agent systems.",
        "metadata": {"source": "observability_guide", "topic": "monitoring"},
    },
    {
        "id": "doc_state_mgmt_001",
        "content": "ADK session state uses a key-value scratchpad with prefix-based scoping. Keys without a prefix are session-scoped and lost when the conversation ends. Keys with user: prefix persist across sessions for the same user. Keys with app: prefix are shared across all users and sessions.",
        "metadata": {"source": "state_management", "topic": "agent_frameworks"},
    },
    {
        "id": "doc_cloud_trace_001",
        "content": "Google Cloud Trace is a distributed tracing system that collects latency data from applications. It integrates with OpenTelemetry to receive spans via the OTLP protocol. Traces show the full request lifecycle including parent-child relationships between spans, enabling identification of performance bottlenecks.",
        "metadata": {"source": "cloud_trace_guide", "topic": "monitoring"},
    },
]

What does this code do?

The GoogleGeminiEmbeddingFunction import connects ChromaDB to the gemini-embedding-001 model. ChromaDB will call this function automatically whenever it needs to turn text into a vector.

Each document in SAMPLE_DOCUMENTS has an id (unique identifier), content (the text to embed), and metadata (source and topic tags for filtering). These eight documents cover cloud architecture topics that the agent will search later.

  • Add the populate() function below the SAMPLE_DOCUMENTS list in the same file:
def populate():
    """Populate the ChromaDB collection with sample documents."""
    client = chromadb.PersistentClient(path="./chroma_data")
    embedding_function = GoogleGeminiEmbeddingFunction(
        model_name="gemini-embedding-001",
        task_type="RETRIEVAL_DOCUMENT",
    )
    collection = client.get_or_create_collection(
        name="research_docs",
        embedding_function=embedding_function,
    )

    collection.upsert(
        ids=[doc["id"] for doc in SAMPLE_DOCUMENTS],
        documents=[doc["content"] for doc in SAMPLE_DOCUMENTS],
        metadatas=[doc["metadata"] for doc in SAMPLE_DOCUMENTS],
    )

    print(f"Successfully populated {len(SAMPLE_DOCUMENTS)} documents.")
    print(f"Collection now has {collection.count()} documents total.")


if __name__ == "__main__":
    populate()

What does this function do?

  • PersistentClient(path="./chroma_data") creates a ChromaDB database that saves to disk instead of disappearing when the script finishes.
  • get_or_create_collection() creates a collection called research_docs (or reuses it if it already exists). The embedding function is attached here so ChromaDB knows how to vectorize documents.
  • upsert() inserts documents or updates them if they already exist. ChromaDB automatically calls the Gemini embedding function to generate vectors for each document.
  • Save populate_db.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

"""Script to populate the ChromaDB vector database with sample documents."""

import chromadb
from chromadb.utils.embedding_functions import GoogleGeminiEmbeddingFunction


SAMPLE_DOCUMENTS = [
    {
        "id": "doc_cloud_run_001",
        "content": "Cloud Run is a fully managed serverless platform on Google Cloud that automatically scales stateless containers. It charges only for the resources used during request handling, making it cost-effective for variable workloads. Cloud Run supports deploying from source code or container images.",
        "metadata": {"source": "cloud_run_overview", "topic": "deployment"},
    },
    {
        "id": "doc_vertex_ai_001",
        "content": "Vertex AI is Google Cloud's unified machine learning platform for building, deploying, and scaling ML models. It provides access to Gemini models, Model Garden with 200+ models, and tools for training, tuning, and evaluation. New customers receive $300 in free credits.",
        "metadata": {"source": "vertex_ai_overview", "topic": "ml_platform"},
    },
    {
        "id": "doc_rag_001",
        "content": "Retrieval-Augmented Generation (RAG) combines information retrieval with text generation. The system first retrieves relevant documents from a knowledge base using semantic search, then passes them as context to a language model for generating grounded responses. This reduces hallucinations compared to pure generation.",
        "metadata": {"source": "rag_architecture", "topic": "ai_patterns"},
    },
    {
        "id": "doc_adk_001",
        "content": "Google Agent Development Kit (ADK) is an open-source framework for building multi-agent AI systems. It supports hierarchical delegation with SequentialAgent, ParallelAgent, and LoopAgent orchestrators. ADK uses sessions and state management to maintain context across conversations.",
        "metadata": {"source": "adk_overview", "topic": "agent_frameworks"},
    },
    {
        "id": "doc_vector_db_001",
        "content": "Vector databases store high-dimensional embeddings and enable similarity search using distance metrics like cosine similarity, dot product, or Euclidean distance. They are essential for RAG pipelines, powering semantic retrieval of relevant documents based on query meaning rather than keyword matching.",
        "metadata": {"source": "vector_db_concepts", "topic": "data_infrastructure"},
    },
    {
        "id": "doc_otel_001",
        "content": "OpenTelemetry is an open-source observability framework that provides APIs for collecting traces, metrics, and logs. When integrated with ADK, it captures spans for each agent invocation, tool call, and LLM interaction, enabling granular performance analysis and debugging of multi-agent systems.",
        "metadata": {"source": "observability_guide", "topic": "monitoring"},
    },
    {
        "id": "doc_state_mgmt_001",
        "content": "ADK session state uses a key-value scratchpad with prefix-based scoping. Keys without a prefix are session-scoped and lost when the conversation ends. Keys with user: prefix persist across sessions for the same user. Keys with app: prefix are shared across all users and sessions.",
        "metadata": {"source": "state_management", "topic": "agent_frameworks"},
    },
    {
        "id": "doc_cloud_trace_001",
        "content": "Google Cloud Trace is a distributed tracing system that collects latency data from applications. It integrates with OpenTelemetry to receive spans via the OTLP protocol. Traces show the full request lifecycle including parent-child relationships between spans, enabling identification of performance bottlenecks.",
        "metadata": {"source": "cloud_trace_guide", "topic": "monitoring"},
    },
]


def populate():
    """Populate the ChromaDB collection with sample documents."""
    client = chromadb.PersistentClient(path="./chroma_data")
    embedding_function = GoogleGeminiEmbeddingFunction(
        model_name="gemini-embedding-001",
        task_type="RETRIEVAL_DOCUMENT",
    )
    collection = client.get_or_create_collection(
        name="research_docs",
        embedding_function=embedding_function,
    )

    collection.upsert(
        ids=[doc["id"] for doc in SAMPLE_DOCUMENTS],
        documents=[doc["content"] for doc in SAMPLE_DOCUMENTS],
        metadatas=[doc["metadata"] for doc in SAMPLE_DOCUMENTS],
    )

    print(f"Successfully populated {len(SAMPLE_DOCUMENTS)} documents.")
    print(f"Collection now has {collection.count()} documents total.")


if __name__ == "__main__":
    populate()
Populate the vector database

With the script written, you can now load all eight documents into ChromaDB. The Gemini embedding model will convert each document into a vector as it's inserted.

  • Run the population script by running this command from your project root (the folder containing research_assistant/):
python -m research_assistant.populate_db

You should see this output:

Successfully populated 8 documents.
Collection now has 8 documents total.

What just happened?

ChromaDB called the Gemini embedding model for each of the 8 documents, converting their text into high-dimensional vectors. These vectors are now stored in the ./chroma_data directory on disk. When you query this collection later, ChromaDB will embed your query with the same model and find the closest vectors using distance calculations.

  • Confirm the data directory was created by running:
ls chroma_data

You should see ChromaDB's internal storage files listed in the output.

Seeing an API key error?

The GoogleGeminiEmbeddingFunction reads your API key from the GEMINI_API_KEY environment variable. Make sure this is set in your terminal session or exported from your .env file. You can set it temporarily by running export GEMINI_API_KEY=your-api-key-here in your terminal.

Help me fix my Gemini API key error when populating ChromaDB

Build and test the vector search tool

Your database is populated, but the agent still needs a way to query it. You'll create a tools.py module with a search function that the Retriever Agent will use in the next step.

  • Create a new file called tools.py inside your research_assistant/ folder.
  • Add the imports and the get_collection() helper function by pasting this code:
import chromadb
from chromadb.utils.embedding_functions import GoogleGeminiEmbeddingFunction


def get_collection():
    """Get or create the ChromaDB collection with Gemini embeddings."""
    client = chromadb.PersistentClient(path="./chroma_data")
    embedding_function = GoogleGeminiEmbeddingFunction(
        model_name="gemini-embedding-001",
        task_type="RETRIEVAL_DOCUMENT",
    )
    collection = client.get_or_create_collection(
        name="research_docs",
        embedding_function=embedding_function,
    )
    return collection

Why a separate get_collection() helper?

This function encapsulates all the ChromaDB connection setup in one place. The search tool (and any future tools) can call get_collection() without needing to know the database path, embedding model, or collection name.

  • Add the search_knowledge_base() function below get_collection() in the same file:
def search_knowledge_base(query: str) -> dict:
    """Searches the vector database for documents relevant to the query.

    Args:
        query: The search query to find relevant documents.

    Returns:
        A dictionary containing the retrieved documents and their metadata.
    """
    collection = get_collection()
    results = collection.query(
        query_texts=[query],
        n_results=5,
        include=["documents", "metadatas", "distances"],
    )

    if not results["documents"][0]:
        return {"status": "no_results", "message": "No relevant documents found."}

    formatted_results = []
    for i, doc in enumerate(results["documents"][0]):
        metadata = results["metadatas"][0][i] if results["metadatas"][0] else {}
        distance = results["distances"][0][i] if results["distances"][0] else None
        formatted_results.append(
            {
                "document_id": metadata.get("source", f"doc_{i}"),
                "content": doc,
                "relevance_score": round(1 - distance, 3) if distance else None,
            }
        )

    return {"status": "success", "results": formatted_results}

What does this function do?

  • collection.query() embeds the query string and finds the 5 most similar document vectors in the collection.
  • The distances field tells you how far each result is from the query vector. Lower distance means higher relevance.
  • The function converts raw distance into a relevance_score (1 minus distance) so higher numbers mean better matches. This makes results easier for the agent to interpret.
  • The detailed docstring is important because ADK reads it to understand when and how to call the tool.
  • Save tools.py.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before testing.

ⓧ I'd like to double check the full code

import chromadb
from chromadb.utils.embedding_functions import GoogleGeminiEmbeddingFunction


def get_collection():
    """Get or create the ChromaDB collection with Gemini embeddings."""
    client = chromadb.PersistentClient(path="./chroma_data")
    embedding_function = GoogleGeminiEmbeddingFunction(
        model_name="gemini-embedding-001",
        task_type="RETRIEVAL_DOCUMENT",
    )
    collection = client.get_or_create_collection(
        name="research_docs",
        embedding_function=embedding_function,
    )
    return collection


def search_knowledge_base(query: str) -> dict:
    """Searches the vector database for documents relevant to the query.

    Args:
        query: The search query to find relevant documents.

    Returns:
        A dictionary containing the retrieved documents and their metadata.
    """
    collection = get_collection()
    results = collection.query(
        query_texts=[query],
        n_results=5,
        include=["documents", "metadatas", "distances"],
    )

    if not results["documents"][0]:
        return {"status": "no_results", "message": "No relevant documents found."}

    formatted_results = []
    for i, doc in enumerate(results["documents"][0]):
        metadata = results["metadatas"][0][i] if results["metadatas"][0] else {}
        distance = results["distances"][0][i] if results["distances"][0] else None
        formatted_results.append(
            {
                "document_id": metadata.get("source", f"doc_{i}"),
                "content": doc,
                "relevance_score": round(1 - distance, 3) if distance else None,
            }
        )

    return {"status": "success", "results": formatted_results}

Now test that vector search actually works. You'll run a quick Python command to query the database and confirm it returns relevant documents.

  • Test a semantic search query by running this command:
python -c "from research_assistant.tools import search_knowledge_base; import json; print(json.dumps(search_knowledge_base('How does RAG reduce hallucinations?'), indent=2))"

You should see a JSON response with "status": "success" and a list of results. The top result should be the RAG architecture document since it directly discusses how RAG reduces hallucinations. Each result includes a relevance_score between 0 and 1.

Why is semantic search powerful here?

Your query "How does RAG reduce hallucinations?" matches the RAG document even though the exact phrase doesn't appear in it. The embedding model understands meaning, not just keywords. This is why vector search is essential for building AI research assistants.

Getting empty results or an error?

Make sure you ran python -m research_assistant.populate_db successfully first. The chroma_data/ directory must exist in the same folder you're running the test command from.

If you see a ModuleNotFoundError, make sure you're running the command from the project root directory (the parent of research_assistant/), not from inside the research_assistant/ folder itself.

Help me debug my vector search test

Your knowledge base is ready and searchable. Next up, you'll build the multi-agent system that uses this search tool to retrieve context and synthesize grounded answers.

Build the Multi-Agent System

Your vector database is loaded with documents and ready to answer semantic queries. Now it is time to build the brain of the system.

Right now, your agent.py has a single greeting agent. In this step, you will replace it with three specialized agents that work together using hierarchical delegation. One retrieves documents, one synthesizes answers, and a coordinator routes the work between them.

In this step, get ready to:
  • Create the Retriever Agent with a vector search tool.
  • Create the Synthesizer Agent with citation-focused instructions.
  • Wire them together under a Coordinator Agent and test the multi-agent system.
Create the Retriever Agent

The retriever is the first specialist in your multi-agent team. Its only job is to take a user question, call the search_knowledge_base tool you built in the previous step, and return the raw results without modification.

  • Open research_assistant/agent.py in VS Code.
  • Select all the existing code and delete it.
  • Replace it with the following imports and retriever agent definition:
from google.adk.agents import Agent, SequentialAgent
from google.adk.tools import google_search

# Import the vector search tool from tools.py
from .tools import search_knowledge_base


# The retriever agent specializes in searching the knowledge base
retriever_agent = Agent(
    name="retriever_agent",
    model="gemini-2.5-flash",
    description="Searches the knowledge base for relevant documents based on a user query.",
    instruction="""You are a retrieval specialist. When given a question, use the 
search_knowledge_base tool to find relevant documents. Return all retrieved 
context exactly as provided by the tool. Do not summarize or modify the results.""",
    tools=[search_knowledge_base],
)

What does this code do?

The imports bring in Agent and SequentialAgent from ADK (you will use SequentialAgent in the Secret Mission). The from .tools import search_knowledge_base line connects the vector search tool you built in the previous step.

The retriever_agent is an ADK Agent with a single tool. Its instruction tells it to return results exactly as-is, so the synthesizer receives raw context to work with.

You will not see a change yet. The next chunk adds the second agent that consumes this retriever's output.

Create the Synthesizer Agent

The synthesizer is your answer-writing specialist. It receives the documents the retriever found and produces a grounded, cited response. Its instructions strictly forbid fabricating information not present in the context.

  • In agent.py, add the following synthesizer agent definition below the retriever_agent:
# The synthesizer agent produces grounded answers from retrieved context
synthesizer_agent = Agent(
    name="synthesizer_agent",
    model="gemini-2.5-flash",
    description="Synthesizes a grounded answer from retrieved document context.",
    instruction="""You are a research synthesizer. You will receive context from 
retrieved documents. Your job is to produce a clear, well-structured answer that:
1. Only uses information present in the provided context
2. Cites which document each piece of information comes from
3. Explicitly states when the context does not contain enough information to fully answer
4. Never fabricates or assumes information not in the context

Format citations as [Source: document_id].""",
)

What does this code do?

The synthesizer has no tools. It only receives text from the coordinator. Its instruction enforces four rules: only use provided context, cite sources, admit gaps, and never fabricate.

The citation format [Source: document_id] references the document_id field your search_knowledge_base tool returns. This creates a traceable chain from answer back to source document.

You will not see a change yet. The next chunk creates the coordinator that ties both agents together.

Create the Coordinator Agent and test the system

The coordinator is the root agent that receives user questions and decides which specialist to delegate to. In ADK, you pass sub-agents via the sub_agents parameter. The coordinator's instruction prompt tells it the order: retriever first, then synthesizer.

How does ADK delegation work?

ADK does not automatically route to sub-agents based on their description alone. The parent agent's instruction prompt must explicitly describe when and how to delegate. Each sub-agent's description field helps the LLM understand what each specialist does, but the routing logic lives in the coordinator's instructions.

  • In agent.py, add the coordinator agent below synthesizer_agent:
# The coordinator routes questions to retriever, then synthesizer
root_agent = Agent(
    name="research_coordinator",
    model="gemini-2.5-flash",
    description="Coordinates a multi-agent research assistant that retrieves and synthesizes information from a knowledge base.",
    instruction="""You are a research coordinator. When a user asks a question:
1. First, delegate to retriever_agent to search the knowledge base
2. Then, delegate to synthesizer_agent to produce a grounded answer from the retrieved context
3. Present the synthesizer's answer to the user

For follow-up questions, consider the conversation history stored in state.
If the user asks about preferences, store them in state with the user: prefix.

Always route through retriever first, then synthesizer. Never answer directly.""",
    sub_agents=[retriever_agent, synthesizer_agent],
)

What does this code do?

The root_agent variable name is important. ADK looks for root_agent as the entry point when you run the project.

The sub_agents=[retriever_agent, synthesizer_agent] parameter registers both specialists with the coordinator. The instruction explicitly says "Never answer directly" to force all questions through the retrieval pipeline.

  • Save agent.py.
  • Launch the ADK web UI to test your multi-agent system by running:
adk web --port 8000
  • Open your browser to http://localhost:8000.
  • Select research_coordinator from the agent dropdown at the top.
  • Type a question like What is RAG? and press Enter.

You should see the agent respond with a grounded answer that references your knowledge base documents. The response should include citations in the format [Source: rag_architecture] or similar.

Agent not responding or giving errors?

  • If you see an import error for search_knowledge_base, make sure tools.py is saved in the same research_assistant/ folder.
  • If you see ModuleNotFoundError: No module named 'chromadb', confirm your virtual environment is activated before running adk web.
  • If the agent answers without citations, try asking a more specific question like What is Cloud Run? to match your knowledge base documents.

Help me debug my ADK multi-agent system

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and your agent is responding with cited answers.

ⓧ I'd like to double check the full code

Here is the complete agent.py file. Compare yours against it:

from google.adk.agents import Agent, SequentialAgent
from google.adk.tools import google_search

from .tools import search_knowledge_base


retriever_agent = Agent(
    name="retriever_agent",
    model="gemini-2.5-flash",
    description="Searches the knowledge base for relevant documents based on a user query.",
    instruction="""You are a retrieval specialist. When given a question, use the 
search_knowledge_base tool to find relevant documents. Return all retrieved 
context exactly as provided by the tool. Do not summarize or modify the results.""",
    tools=[search_knowledge_base],
)

synthesizer_agent = Agent(
    name="synthesizer_agent",
    model="gemini-2.5-flash",
    description="Synthesizes a grounded answer from retrieved document context.",
    instruction="""You are a research synthesizer. You will receive context from 
retrieved documents. Your job is to produce a clear, well-structured answer that:
1. Only uses information present in the provided context
2. Cites which document each piece of information comes from
3. Explicitly states when the context does not contain enough information to fully answer
4. Never fabricates or assumes information not in the context

Format citations as [Source: document_id].""",
)

root_agent = Agent(
    name="research_coordinator",
    model="gemini-2.5-flash",
    description="Coordinates a multi-agent research assistant that retrieves and synthesizes information from a knowledge base.",
    instruction="""You are a research coordinator. When a user asks a question:
1. First, delegate to retriever_agent to search the knowledge base
2. Then, delegate to synthesizer_agent to produce a grounded answer from the retrieved context
3. Present the synthesizer's answer to the user

For follow-up questions, consider the conversation history stored in state.
If the user asks about preferences, store them in state with the user: prefix.

Always route through retriever first, then synthesizer. Never answer directly.""",
    sub_agents=[retriever_agent, synthesizer_agent],
)

You now have a working multi-agent research assistant. Next up, you will add session state management so the system remembers context across turns in a conversation.

Add Session State Management

Your multi-agent system can answer questions, but each interaction exists in isolation. Ask a follow-up question and the coordinator has no memory of what it just retrieved or synthesized.

In this step, you will learn how ADK session state gives your agents a scratchpad to track conversation context. You will explore how state scoping with key prefixes controls what persists and where, then test multi-turn conversations to verify the coordinator remembers context.

In this step, get ready to:
  • Understand ADK session state and scoped key prefixes.
  • Configure user-scoped state with the user: prefix for cross-session preferences.
  • Test multi-turn conversations to verify state persists between interactions.
Understand ADK session state and scoped key prefixes

ADK provides a built-in state mechanism through session.state. This is a key-value scratchpad attached to each conversation session, separate from the full event history.

What is session state in ADK?

Session state is a serializable key-value store that agents read and write during a conversation. Keys are strings and values can be strings, numbers, booleans, or simple lists.

ADK uses key prefixes to control scope and persistence. The prefix determines whether data lives within a single conversation, follows a user across conversations, or is shared globally across the entire app.

ADK defines four scoping levels based on the key prefix:

  • No prefix (session-scoped): Data lives only in the current session. Example: session.state['last_query'] = 'What is RAG?'.
  • user: prefix (user-scoped): Data persists across all sessions for the same user. Example: session.state['user:preferred_detail'] = 'concise'.
  • app: prefix (app-scoped): Data is shared across all users and all sessions for the application. Example: session.state['app:api_endpoint'] = '/v1'.
  • temp: prefix (invocation-scoped): Data is discarded after the current invocation completes. Useful for intermediate calculations within a single turn.

How does the agent access state without explicit Python code?

ADK's LLM-based agents interact with state through their instruction prompt. When you tell the coordinator to "consider the conversation history stored in state" or "store preferences with the user: prefix," the LLM uses ADK's built-in state mechanism to read and write values automatically.

For programmatic state access (in tools or callbacks), you use tool_context.state['key'] = value. ADK tracks these changes in the event delta and persists them through the SessionService.

  • Open research_assistant/agent.py in VS Code.
  • Find the root_agent definition and locate the instruction parameter. Look at these two lines:
For follow-up questions, consider the conversation history stored in state.
If the user asks about preferences, store them in state with the user: prefix.

What do these instruction lines do?

The first line tells the coordinator to look at session state when handling follow-up questions. This means it will check for previously stored queries, retrieved documents, and synthesis results before deciding how to route the next request.

The second line instructs the coordinator to save user preferences (like detail level or topic focus) using the user: prefix. This means those preferences will persist across different sessions for the same user.

Understand user-scoped state for cross-session preferences

Session-scoped state (no prefix) is lost when the conversation ends. For preferences that should follow a user across conversations, ADK uses the user: prefix.

Your coordinator's instruction already handles both patterns. Here is how state flows during a conversation with your research assistant:

  • User asks "What is RAG?" The coordinator stores the query and retrieved results in session-scoped state (no prefix). These values are available for follow-up questions in the same conversation.
  • User says "I prefer concise answers." The coordinator stores this with the user: prefix. Next time this user starts a new session, the preference is still available.
  • User asks a follow-up like "Tell me more about embeddings." The coordinator checks state for prior context and builds on the previous answer.

Why does InMemorySessionService still work for testing?

The adk web command uses InMemorySessionService by default. This stores state in memory during the running process. User-scoped state works within the same running server, but is lost when you restart the process.

For production persistence, you would swap to DatabaseSessionService or VertexAiSessionService. For this project, in-memory is sufficient to demonstrate the patterns.

Test multi-turn conversations through the ADK web UI

Time to verify that state actually persists between conversation turns. You will run a multi-turn conversation and confirm the coordinator uses prior context for follow-up questions.

  • Make sure you are in the parent directory that contains the research_assistant folder (not inside it).
  • Start the ADK web server by running this command:
adk web --port 8000
  • Open your browser to http://localhost:8000.
  • Select research_assistant from the agent dropdown if it is not already selected.

Now run a multi-turn test to verify session-scoped state works:

  • Type your first question: What is RAG and how does it reduce hallucinations?
  • Wait for the full response with citations from your knowledge base.
  • Type a follow-up that references the previous answer: How does that relate to vector databases?
  • Verify that the coordinator connects the follow-up to the previous RAG context. The response should reference both RAG and vector databases together.

Next, test user-scoped state by setting a preference:

  • Type: I prefer very detailed, technical answers with examples.
  • The coordinator should acknowledge and store this preference.
  • Ask another question: What is OpenTelemetry?
  • Verify the response is more detailed than your earlier answers, reflecting the stored preference.

What should I see?

The follow-up question should produce a response that builds on the previous answer. The coordinator recognizes that "that" in your follow-up refers to RAG from the prior turn. After setting a preference, subsequent answers should reflect that preference in their depth and style.

Follow-up answers seem unrelated to previous context?

Make sure you are sending messages within the same session (same chat thread in the ADK web UI). Starting a new session clears session-scoped state.

Check that the coordinator's instruction includes the line about considering "conversation history stored in state." Without this instruction, the LLM will not look at prior state.

help me debug why my ADK agent doesn't remember previous conversation context

✔️ Awesome, I've got everything!

Great. Double check you have saved agent.py and confirmed multi-turn conversations work in the web UI.

ⓧ I'd like to double check the full code

Here is the full agent.py file. Your code should match this exactly:

from google.adk.agents import Agent, SequentialAgent
from google.adk.tools import google_search

from .tools import search_knowledge_base


retriever_agent = Agent(
    name="retriever_agent",
    model="gemini-2.5-flash",
    description="Searches the knowledge base for relevant documents based on a user query.",
    instruction="""You are a retrieval specialist. When given a question, use the 
search_knowledge_base tool to find relevant documents. Return all retrieved 
context exactly as provided by the tool. Do not summarize or modify the results.""",
    tools=[search_knowledge_base],
)

synthesizer_agent = Agent(
    name="synthesizer_agent",
    model="gemini-2.5-flash",
    description="Synthesizes a grounded answer from retrieved document context.",
    instruction="""You are a research synthesizer. You will receive context from 
retrieved documents. Your job is to produce a clear, well-structured answer that:
1. Only uses information present in the provided context
2. Cites which document each piece of information comes from
3. Explicitly states when the context does not contain enough information to fully answer
4. Never fabricates or assumes information not in the context

Format citations as [Source: document_id].""",
)

root_agent = Agent(
    name="research_coordinator",
    model="gemini-2.5-flash",
    description="Coordinates a multi-agent research assistant that retrieves and synthesizes information from a knowledge base.",
    instruction="""You are a research coordinator. When a user asks a question:
1. First, delegate to retriever_agent to search the knowledge base
2. Then, delegate to synthesizer_agent to produce a grounded answer from the retrieved context
3. Present the synthesizer's answer to the user

For follow-up questions, consider the conversation history stored in state.
If the user asks about preferences, store them in state with the user: prefix.

Always route through retriever first, then synthesizer. Never answer directly.""",
    sub_agents=[retriever_agent, synthesizer_agent],
)

Your research assistant now remembers conversation context and user preferences across turns. Next up, you will deploy this entire multi-agent system to Google Cloud Run so it is accessible as a production service.

Deploy to Google Cloud Run

Your multi-agent research assistant works locally. Now it is time to make it accessible from anywhere by deploying it to Google Cloud Run.

Cloud Run is a fully managed serverless platform that automatically builds a container from your source code and scales it based on demand. It only charges when your service is handling requests, so it costs essentially nothing when idle.

In this step, get ready to:
  • Create a requirements.txt and update your .env for production deployment.
  • Deploy the agent to Cloud Run using a single command.
  • Test the live agent by sending curl requests to its public URL.
Create the requirements file

Cloud Run needs a requirements.txt in your project to know which Python packages to install when building the container. This file lists every dependency your agent needs to run in production.

  • Create a new file called requirements.txt in your project root (the folder that contains the research_assistant/ directory).
  • Add the following dependencies:
google-adk>=1.28.0
chromadb>=1.5.9
google-genai
opentelemetry-exporter-gcp-logging
opentelemetry-exporter-otlp-proto-grpc
opentelemetry-instrumentation-vertexai

What are these packages?

  • google-adk is the Agent Development Kit framework that runs your agents.
  • chromadb is the vector database for your knowledge base.
  • google-genai provides the Gemini embedding function for ChromaDB.
  • The three opentelemetry packages handle tracing and observability. You will configure these in the next step.
  • Save requirements.txt.
  • Confirm the file is in the right place by running:
ls requirements.txt

You should see requirements.txt listed in the output.

File not found?

Make sure you created requirements.txt in the parent folder that contains research_assistant/. It should be at the same level as the research_assistant/ directory, not inside it.

help me figure out where to place my requirements.txt for Cloud Run deployment

Update your .env for Vertex AI

When running on Cloud Run, your agent uses Vertex AI as the model backend instead of a direct API key. Your .env file already has the right structure, but you need to replace the placeholder project ID with your actual GCP project ID.

  • In research_assistant/.env, find this line:
GOOGLE_CLOUD_PROJECT=your-project-id-here
  • Replace your-project-id-here with your actual GCP project ID, so the line looks like this:
GOOGLE_CLOUD_PROJECT=[[GCP_PROJECT_ID="your-gcp-project-id"]]

Find your project ID

Run gcloud config get project in your terminal to see the project ID currently configured in your CLI. Use that exact value.

  • Save research_assistant/.env.

Why Vertex AI instead of an API key?

On Cloud Run, your service authenticates using the attached service account rather than an API key stored in the environment. This is more secure for production workloads because the credentials are managed by Google Cloud's IAM system.

Deploy to Cloud Run

Cloud Run's source-based deployment handles everything for you. It builds a container image from your code, pushes it to Artifact Registry, and starts the service. No Dockerfile needed.

  • First, grant your default compute service account permission to use Cloud Build by running this command:
gcloud projects add-iam-policy-binding [[GCP_PROJECT_ID="your-gcp-project-id"]] \
  --member="serviceAccount:$(gcloud projects describe [[GCP_PROJECT_ID="your-gcp-project-id"]] --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role="roles/cloudbuild.builds.builder"

What does this command do?

Cloud Run uses Cloud Build to turn your source code into a container image. This command gives the default compute service account the roles/cloudbuild.builds.builder role so it has permission to run builds.

  • Make sure you are in the parent directory that contains research_assistant/ and requirements.txt.
  • Deploy to Cloud Run by running this command:
gcloud run deploy research-assistant --source . --region=us-central1

What happens during deployment?

This command uploads your source code, builds a container image using Cloud Build, pushes it to Artifact Registry, and deploys it as a Cloud Run service named research-assistant in us-central1.

The deployment takes 2-5 minutes. You will see progress messages in your terminal as each phase completes.

  • When prompted to enable the Artifact Registry API, press y and then Enter.
  • When prompted to create a repository in the specified region, press y and then Enter.
  • When prompted to allow unauthenticated invocations, press y and then Enter.

On success, the terminal displays your service URL. It looks like https://research-assistant-XXXXXXXXXX-uc.a.run.app.

Seeing a permissions error?

If you see a 403 or permission denied error, make sure the Cloud Build IAM binding from the previous command has propagated. IAM changes can take up to 60 seconds. Wait a minute and try the deploy command again.

help me debug a permissions error when deploying to Cloud Run with gcloud run deploy

Test the deployed agent with curl

Your agent is now live on the internet. Let's verify it works by creating a session and sending a query using curl.

  • Store your service URL in an environment variable by running this command (replace the URL with yours):
export APP_URL="[[CLOUD_RUN_URL="https://research-assistant-XXXXXXXXXX-uc.a.run.app"]]"
  • Verify the agent app is listed by running:
curl -X GET $APP_URL/list-apps

You should see a JSON response containing research_assistant in the output.

  • Create a session for your agent by running:
curl -X POST $APP_URL/apps/research_assistant/users/test_user/sessions/test_session_1 \
  -H "Content-Type: application/json" \
  -d '{}'

What does this do?

This creates a new session with user ID test_user and session ID test_session_1. ADK requires an active session before you can send messages to the agent.

  • Query the agent by running:
curl -X POST $APP_URL/run \
  -H "Content-Type: application/json" \
  -d '{
    "appName": "research_assistant",
    "userId": "test_user",
    "sessionId": "test_session_1",
    "newMessage": {
      "role": "user",
      "parts": [{"text": "What is RAG and how does it reduce hallucinations?"}]
    }
  }'

You should see a JSON response containing the agent's grounded answer with citations from your knowledge base documents.

What should I see?

The response contains events from the agent execution. Look for content that references your sample documents with [Source: rag_architecture] or similar citation markers. This confirms the retriever searched ChromaDB and the synthesizer produced a grounded answer.

Getting a 422 or empty response?

A 422 Field required error means the request body format is incorrect. Make sure you are using newMessage with a parts array, not a plain text message field.

If you get a connection refused or 404, double-check that your APP_URL variable is set correctly. Run echo $APP_URL to verify.

help me troubleshoot my ADK agent deployed on Cloud Run that is not responding correctly to curl requests

✔️ Awesome, I've got everything!

Great. Double check you have saved both requirements.txt and research_assistant/.env.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file:

google-adk>=1.28.0
chromadb>=1.5.9
google-genai
opentelemetry-exporter-gcp-logging
opentelemetry-exporter-otlp-proto-grpc
opentelemetry-instrumentation-vertexai

Here is the complete research_assistant/.env file:

GOOGLE_GENAI_USE_VERTEXAI=TRUE
GOOGLE_CLOUD_PROJECT=your-project-id-here
GOOGLE_CLOUD_LOCATION=us-central1

Your multi-agent research assistant is now live on the internet, answering questions from your knowledge base. Next up, you will add observability with OpenTelemetry so you can trace every agent interaction in Google Cloud Trace.

Enable OpenTelemetry Tracing

Your multi-agent system is deployed and answering questions from a real knowledge base. But how do you know what is happening inside the system when a request flows through the coordinator, retriever, and synthesizer?

In this step, you will add production-grade observability using OpenTelemetry. ADK has built-in tracing that captures every agent invocation, tool call, and LLM interaction as spans. You will configure these spans to export to Google Cloud Trace, giving you a complete picture of latency, token usage, and delegation flow.

In this step, get ready to:
  • Install the OpenTelemetry packages and enable the Telemetry API.
  • Grant tracing IAM roles to your Cloud Run service account and redeploy.
  • View agent traces in the Google Cloud Trace console.
Install OpenTelemetry packages and enable the Telemetry API

ADK versions 1.17.0+ include built-in OpenTelemetry support. The tracing packages in your requirements.txt handle exporting trace data, metrics, and logs to Google Cloud.

  • Install the OpenTelemetry packages in your virtual environment by running:
pip install opentelemetry-exporter-gcp-logging opentelemetry-exporter-otlp-proto-grpc opentelemetry-instrumentation-vertexai

What do these packages do?

  • opentelemetry-exporter-gcp-logging sends log data to Cloud Logging.
  • opentelemetry-exporter-otlp-proto-grpc exports trace spans to the Telemetry (OTLP) API using gRPC.
  • opentelemetry-instrumentation-vertexai automatically instruments Vertex AI calls, capturing LLM request/response details as span attributes.
  • Enable the Telemetry and Cloud Trace APIs for your project by running:
gcloud services enable telemetry.googleapis.com cloudtrace.googleapis.com
  • Verify the APIs are enabled by running:
gcloud services list --enabled --filter="NAME:(telemetry OR cloudtrace)"

You should see both telemetry.googleapis.com and cloudtrace.googleapis.com in the output.

APIs not showing up?

Make sure your gcloud is pointed at the correct project. Run gcloud config get-value project to confirm. If it shows the wrong project, run gcloud config set project YOUR_PROJECT_ID with your actual project ID.

help me enable the Telemetry and Cloud Trace APIs on my GCP project

Now create the OpenTelemetry environment configuration. ADK reads these variables to activate its built-in tracing.

  • Create a file called opentelemetry.env in your project root (the same folder as requirements.txt) with the following content:
OTEL_SERVICE_NAME='research-assistant'
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED='true'
OTEL_SEMCONV_STABILITY_OPT_IN='gen_ai_latest_experimental'
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT='EVENT_ONLY'
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='false'

What do these environment variables control?

  • OTEL_SERVICE_NAME labels all spans with your service name so you can filter traces.
  • OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED enables automatic log correlation with traces.
  • OTEL_SEMCONV_STABILITY_OPT_IN uses the latest generative AI semantic conventions for span attributes.
  • OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT captures prompts and responses as span events rather than inline attributes.
  • ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is set to false to avoid duplicating content that is already captured as events.
  • Test the tracing locally by running the ADK web server with the --otel_to_cloud flag:
env $(cat opentelemetry.env | xargs) adk web --otel_to_cloud

You should see the ADK web server start on localhost:8000. The --otel_to_cloud flag tells ADK to export telemetry data to your Google Cloud project.

  • Open http://localhost:8000 in your browser and send a test query like "What is RAG?" to generate trace data.
  • Stop the local server with Ctrl+C after testing.

Seeing a permission denied error?

Your local gcloud credentials need permission to write traces. Run gcloud auth application-default login to refresh your Application Default Credentials.

help me fix permission denied errors when running ADK with --otel_to_cloud

Grant IAM roles to the Cloud Run service account and redeploy

Your Cloud Run service uses a service account to authenticate with other Google Cloud services. For tracing to work in production, this service account needs permission to write traces, logs, and metrics.

  • Find your project number by running:
gcloud projects describe $(gcloud config get-value project) --format="value(projectNumber)"

The default Cloud Run service account follows the format PROJECT_NUMBER-compute@developer.gserviceaccount.com. Note the number that is returned.

  • Grant the Cloud Telemetry Traces Writer role by running:
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member="serviceAccount:$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role="roles/telemetry.tracesWriter"
  • Grant the Logs Writer role by running:
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member="serviceAccount:$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role="roles/logging.logWriter"
  • Grant the Monitoring Metric Writer role by running:
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member="serviceAccount:$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role="roles/monitoring.metricWriter"

Why three separate roles?

OpenTelemetry separates observability into three signals: traces, logs, and metrics. Each signal writes to a different Google Cloud service (Cloud Trace, Cloud Logging, Cloud Monitoring), and each service requires its own IAM role.

The roles/telemetry.tracesWriter role allows writing trace spans via the OTLP API. The roles/logging.logWriter role allows writing log entries. The roles/monitoring.metricWriter role allows writing metric data.

You should see a policy binding confirmation after each command.

  • Redeploy your service to Cloud Run with the tracing environment variables by running:
gcloud run deploy research-assistant \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars="OTEL_SERVICE_NAME=research-assistant,OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true,OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY,ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false"

What changed in this deployment?

The --set-env-vars flag injects the OpenTelemetry configuration directly into the Cloud Run container environment. ADK detects these variables at startup and automatically begins exporting telemetry to your Google Cloud project.

Wait for the deployment to complete. You should see the service URL printed in the terminal output.

Deployment failing?

If you see a build error, make sure all three OpenTelemetry packages are in your requirements.txt. Run cat requirements.txt and confirm you see opentelemetry-exporter-gcp-logging, opentelemetry-exporter-otlp-proto-grpc, and opentelemetry-instrumentation-vertexai listed.

help me fix a Cloud Run deployment error when adding OpenTelemetry tracing

Inspect traces in Google Cloud Trace

With tracing enabled and the service redeployed, every request to your agent now generates spans that flow into Cloud Trace. Time to generate some traces and explore the observability data.

  • Send a few queries to your deployed agent to generate trace data. Run this curl command:
curl -X POST $(gcloud run services describe research-assistant --region us-central1 --format='value(status.url)')/run \
  -H "Content-Type: application/json" \
  -d '{"appName": "research_coordinator", "userId": "u_trace_test", "sessionId": "s_trace_1", "newMessage": {"role": "user", "parts": [{"text": "What is RAG and how does it reduce hallucinations?"}]}}'
  • Send a second query to generate more trace data:
curl -X POST $(gcloud run services describe research-assistant --region us-central1 --format='value(status.url)')/run \
  -H "Content-Type: application/json" \
  -d '{"appName": "research_coordinator", "userId": "u_trace_test", "sessionId": "s_trace_2", "newMessage": {"role": "user", "parts": [{"text": "How does OpenTelemetry work with ADK?"}]}}'

Wait about 1-2 minutes for the traces to propagate to Cloud Trace.

  • Open the Google Cloud Console in your browser.
  • In the search bar at the top, type Trace Explorer and select Trace Explorer from the results.
  • You should see a list of recent traces. Click on any trace to expand it.

Inside a trace, you will see the full span hierarchy showing how your request flowed through the system.

What should I see in Cloud Trace?

Each trace shows a waterfall of spans representing the request lifecycle. Look for these spans:

  • A root span for the incoming HTTP request to Cloud Run.
  • Child spans for each agent invocation (research_coordinator, retriever_agent, synthesizer_agent).
  • Spans for tool calls (search_knowledge_base) showing how long the vector search took.
  • LLM interaction spans with attributes showing token counts (input tokens, output tokens) and latency.

Click the Logs & Events tab on any span to see the captured prompts and responses (stored as events per your OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT setting).

  • Click on an LLM span (look for spans with "gemini" in the name).
  • In the span details panel, look at the Attributes section. You should see gen_ai.usage.input_tokens and gen_ai.usage.output_tokens showing exactly how many tokens each LLM call consumed.
  • Note the duration of each span in the waterfall view. This shows where latency is concentrated in your multi-agent pipeline.

Not seeing any traces?

Traces can take 2-3 minutes to appear after sending requests. If you still do not see them after waiting, check that you granted the roles/telemetry.tracesWriter role to the correct service account. Run gcloud run services describe research-assistant --region us-central1 --format='value(spec.template.spec.serviceAccountName)' to confirm which service account your Cloud Run service uses.

help me troubleshoot why my ADK traces are not appearing in Google Cloud Trace

You now have full observability into your multi-agent system. Every query generates a trace that shows exactly how the coordinator delegates to the retriever and synthesizer, how long each step takes, and how many tokens are consumed. This is the same observability pattern used in production AI systems at scale.

Secret mission

Add a Critic Agent (Self-Reflection Pattern)

Your research assistant answers questions well, but how do you know it is not making things up? In production AI systems, self-reflection is the pattern that catches hallucinations before they reach users. Your mission: add a Critic agent that fact-checks the Synthesizer's output against the source documents.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Cloud Run services have no idle cost, so this project is essentially free to leave running within your GCP free trial credits.

Cost warning

Your GCP free trial provides $300 in credits for 90 days. Cloud Run only charges when handling requests (2M free requests per month). As long as you stay within the free trial, this project costs $0. However, if your free trial expires and you have billing activated, you could incur small charges from Vertex AI API calls or Cloud Build storage.

Resources you used:

  • Google Cloud project with billing enabled (covered by $300 free trial credits).
  • Cloud Run service (no cost when idle, charges only per request).
  • Artifact Registry repository (minimal storage cost for container image).
  • ChromaDB data stored locally in the chroma_data directory.
  • Python virtual environment and installed packages.

Keep everything running

No action needed. Choose this if you want to keep demonstrating your research assistant or continue building on it.

  • Your Cloud Run service stays deployed and accessible via its URL.
  • Cloud Run has zero idle cost. You only pay when the service handles requests.
  • Your $300 free trial credits cover any API usage during testing.

Pause - I'll come back to this later

Cloud Run services have no idle cost, so there is nothing to pause. Your service stays deployed but costs nothing when not receiving requests.

  • Deactivate your Python virtual environment by running deactivate in your terminal.
  • Close VS Code and any open terminal sessions to free up local memory.
  • Your Cloud Run service, ChromaDB data, and project files remain intact for when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Delete your Cloud Run service by running:
gcloud run services delete research-assistant --region=us-central1
  • Confirm the deletion when prompted.
  • To remove all GCP resources at once, delete the entire project from the Google Cloud Console by navigating to IAM & Admin then Manage Resources. Select your project and click Delete.
  • Delete your local project folder and the chroma_data directory from your Desktop.
  • Remove the Python virtual environment by deleting the .venv folder inside your project directory (if not already deleted with the project folder).

That's a Wrap!

That's a Wrap!

Nice work! You just built and deployed a production-grade multi-agent research assistant on Google Cloud. This is the exact architecture that Google's Forward Deployed Engineers build for enterprise customers.

Here's what you learned:

  • Built a multi-agent system using Google's ADK with hierarchical delegation between a coordinator, retriever, and synthesizer agent.
  • Created a RAG pipeline powered by a ChromaDB vector database with Gemini embeddings, and deployed the entire system to Cloud Run as a serverless service.
  • Configured OpenTelemetry tracing with spans visible in Google Cloud Trace for production-grade observability of agent interactions, tool calls, and LLM latency.
  • Secret Mission: Added a Critic agent implementing the self-reflection pattern to fact-check answers against source documents and catch hallucinations.

Ready to quiz yourself?