Design a Production-Grade RAG System

Design a complete RAG system architecture on Google Cloud with diagrams.

Introduction

30 Second Summary

Every time you ask an AI chatbot a question about your company's data, something has to find the right information and feed it to the language model. The system that makes this possible is called Retrieval-Augmented Generation (RAG). Designing one that actually works at scale is the difference between a demo and a production system.

In this project, you will design a complete system architecture for a production-grade RAG system on Google Cloud. You will produce a submission-ready design document covering ingestion, retrieval, orchestration, optimization, and a professional architecture diagram.

What You'll Build

A comprehensive system design document for a production-grade RAG system built on Google Cloud, complete with technology choices, optimization strategies, and a polished Excalidraw architecture diagram.

By the end of this project, you'll have:

  • A written design document covering all 5 required sections: Ingestion Pipeline, Retrieval Strategy, Orchestration Layer, Optimization, and System Diagram.
  • A production-quality architecture diagram showing the full data path from user query through cache, vector database, LLM, and back to final response.
  • Research-backed justifications for every architecture decision, demonstrating deep understanding of the RAG design space.
  • Secret Mission: Add a "Failure Modes & Resilience" section covering graceful degradation, cache invalidation strategies, and index staleness handling.

Are there any prerequisites?

You should have a basic understanding of how LLMs generate text and general familiarity with cloud computing concepts. No coding is required for this project.

Set Up Your Workspace

Before you dive into writing about ingestion pipelines and caching strategies, you need a clean workspace.

In this step, you will open your diagramming tool and create a skeleton document with all five required sections. This gives you a clear structure to fill in as you work through the remaining steps.

In this step, get ready to:
  • Open Excalidraw for your architecture diagram.
  • Create a design document with 5 section headings.
  • Review the challenge requirements to confirm coverage.
Open Excalidraw in your browser

Excalidraw is a free, browser-based diagramming tool. It produces clean architecture diagrams without requiring an account or installation.

You should see a blank white canvas with a toolbar across the top offering shape, line, and text tools.

Why Excalidraw?

Excalidraw gives you drag-and-drop shapes, connectors, and text labels in a hand-drawn style that keeps architecture diagrams readable. No sign-up, no payment, no software to install.

Save your work regularly

Excalidraw saves drawings in your browser's local storage, but this can be cleared unexpectedly. Use Ctrl+S (Windows) or Cmd+S (macOS) periodically to export your diagram to a file on your computer.

Create your design document skeleton

Your written design document needs five clearly labeled sections. Creating the skeleton now means you can focus on content in later steps without worrying about structure.

  • Open your preferred text editor. Google Docs, VS Code with a Markdown file, or Notion all work well.
  • Create a new document or file.
  • Add the following five section headings to your document:
# 1. Ingestion Pipeline

# 2. Retrieval Strategy

# 3. Orchestration Layer

# 4. Optimization

# 5. System Diagram

What do these sections cover?

  • Ingestion Pipeline describes how raw documents become searchable vector embeddings.
  • Retrieval Strategy covers how user queries find the most relevant document chunks.
  • Orchestration Layer explains the framework coordinating the full query-to-response flow.
  • Optimization documents caching, streaming, and latency reduction techniques.
  • System Diagram is where you will embed your finished Excalidraw architecture diagram.

Leave each section empty for now. You will fill them in during the next four steps.

✔️ Awesome, I've got everything!

Great. Double check you have saved your document with all five headings visible.

ⓧ I'd like to double check the full skeleton

Your document should contain exactly these five headings, in this order, with empty space beneath each one:

# 1. Ingestion Pipeline

# 2. Retrieval Strategy

# 3. Orchestration Layer

# 4. Optimization

# 5. System Diagram
Review the challenge requirements

The Gen AI Academy APAC System Design Challenge evaluates five components. Confirming your document maps to each one now prevents rework later.

  • Compare your five headings against the challenge criteria below:
  • Ingestion Pipeline (how data enters the system and becomes searchable).
  • Retrieval Strategy (how the system finds relevant context for a query).
  • Orchestration Layer (how the system manages the flow from query to response).
  • Optimization (how speed and cost are improved at scale).
  • System Diagram (a visual architecture showing the complete data path).

Each heading in your document maps one-to-one with a challenge component. You are ready to start writing.

What makes a strong submission?

The challenge rewards specific technology choices backed by reasoning, not vague descriptions. In the next steps, you will name exact Google Cloud services, cite latency benchmarks, and justify every decision.

Your workspace is ready. Next up, you will design the ingestion pipeline that transforms raw documents into searchable vector embeddings.

Design the Ingestion Pipeline

Your workspace is set up and ready to go. Now it's time to tackle the first and most foundational section of your design document: the ingestion pipeline.

The ingestion pipeline is responsible for transforming raw documents into searchable vector embeddings. Without a well-designed ingestion layer, the rest of your RAG system has nothing to retrieve. In this step, you'll describe how data flows from upload through parsing, chunking, embedding, and storage.

In this step, get ready to:
  • Write the data ingestion flow from document upload to embedding generation.
  • Describe the vector storage and indexing strategy.
  • Document the metadata approach for filtered retrieval.
Write the data ingestion flow

Your RAG system needs a way to get raw documents into a format that supports fast semantic search. This first subsection describes the end-to-end flow from upload to embedding.

  • Switch to your design document and find the Ingestion Pipeline section heading.
  • Write the following content under that heading:
## 1. Ingestion Pipeline

### Data Ingestion Flow

Raw documents (PDFs, web pages, databases) are uploaded to Google Cloud Storage.
When a new object is finalized in the bucket, it triggers a Pub/Sub message.
A Cloud Run function subscribes to this topic, picks up the message, and
begins processing the raw data.

The Cloud Run function performs two key operations:

1. **Parsing and Chunking**: The raw document is parsed into plain text, then
   chunked into semantically meaningful segments using recursive character
   splitting. Each chunk targets 512 tokens with a 50-token overlap between
   consecutive chunks. This overlap ensures that context is not lost at
   chunk boundaries.

2. **Embedding Generation**: Each chunk is sent to the Vertex AI Embeddings
   API, which returns a 768-dimensional vector embedding. These embeddings
   capture the semantic meaning of each chunk in a format optimized for
   similarity search.

What does this section describe?

This section maps the journey of a single document from raw upload to vector representation. Each component plays a specific role:

  • Google Cloud Storage acts as the landing zone for all incoming documents.
  • Pub/Sub decouples the upload event from the processing logic, enabling asynchronous and scalable ingestion.
  • Cloud Run provides a serverless compute layer that auto-scales with the volume of incoming documents.
  • Recursive character splitting preserves paragraph and sentence boundaries rather than cutting at arbitrary character counts. The 50-token overlap prevents context from being lost at chunk edges.
  • 768-dimensional embeddings from the Vertex AI Embeddings API compress each chunk's meaning into a fixed-length vector for fast similarity comparison.
  • Save your document after adding this section.
  • Re-read what you wrote and confirm that the flow moves clearly from upload to Pub/Sub to Cloud Run to chunking to embedding. Each stage should feel like a logical handoff.

Not sure what to write for the chunking details?

The key numbers to include are: 512 tokens per chunk and 50-token overlap. These are common production defaults for dense technical content. Recursive character splitting means the system tries paragraph breaks first, then sentences, then words, falling back to characters only as a last resort.

If you need more clarity, help me understand why 512 tokens with 50-token overlap is a good chunking strategy for RAG

.

Describe the vector storage and indexing

Once embeddings are generated, they need a home. This subsection covers where vectors live and how the system finds the most relevant ones quickly.

  • Below the data ingestion flow you just wrote, add the following content:
### Vector Storage and Indexing

Embeddings are stored in Vertex AI Vector Search, a fully managed vector
similarity-matching service. It builds a ScaNN-based approximate nearest
neighbor (ANN) index over the stored embeddings, enabling sub-linear search
time even at large scale.

Key design choices:

- **Distance metric**: Cosine similarity. This measures the angle between
  vectors rather than magnitude, making it ideal for text embeddings where
  direction encodes meaning.

- **Streaming updates**: The index supports streaming upserts, allowing new
  document chunks to become searchable without requiring a full index rebuild.
  As new documents arrive through the ingestion pipeline, their embeddings
  are upserted into the live index.

- **Autoscaling**: Deployed indexes are configured with `minReplicaCount`
  and `maxReplicaCount` to handle variable query load automatically.

Why Vertex AI Vector Search?

Vertex AI Vector Search uses Google's ScaNN (Scalable Nearest Neighbors) algorithm internally. ANN indexes trade a small amount of recall accuracy for massive speed gains over brute-force search. This is what makes it possible to search millions of embeddings in milliseconds.

Streaming upserts are critical for production systems. Without them, you would need to rebuild the entire index every time a new document arrives. This would create delays and downtime.

  • Save your document.
  • Check that you have three distinct design choices listed (distance metric, streaming updates, autoscaling). Each should explain both the what and the why.

Unsure about ScaNN or ANN?

ANN stands for Approximate Nearest Neighbor. Instead of comparing a query vector against every single stored vector (brute-force), ANN builds a data structure that narrows the search space. ScaNN is Google's specific implementation of this approach, optimized for high-dimensional vectors.

If this is still unclear, explain how approximate nearest neighbor search works and why it is faster than brute-force search

.

Document the metadata strategy

Vectors alone are not enough for precise retrieval. Metadata lets you filter results by source, date, or category before the LLM ever sees them.

  • Below the vector storage section, add the following content to complete your Ingestion Pipeline section:
### Metadata Strategy

Each chunk retains structured metadata stored alongside its embedding to
enable filtered retrieval. Metadata fields include:

- **Source document**: The original file name or URL the chunk was extracted
  from.
- **Page number**: For paginated documents (PDFs), the specific page.
- **Timestamp**: When the document was ingested, enabling recency filtering.
- **Category**: A classification tag (e.g., "policy", "technical",
  "financial") for domain-specific filtering.

This metadata enables hybrid queries such as "find the most relevant chunks
about security policies updated in the last 30 days."

**Critical constraint**: The same embedding model and parameters must be used
at both ingestion time and query time. If the ingestion pipeline uses
Vertex AI's text-embedding model to produce 768-dimensional vectors, the
query pipeline must use the identical model and parameters. Mismatched
embeddings live in different vector spaces and produce meaningless similarity
scores.

Why does the embedding model need to match?

Vector embeddings are only comparable if they come from the same model with the same parameters. A 768-dimensional vector from one model exists in a completely different mathematical space than a 768-dimensional vector from another model. Comparing them with cosine similarity would produce random results.

This is a common production mistake. Teams sometimes upgrade their embedding model for new documents without re-embedding existing ones, which silently degrades retrieval quality.

  • Save your document.
  • Read through the full Ingestion Pipeline section from top to bottom. You should have three subsections: Data Ingestion Flow, Vector Storage and Indexing, and Metadata Strategy.

Section feeling disjointed?

Each subsection should connect logically. The flow is: documents arrive (Data Ingestion Flow), their embeddings are stored and indexed (Vector Storage and Indexing), and each chunk carries metadata for filtering (Metadata Strategy). If yours reads as three disconnected topics, add a sentence at the start of each subsection that references what came before it.

If you are stuck on wording, help me connect my three ingestion pipeline subsections with smooth transitions

.

✔️ Awesome, I've got everything!

Great. Double check you have saved your document with all three subsections complete.

ⓧ I'd like to double check the full code

Here is the complete Ingestion Pipeline section as it should appear in your document:

## 1. Ingestion Pipeline

### Data Ingestion Flow

Raw documents (PDFs, web pages, databases) are uploaded to Google Cloud Storage.
When a new object is finalized in the bucket, it triggers a Pub/Sub message.
A Cloud Run function subscribes to this topic, picks up the message, and
begins processing the raw data.

The Cloud Run function performs two key operations:

1. **Parsing and Chunking**: The raw document is parsed into plain text, then
   chunked into semantically meaningful segments using recursive character
   splitting. Each chunk targets 512 tokens with a 50-token overlap between
   consecutive chunks. This overlap ensures that context is not lost at
   chunk boundaries.

2. **Embedding Generation**: Each chunk is sent to the Vertex AI Embeddings
   API, which returns a 768-dimensional vector embedding. These embeddings
   capture the semantic meaning of each chunk in a format optimized for
   similarity search.

### Vector Storage and Indexing

Embeddings are stored in Vertex AI Vector Search, a fully managed vector
similarity-matching service. It builds a ScaNN-based approximate nearest
neighbor (ANN) index over the stored embeddings, enabling sub-linear search
time even at large scale.

Key design choices:

- **Distance metric**: Cosine similarity. This measures the angle between
  vectors rather than magnitude, making it ideal for text embeddings where
  direction encodes meaning.

- **Streaming updates**: The index supports streaming upserts, allowing new
  document chunks to become searchable without requiring a full index rebuild.
  As new documents arrive through the ingestion pipeline, their embeddings
  are upserted into the live index.

- **Autoscaling**: Deployed indexes are configured with `minReplicaCount`
  and `maxReplicaCount` to handle variable query load automatically.

### Metadata Strategy

Each chunk retains structured metadata stored alongside its embedding to
enable filtered retrieval. Metadata fields include:

- **Source document**: The original file name or URL the chunk was extracted
  from.
- **Page number**: For paginated documents (PDFs), the specific page.
- **Timestamp**: When the document was ingested, enabling recency filtering.
- **Category**: A classification tag (e.g., "policy", "technical",
  "financial") for domain-specific filtering.

This metadata enables hybrid queries such as "find the most relevant chunks
about security policies updated in the last 30 days."

**Critical constraint**: The same embedding model and parameters must be used
at both ingestion time and query time. If the ingestion pipeline uses
Vertex AI's text-embedding model to produce 768-dimensional vectors, the
query pipeline must use the identical model and parameters. Mismatched
embeddings live in different vector spaces and produce meaningless similarity
scores.

Your ingestion pipeline section is complete. Next up, you'll design the retrieval strategy that actually uses these embeddings to find relevant context for user queries.

Design the Retrieval Strategy

Your ingestion pipeline gets documents into the vector database. Now you need to get them back out at query time.

In this step, you'll design the retrieval strategy that takes a user's natural language query and finds the most relevant chunks from your Vertex AI Vector Search index. This is where the "R" in RAG happens.

In this step, get ready to:
  • Write the query processing flow that converts user queries into vector embeddings.
  • Describe the semantic search mechanism using approximate nearest neighbor (ANN) search.
  • Document post-retrieval processing including re-ranking and metadata filtering.
Write the query processing flow

For retrieval to work, the user's query must live in the same mathematical space as your stored document chunks. This means converting the query into a vector embedding using the exact same model and parameters you used during ingestion.

  • In your design document, scroll down to the Retrieval Strategy section heading.
  • Write the following content describing the query embedding process:
## Retrieval Strategy

### Query Processing

When a user submits a natural language query, the system converts it into a
768-dimensional vector embedding using the Vertex AI Embeddings API. This is the
same model and parameters used during the ingestion phase.

Using identical embedding configuration at both ingestion and query time is
critical. It ensures the query vector lives in the same mathematical space as
the stored document vectors, making similarity comparisons meaningful.

Why must the embedding model match?

Different embedding models map text to different vector spaces. If you embedded your documents with Model A but embed queries with Model B, the vectors won't be comparable. A query about "machine learning" might land far from a relevant document chunk simply because the two models represent that concept differently.

This is why Google Cloud's reference architecture explicitly requires the same embeddings model and parameters at both ingestion and query time.

  • Save your document.
  • You should see a Query Processing subsection under your Retrieval Strategy heading explaining how queries are embedded using the same Vertex AI model as ingestion.

Not sure what to write?

Focus on what happens to the query before any search occurs. The key point is: raw text query goes in, 768-dimensional vector comes out, using the same model from Step 2.

Help me explain query embedding in my RAG design document

Describe the semantic search mechanism

Once the query is a vector, it needs to find the closest document vectors in your index. This is where Vertex AI Vector Search performs an approximate nearest neighbor (ANN) search to return the most relevant chunks.

  • Below your Query Processing subsection, add the following content describing the search mechanism:
### Semantic Search

The query embedding is sent to the Vertex AI Vector Search index, which performs
an approximate nearest neighbor (ANN) search. This returns the top-k most
relevant document chunks, typically between 5 and 10 results.

The `approximateNeighborsCount` parameter controls how many candidates the index
evaluates during search. Decreasing this value reduces both latency and cost,
though it may slightly reduce recall. For most production workloads, tuning this
parameter provides the best cost-performance tradeoff.

The search uses cosine similarity as the distance metric, matching the metric
used during index creation in the ingestion pipeline.

What does this section cover?

This section explains three things: what type of search runs (ANN, not exact nearest neighbor), how many results come back (top-k, typically 5-10), and how to tune performance using the approximateNeighborsCount parameter.

ANN search trades a small amount of accuracy for dramatically faster search times. At production scale with millions of vectors, exact search is impractical.

What is Semantic Search?

Traditional keyword search matches exact words. Semantic search matches meaning. A query like "how do I fix a broken pipe" finds documents about plumbing repairs even if those documents never use the exact word "fix." This works because the embedding model places semantically similar text close together in vector space.

  • Save your document.
  • You should now see a Semantic Search subsection that explains ANN search, top-k results, and the approximateNeighborsCount tuning parameter.

Unsure about top-k values?

5-10 is a common range for production RAG systems. Too few results and you miss relevant context. Too many and you flood the LLM's context window with noise, increasing cost and reducing response quality.

Help me understand top-k selection in vector search

Document post-retrieval processing

Raw search results are rarely perfect. The chunks that come back from vector search are ranked by vector similarity alone. Re-ranking and filtering improve relevance before anything reaches the LLM.

  • Below your Semantic Search subsection, add the following content describing post-retrieval processing:
### Post-Retrieval Processing

#### Re-ranking

After initial retrieval, chunks are re-ranked to improve relevance ordering.
The system uses either a cross-encoder model or the Vertex AI Search ranking
API. Unlike the initial vector similarity search (which scores query and
document independently), a cross-encoder evaluates the query-document pair
together, producing more accurate relevance scores.

#### Metadata Filtering

Retrieved chunks are further filtered by metadata before reaching the LLM:

- **Recency**: Prioritize recently updated documents when freshness matters.
- **Category**: Narrow results to the relevant domain (e.g., only return
  financial docs for finance queries).

This filtering uses the metadata stored alongside each embedding during
ingestion (source document, page number, timestamp, category). The combination
of re-ranking and filtering ensures only the most relevant, appropriate chunks
are passed to the LLM for response generation.

Why re-rank after search?

Vector similarity gives you a rough ranking. A cross-encoder gives you a precise one. The tradeoff is speed: cross-encoders are too slow to run against millions of documents, but fast enough to re-score 5-10 candidates.

This two-stage approach (fast ANN retrieval followed by precise re-ranking) is the standard pattern in production search systems.

  • Save your document.
  • Your Retrieval Strategy section should now have three subsections: Query Processing, Semantic Search, and Post-Retrieval Processing.

Not sure about the difference between re-ranking and filtering?

Re-ranking reorders results by relevance (all results stay, just in a better order). Filtering removes results entirely based on criteria like date or category. Both happen after the initial search, but they solve different problems.

Help me explain re-ranking vs filtering in my retrieval strategy

✔️ Awesome, I've got everything!

Great. Double check you've saved your document with all three subsections under Retrieval Strategy.

ⓧ I'd like to double check the full section

Here is the complete Retrieval Strategy section for comparison:

## Retrieval Strategy

### Query Processing

When a user submits a natural language query, the system converts it into a
768-dimensional vector embedding using the Vertex AI Embeddings API. This is the
same model and parameters used during the ingestion phase.

Using identical embedding configuration at both ingestion and query time is
critical. It ensures the query vector lives in the same mathematical space as
the stored document vectors, making similarity comparisons meaningful.

### Semantic Search

The query embedding is sent to the Vertex AI Vector Search index, which performs
an approximate nearest neighbor (ANN) search. This returns the top-k most
relevant document chunks, typically between 5 and 10 results.

The `approximateNeighborsCount` parameter controls how many candidates the index
evaluates during search. Decreasing this value reduces both latency and cost,
though it may slightly reduce recall. For most production workloads, tuning this
parameter provides the best cost-performance tradeoff.

The search uses cosine similarity as the distance metric, matching the metric
used during index creation in the ingestion pipeline.

### Post-Retrieval Processing

#### Re-ranking

After initial retrieval, chunks are re-ranked to improve relevance ordering.
The system uses either a cross-encoder model or the Vertex AI Search ranking
API. Unlike the initial vector similarity search (which scores query and
document independently), a cross-encoder evaluates the query-document pair
together, producing more accurate relevance scores.

#### Metadata Filtering

Retrieved chunks are further filtered by metadata before reaching the LLM:

- **Recency**: Prioritize recently updated documents when freshness matters.
- **Category**: Narrow results to the relevant domain (e.g., only return
  financial docs for finance queries).

This filtering uses the metadata stored alongside each embedding during
ingestion (source document, page number, timestamp, category). The combination
of re-ranking and filtering ensures only the most relevant, appropriate chunks
are passed to the LLM for response generation.

Your retrieval strategy is locked in. You now have a complete path from user query to relevant document chunks. Next up, you'll design the orchestration layer that ties retrieval to prompt construction and LLM invocation.

Design the Orchestration Layer

Your ingestion pipeline converts documents into embeddings, and your retrieval strategy finds the most relevant chunks. Now you need a coordinator that ties everything together.

The orchestration layer is the brain of your RAG system. It manages the complete flow from the moment a user submits a query to the moment they receive a grounded response.

In this step, get ready to:
  • Justify your orchestration framework choice.
  • Map the complete query-to-response chain.
  • Document the prompt engineering strategy within the orchestrator.
Justify your framework choice

The orchestration framework determines how your system composes retrieval, prompt construction, and LLM invocation into a coherent pipeline. Your choice here affects developer experience, streaming support, and long-term maintainability.

  • Switch back to your design document and find the empty Orchestration Layer section.
  • Write the following framework justification as the opening of Section 3:
## 3. Orchestration Layer

### Framework Choice: LangChain

This system uses LangChain as its orchestration framework. LangChain excels at
orchestrating multi-step flows, connecting retrieval to prompt construction to
LLM invocation. Its LCEL (LangChain Expression Language) provides composable,
streaming-native pipelines that allow each stage of the RAG flow to be defined
as an independent, reusable component.

For the ingestion/indexing layer specifically, LlamaIndex would be a strong
alternative if the primary concern were retrieval accuracy (~92% vs ~85% in
independent benchmarks). However, the orchestration challenge here is managing
the full query-to-response flow, which is LangChain's core strength.

The langchain-google-vertexai package provides native integration with Vertex AI
platform services, including Model Garden, Vector Search, and evaluation tools.

Why LangChain over LlamaIndex?

The challenge is to design the orchestration of a complete pipeline, not just the retrieval layer. LangChain's LCEL syntax lets you compose each stage (cache check, embedding, search, prompt, LLM call) as a declarative chain.

In production, many teams use both: LlamaIndex for ingestion and retrieval accuracy, LangChain for orchestrating the end-to-end flow. Naming both in your design shows awareness of the trade-offs.

Map the complete orchestration chain

A strong design document doesn't just name the framework. It maps out exactly what happens at each stage when a user submits a query.

  • Below your framework justification, write the full orchestration flow:
### Orchestration Flow

The complete chain from user query to final response:

1. **User Query** — The user submits a natural language question to the
   application layer (hosted on Cloud Run).

2. **Semantic Cache Check** — The query is embedded and compared against cached
   query vectors in Redis. If cosine similarity exceeds the threshold
   (~0.85-0.95), the cached response is returned immediately (~40ms median
   latency). Approximately 31% of queries are contextually repeatable and
   benefit from caching.

3. **On Cache Miss — Query Embedding** — The query is converted to a
   768-dimensional vector using the Vertex AI Embeddings API (same model and
   parameters as ingestion to ensure shared vector space).

4. **Vector Search** — The query embedding is sent to the Vertex AI Vector
   Search index, which performs ANN search to return the top-k most relevant
   document chunks.

5. **Context Retrieval** — Retrieved chunks are re-ranked and filtered by
   metadata (as described in Section 2).

6. **Prompt Augmentation** — The orchestrator constructs a grounded prompt
   combining the original query, retrieved context, and system instructions
   (detailed in the next subsection).

7. **LLM Generation** — The augmented prompt is sent to Gemini on Vertex AI
   for response generation.

8. **Safety Filters** — Vertex AI Responsible AI safety filters are applied
   before the response is returned to the user.

9. **Response + Cache Update** — The final response is returned to the user.
   The query-response pair is stored in the semantic cache for future use.

What makes this flow production-grade?

The cache check happens before any expensive computation. This means roughly 31% of queries skip the entire retrieval and LLM pipeline, returning in ~40ms instead of 400-800ms.

The cache update at the end creates a self-improving system. As more users interact with the system, the cache hit rate increases over time.

Document the prompt engineering strategy

The prompt is what separates a RAG system from a naive LLM call. Your orchestrator needs to construct a prompt that grounds the LLM's response in the retrieved evidence rather than its parametric knowledge.

  • Add the following prompt engineering subsection to complete Section 3:
### Prompt Engineering

The orchestrator constructs a grounded prompt with four components:

1. **System Instruction** — Enforces factual responses only from the provided
   context. The system instruction explicitly tells the model: "Answer the
   user's question using ONLY the information provided in the context below.
   If the context does not contain enough information to answer, say so."

2. **Retrieved Document Chunks** — The top-k re-ranked chunks from the
   retrieval stage are injected as context. Each chunk includes its source
   metadata (document title, page number) so the model can cite its sources.

3. **Original User Query** — The user's question is included verbatim after
   the context, preserving the exact intent without modification.

4. **Vertex AI Responsible AI Safety Filters** — Before the response is
   returned to the user, Vertex AI's built-in safety filters screen for
   harmful content, bias, and policy violations. This is applied at the
   platform level, not within the prompt itself.

This prompt structure ensures the LLM generates responses grounded in retrieved
evidence rather than hallucinating from parametric knowledge. The system
instruction acts as a guardrail, while the safety filters provide a second
layer of content governance.

Why does grounding matter?

Without a grounding instruction, the LLM might ignore your retrieved context and answer from its training data. This defeats the entire purpose of RAG.

The explicit "say so" fallback is equally important. It prevents the model from filling gaps with hallucinated information when the retrieved context doesn't cover the question.

  • Save your design document.
  • Review Section 3 from top to bottom. You should see three subsections: Framework Choice, Orchestration Flow (9 numbered steps), and Prompt Engineering (4 components).

Section 3 feels disconnected from Sections 1 and 2?

Check that your orchestration flow references the same embedding model and parameters from Section 1. The phrase "same model and parameters as ingestion" in step 3 of the flow creates a direct link to your ingestion pipeline design.

Also confirm that step 5 (Context Retrieval) references the re-ranking and metadata filtering you documented in Section 2. Cross-references between sections show the system works as a coherent whole.

Help me connect my orchestration layer design to the other sections of my document.

✔️ Awesome, I've got everything!

Great. Double check you've saved your document with all three subsections in Section 3.

ⓧ I'd like to double check the full section

Here is the complete Section 3 for reference:

## 3. Orchestration Layer

### Framework Choice: LangChain

This system uses LangChain as its orchestration framework. LangChain excels at
orchestrating multi-step flows, connecting retrieval to prompt construction to
LLM invocation. Its LCEL (LangChain Expression Language) provides composable,
streaming-native pipelines that allow each stage of the RAG flow to be defined
as an independent, reusable component.

For the ingestion/indexing layer specifically, LlamaIndex would be a strong
alternative if the primary concern were retrieval accuracy (~92% vs ~85% in
independent benchmarks). However, the orchestration challenge here is managing
the full query-to-response flow, which is LangChain's core strength.

The langchain-google-vertexai package provides native integration with Vertex AI
platform services, including Model Garden, Vector Search, and evaluation tools.

### Orchestration Flow

The complete chain from user query to final response:

1. **User Query** — The user submits a natural language question to the
   application layer (hosted on Cloud Run).

2. **Semantic Cache Check** — The query is embedded and compared against cached
   query vectors in Redis. If cosine similarity exceeds the threshold
   (~0.85-0.95), the cached response is returned immediately (~40ms median
   latency). Approximately 31% of queries are contextually repeatable and
   benefit from caching.

3. **On Cache Miss — Query Embedding** — The query is converted to a
   768-dimensional vector using the Vertex AI Embeddings API (same model and
   parameters as ingestion to ensure shared vector space).

4. **Vector Search** — The query embedding is sent to the Vertex AI Vector
   Search index, which performs ANN search to return the top-k most relevant
   document chunks.

5. **Context Retrieval** — Retrieved chunks are re-ranked and filtered by
   metadata (as described in Section 2).

6. **Prompt Augmentation** — The orchestrator constructs a grounded prompt
   combining the original query, retrieved context, and system instructions
   (detailed in the next subsection).

7. **LLM Generation** — The augmented prompt is sent to Gemini on Vertex AI
   for response generation.

8. **Safety Filters** — Vertex AI Responsible AI safety filters are applied
   before the response is returned to the user.

9. **Response + Cache Update** — The final response is returned to the user.
   The query-response pair is stored in the semantic cache for future use.

### Prompt Engineering

The orchestrator constructs a grounded prompt with four components:

1. **System Instruction** — Enforces factual responses only from the provided
   context. The system instruction explicitly tells the model: "Answer the
   user's question using ONLY the information provided in the context below.
   If the context does not contain enough information to answer, say so."

2. **Retrieved Document Chunks** — The top-k re-ranked chunks from the
   retrieval stage are injected as context. Each chunk includes its source
   metadata (document title, page number) so the model can cite its sources.

3. **Original User Query** — The user's question is included verbatim after
   the context, preserving the exact intent without modification.

4. **Vertex AI Responsible AI Safety Filters** — Before the response is
   returned to the user, Vertex AI's built-in safety filters screen for
   harmful content, bias, and policy violations. This is applied at the
   platform level, not within the prompt itself.

This prompt structure ensures the LLM generates responses grounded in retrieved
evidence rather than hallucinating from parametric knowledge. The system
instruction acts as a guardrail, while the safety filters provide a second
layer of content governance.

Your orchestration layer ties the entire system together. Next up, you'll design the optimization strategies that make this pipeline fast enough for production use.

Design the Optimization Layer

Your orchestration layer maps the complete path from query to response. But in production, a system that works correctly is only half the battle.

Users expect fast answers. The optimization layer is what separates a demo from a production system. In this step, you'll design caching, latency reduction, and scaling strategies that make your RAG system responsive under real-world load.

In this step, get ready to:
  • Design a Redis semantic caching strategy with concrete performance metrics.
  • Document three latency reduction techniques (streaming, model routing, pipeline parallelism).
  • Address scalability with autoscaling configurations for Vector Search and Cloud Run.
Describe the semantic caching strategy using Redis

The most effective optimization for a RAG system is avoiding the expensive pipeline entirely. Semantic caching stores previous query-response pairs and serves them again when a similar question arrives.

  • In your design document, scroll to the Optimization section (Section 4).
  • Write the following content describing how Redis semantic caching works in your system:
## 4. Optimization

### Semantic Caching (Redis)

The system uses Redis as a semantic cache layer to eliminate redundant LLM
calls. When a user submits a query, the system:

1. Embeds the query into a vector (same embedding model as ingestion)
2. Compares the query vector against cached query vectors using cosine
   similarity
3. If similarity exceeds the threshold (~0.85-0.95), returns the cached
   response instantly (~40ms median latency)
4. On a cache miss, executes the full RAG pipeline and stores the new
   query-response pair for future use

Approximately 30% of queries to an LLM are contextually repeatable,
making caching a high-impact optimization. The cache dramatically
reduces both latency and cost for these repeated queries.

**TTL Policies for Cache Freshness:**
- Fast-changing data sources: 1-hour TTL
- Stable knowledge bases: 24-hour TTL
- Event-driven invalidation when source documents are updated

Why Redis for semantic caching?

Redis offers sub-millisecond vector search and has production-proven semantic caching libraries (like RedisVL's SemanticCache class). The 40ms median latency for cache hits compares favorably to the 400-800ms a full RAG pipeline takes.

The similarity threshold (0.85-0.95) is a tuning knob. A lower threshold catches more queries but risks returning irrelevant cached responses. A higher threshold is safer but caches fewer queries.

  • Save your document after adding the semantic caching section.
  • Review that your section mentions all four elements: the embedding comparison, the cosine similarity threshold range, the 40ms median latency on hits, and TTL policies.

Not sure where to put the threshold value?

The threshold (~0.85-0.95) goes in the step where the system decides whether a cached response is close enough to return. It is not the same as retrieval similarity. Think of it as: how similar must a new question be to a previously answered question before we reuse that answer?

Help me understand the difference between caching similarity and retrieval similarity in a RAG system.

Document latency reduction techniques

Caching helps when a similar question has been asked before. For cache misses, you need additional techniques to reduce the time users spend waiting for the full pipeline to complete.

  • Below your semantic caching section, add the following content covering three latency reduction strategies:
### Latency Reduction Techniques

**1. Streaming Responses**

Rather than waiting for the LLM to generate a complete response, the
system streams tokens back to the user as they are produced. This
reduces perceived latency significantly. Users see output immediately
instead of staring at a loading spinner for several seconds.

**2. Model Routing**

Not all queries need the most powerful model. The system uses a routing
layer that classifies incoming queries:
- Simple factual queries -> Gemini Flash (faster, lower cost)
- Complex reasoning queries -> Gemini Pro (higher capability)

This maintains answer quality while reducing average response time
across all queries.

**3. Pipeline Parallelism**

Instead of executing retrieval and generation strictly in sequence, the
system overlaps these stages. As relevant chunks are retrieved, they
are fed to the LLM incrementally rather than waiting for all chunks to
arrive. Research (PipeRAG) demonstrates this approach can reduce
end-to-end latency by up to 2.6x without compromising generation
quality.

What does this content cover?

Each technique targets a different bottleneck. Streaming fixes perceived latency (the user sees progress). Model routing fixes unnecessary compute (not every question needs the biggest model). Pipeline parallelism fixes sequential bottlenecks (retrieval and generation can overlap).

The 2.6x latency reduction from PipeRAG is a research result. Citing specific numbers from published research strengthens your design document.

  • Save your document.
  • Confirm that your latency section names all three techniques (streaming, model routing, pipeline parallelism) and includes the 2.6x performance claim with its PipeRAG attribution.

Unsure which model names to use?

The outline uses Gemini Flash for the smaller/faster model and Gemini Pro for the larger/more capable model. These are the two tiers within Google's Gemini model family on Vertex AI. Flash prioritizes speed. Pro prioritizes reasoning depth.

Help me understand the difference between Gemini Flash and Gemini Pro for model routing.

Address scalability

A system that is fast for one user but falls over at 100 concurrent users is not production-grade. Your final optimization subsection covers how each component scales independently.

  • Below your latency reduction section, add the following scalability content to complete Section 4:
### Scalability

**Vertex AI Vector Search Autoscaling**

When deploying a Vector Search index, configure `minReplicaCount` and
`maxReplicaCount` to enable autoscaling. The service automatically
adds replicas under high query load and scales back down during quiet
periods. This keeps costs proportional to actual usage.

**Cloud Run Auto-scaling**

The orchestration layer runs on Cloud Run, which scales from zero to
thousands of instances based on incoming request volume. Each instance
handles its own share of queries independently, making the
orchestration layer horizontally scalable without manual intervention.

**Asynchronous Ingestion Processing**

New document ingestion runs asynchronously via Pub/Sub triggers. This
means the serving path (answering user queries) is never blocked by
data updates happening in the background. Streaming index updates
(upserts) keep the Vector Search index fresh without requiring full
index rebuilds.

Why mention autoscaling configuration details?

Naming the specific parameters (minReplicaCount and maxReplicaCount) shows you've read the actual Vertex AI Vector Search documentation. It signals that your design is implementable, not just conceptual.

The async ingestion point ties back to your ingestion pipeline design in Section 1. The Pub/Sub trigger you described there is what enables non-blocking updates here.

  • Save your document.
  • Read through your complete Section 4 from top to bottom. It should cover three areas: semantic caching (with Redis, thresholds, and TTL), latency reduction (streaming, model routing, pipeline parallelism), and scalability (Vector Search autoscaling, Cloud Run, async ingestion).

Section 4 feeling too long or too short?

A strong Section 4 is typically 400-600 words. If yours is much shorter, check that you've included specific numbers (40ms, 30% hit rate, 2.6x reduction). If it's much longer, look for repeated explanations you can trim.

Help me check if my Optimization section covers enough detail for a system design submission.

Your optimization layer is complete. You now have a system that is not just correct but fast and scalable. Next up, you'll bring everything together visually by creating the architecture diagram in Excalidraw.

Create the Architecture Diagram

Your design document now covers how data flows in, how retrieval works, how orchestration ties everything together, and how caching and scaling keep it fast.

A professional architecture diagram makes all of that visible in a single glance. In this step, you'll draw the complete system in Excalidraw, showing every component, decision point, and data path your design describes.

In this step, get ready to:
  • Draw the main component boxes and connect them with labeled arrows showing the query flow.
  • Add a decision node for the cache check and a separate ingestion pipeline flow.
  • Polish the diagram with color-coding, latency annotations, and export it as a PNG.
Create the main flow path

The main flow path represents the journey of a user query through the entire RAG system. Each box is a component your design document already describes.

  • Return to your Excalidraw canvas in the browser.
  • Select the Rectangle tool from the toolbar at the top (or press R on your keyboard).
  • Draw 7 boxes arranged left-to-right across the canvas, with enough space between each for arrows.
  • Double-click each box to add a label. Name them in this order:
  • User
  • Application Layer (Cloud Run)
  • Semantic Cache (Redis)
  • Embedding Service (Vertex AI Embeddings API)
  • Vector Database (Vertex AI Vector Search)
  • LLM (Gemini on Vertex AI)
  • Final Response
  • Select the Arrow tool from the toolbar (or press A on your keyboard).
  • Draw an arrow from User to Application Layer (Cloud Run).
  • Draw an arrow from Application Layer to Semantic Cache (Redis).
  • Double-click each arrow to add a label describing the data flowing through it (e.g., User Query, Check Cache).

What does the main flow represent?

This left-to-right path mirrors the serving subsystem you described in your design document. A user query enters the system, hits the cache, and if needed, passes through embedding, vector search, and the LLM before returning a response.

Labeling every arrow forces you to be explicit about what data moves between components. This clarity is what separates a professional architecture diagram from a vague box-and-line sketch.

You should now see 7 labeled boxes connected by labeled arrows forming a horizontal flow across your canvas.

Arrows not connecting to boxes?

Make sure you start and end each arrow on the edge of a box. When you hover over a box edge with the arrow tool, you should see a blue highlight indicating a connection point.

If your arrows are floating free, delete them and redraw starting directly from a box border.

Help me fix my Excalidraw arrows and connections.

Add the decision point and dual paths

Your system has a critical branching point: does the cache already have a good answer? The diagram needs a diamond-shaped decision node to show this fork clearly.

  • Select the Diamond tool from the toolbar (or press D on your keyboard).
  • Draw a diamond shape between the Semantic Cache (Redis) box and the rest of the pipeline.
  • Double-click the diamond and label it Cache Hit?.
  • Draw a Yes arrow from the diamond directly to Final Response. Label the arrow Yes (Cache Hit).
  • Draw a No arrow from the diamond continuing through Embedding Service, Vector Database, and LLM to Final Response. Label it No (Cache Miss).

Now add the ingestion pipeline as a separate flow below the query path.

  • Draw 3 new boxes below the main flow. Label them Cloud Storage, Pub/Sub, and Cloud Run (Processing).
  • Connect them with arrows: Cloud Storage to Pub/Sub to Cloud Run (Processing).
  • Draw a final arrow from Cloud Run (Processing) up to the Vector Database (Vertex AI Vector Search) box in the main flow.
  • Label these arrows with their data flow: New Document, Trigger, Parse/Chunk/Embed, and Store Embeddings.

Why separate the ingestion flow visually?

The ingestion pipeline runs asynchronously from the query path. Documents are processed in the background as they arrive. Placing this flow below the main query path makes the async nature immediately obvious to anyone reading your diagram.

Your canvas should now show two distinct paths: the horizontal query flow with a diamond decision node, and a lower ingestion flow feeding into the vector database.

Diagram looking cluttered?

Select all elements with Cmd+A (macOS) or Ctrl+A (Windows), then use the alignment tools in the top toolbar to distribute boxes evenly. Leave generous spacing between the query flow and ingestion flow.

Help me organize my architecture diagram layout in Excalidraw.

Polish the diagram with color-coding and annotations

Color-coding turns a readable diagram into a scannable one. A reviewer can instantly identify which components belong to ingestion, querying, or caching without reading every label.

  • Select all the ingestion pipeline boxes (Cloud Storage, Pub/Sub, Cloud Run (Processing)) by holding Shift and clicking each one.
  • In the color picker on the left panel, set the background to blue. This marks the ingestion flow.
  • Select all query/serving path boxes (User, Application Layer, Embedding Service, Vector Database, LLM, Final Response) and set their background to green.
  • Select the Semantic Cache (Redis) box and the Cache Hit? diamond. Set their background to orange.

Use light background shades

Pick pastel or light versions of each color so your text labels remain readable. Dark backgrounds with dark text are hard to read when exported.

Now add latency annotations so reviewers understand the performance characteristics at a glance.

  • Select the Text tool from the toolbar (or press T on your keyboard).
  • Click near the Yes (Cache Hit) arrow and type ~40ms.
  • Click near the end of the full pipeline path (after the LLM box) and type ~400-800ms.
  • Make both annotations smaller than your box labels so they don't dominate the diagram. Select the text and reduce the font size in the left panel.

Where do these latency numbers come from?

The ~40ms figure is the median latency for a Redis semantic cache hit, based on Redis benchmarks. The ~400-800ms range covers the full retrieval-augmented generation pipeline including embedding, vector search, and LLM generation.

Putting real numbers on your diagram shows reviewers you understand the performance impact of each design decision.

Time to export your finished diagram.

  • Click the hamburger menu (three horizontal lines) in the top-left corner of Excalidraw.
  • Select Export image.
  • Choose PNG format and click Export to save the file to your computer.
  • Switch back to your design document and add the exported PNG to Section 5 (System Diagram). In Google Docs, use Insert > Image > Upload from computer. In a Markdown file, add an image reference to the PNG file.

What makes this diagram submission-ready?

Your diagram now has three layers of information: component names (what exists), arrow labels (what flows), and latency annotations (how fast). Combined with color-coding, a reviewer can understand your entire system architecture in seconds.

Your design document is now complete with all 5 sections written and a professional architecture diagram embedded.

Export looking blurry or cut off?

Before exporting, zoom out until your entire diagram fits on screen. Excalidraw exports what is visible in the canvas. If boxes are cut off at the edges, the export will miss them.

For a sharper export, check the Scale option in the export dialog and set it to 2x.

Help me export a clean PNG from Excalidraw.

Secret mission

Add a Failure Modes & Resilience Section

A production system that only works on sunny days isn't production-ready. Real architecture documents address what happens when things break. In this mission, you'll add a 6th section to your design document covering graceful degradation, cache invalidation, and observability.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely in your browser and local text editor, so there are no ongoing costs or cloud resources to worry about.

Decide whether to keep your project files for future reference, or delete them if you no longer need them.

Resources you used:

  • Excalidraw canvas (browser local storage or saved .excalidraw file).
  • Design document (Google Docs, local Markdown file, or Notion page).
  • Exported architecture diagram PNG.

Keep everything running

No action needed. Choose this if you want to revisit your design document or refine your architecture diagram later.

  • Your Excalidraw canvas is saved in your browser's local storage. You can return to https://excalidraw.com and it will still be there (as long as you don't clear your browser data).
  • Your design document stays wherever you created it (Google Docs, your local file system, or Notion).
  • Your exported PNG is on your local machine wherever you saved it.

Pause - I'll come back to this later

Close your browser tabs to free up memory. Your work is already saved.

  • Close the Excalidraw browser tab. Your canvas is preserved in browser local storage and will reload when you return.
  • Close your text editor tab or window. Your document is already saved to disk or to the cloud (Google Docs and Notion auto-save).

Delete - I don't want to use this again

Remove all project files and clear your Excalidraw canvas from local storage.

  • Delete the exported PNG from your local machine. Find it in the folder where you saved it and move it to the Trash (macOS) or Recycle Bin (Windows).
  • Delete any local design document files (Markdown or other) from your file system. Move them to the Trash (macOS) or Recycle Bin (Windows).
  • If you used Google Docs, open the document in Google Docs, click File, then click Move to trash.
  • Clear your Excalidraw canvas from browser local storage. In your browser, open Settings, go to Privacy and security, select Clear browsing data, check Cookies and other site data, and clear data for excalidraw.com only.
  • If you saved a .excalidraw file locally, delete that file as well.

Nice Work!

Nice Work!

You just designed a production-grade RAG system from scratch. You now have a submission-ready architecture document that proves you can design the infrastructure layer that makes Gen AI reliable at scale.

Here's what you accomplished:

  • Designed a complete ingestion pipeline and semantic retrieval strategy that converts raw documents into searchable vector embeddings via Vertex AI Vector Search, with re-ranking and metadata filtering for precision.
  • Architected an orchestration layer using LangChain and an optimization strategy with Redis semantic caching and model routing for speed.
  • Created a professional architecture diagram in Excalidraw showing the complete data path from user query through cache, vector database, LLM, and back to final response.
  • Added a Failure Modes & Resilience section covering graceful degradation, cache invalidation strategies, circuit breaker patterns, and observability.

Ready to quiz yourself?