Document Intelligence RAG Copilot

Build a RAG copilot that answers questions grounded in your documents.

Introduction

30 Second Summary

You have a 50-page report and need one specific number buried on page 37. Right now, that means skimming through dozens of pages or hitting Ctrl+F and hoping you guess the right keyword.

In this project, you will build a Document Intelligence Copilot powered by Retrieval-Augmented Generation (RAG). Upload any PDF or image, and an AI sidebar answers your questions using the actual content of your document.

What You'll Build

Drag a PDF into your app, watch it get parsed into clean structured Markdown, then type a question in the AI sidebar and see it pull the exact answer straight from your document.

By the end of this project, you'll have:

  • Upload a PDF or image and watch Landing AI ADE parse it into structured Markdown with tables reconstructed, layout understood, and reading order preserved.
  • Ask the copilot a question and see Pinecone retrieve the most relevant document chunks, rerank them for precision, and deliver a grounded answer.
  • Chat with a CopilotKit-powered AI sidebar backed by Google Gemini that responds from your actual document content instead of hallucinating.
  • Secret Mission: Upgrade to hybrid search combining dense vector similarity with sparse keyword matching for more precise retrieval.

Are there any prerequisites?

You'll need Node.js 20+ and a code editor (like VS Code) installed on your machine. You'll also need a Google AI Studio account for a free Gemini API key. We'll create Landing AI ADE and Pinecone accounts together in Step 1.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. Taking a moment to picture the finished product makes every step that follows click into place.

Set Up the Project and API Keys

Every feature you build in this project depends on three external services: Google Gemini for the AI brain, Landing AI ADE for document parsing, and Pinecone for vector search.

Before any of that can work, you need a running Next.js app with all three services configured. In this step, you will scaffold the project, create free accounts, and wire up your API keys so the dev server starts clean.

In this step, get ready to:
  • Scaffold a Next.js app and install all project dependencies.
  • Create free accounts for Landing AI ADE and Pinecone, and grab your Gemini API key.
  • Configure your environment variables and verify the dev server starts.
Scaffold the Next.js app

The project lives inside a folder called doc-rag-copilot. You will create it on your Desktop so it is easy to find.

  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the Next.js app by running this command:
npx create-next-app@latest doc-rag-copilot

What will I see in the terminal?

The command will ask you: "Would you like to use the recommended Next.js defaults?" The recommended defaults include TypeScript, ESLint, Tailwind CSS, and App Router.

  • Select Yes, use recommended defaults and press Enter.

The scaffolder will create the project folder and install the base Next.js dependencies automatically. This may take a minute.

  • Confirm the folder was created by running:
ls

You should see doc-rag-copilot listed in the output.

  • Move into the project folder by running:
cd doc-rag-copilot
  • Install the four project-specific packages by running this command:
npm install @copilotkit/react-core @copilotkit/runtime landingai-ade @pinecone-database/pinecone

What do these packages do?

  • @copilotkit/react-core and @copilotkit/runtime provide the AI sidebar UI and the server-side runtime that connects to your LLM.
  • landingai-ade is the SDK for Landing AI's document parsing API, which turns PDFs and images into structured Markdown.
  • @pinecone-database/pinecone is the SDK for Pinecone's vector database, where you will store and search document chunks.

You should see a success message once the install finishes. The terminal prints the number of packages added.

Seeing permission errors or install failures?

  • Make sure you are inside the doc-rag-copilot folder. Run pwd to check your current directory.
  • If you see EACCES errors on macOS, try prefixing the command with sudo.

Help me fix my npm install error.

Get your three free API keys

Each service needs an API key so your app can authenticate. All three offer free tiers with no credit card required.

Google Gemini API key

You already have a Google AI Studio account from the prerequisites. Now you need to grab an API key from it.

  • Open aistudio.google.com in your browser and sign in with your Google account.
  • Click Get API key in the left sidebar.
  • Click Create API key.
  • Copy the generated key and paste it somewhere safe (you will need it shortly).

Why is Gemini free?

Google offers free input and output tokens on the Gemini API Free Tier with no billing setup required. This project stays well within those limits.

Landing AI ADE API key
  • Open ade.landing.ai in your browser.
  • Sign up for a free account. You will start on the Explore plan, which includes 1,000 free credits and does not require a credit card.
  • After signing in, click your profile icon at the bottom-left corner of the page.
  • Click API Key.
  • Click the Copy icon to copy your API key. Paste it somewhere safe.
Pinecone API key
  • Open app.pinecone.io in your browser.
  • Sign up for a free account and choose the Starter plan (free, no credit card required).
  • After signing up, you will receive an API key in the console. Copy and save it somewhere safe.

What if I already have a Pinecone account?

If you already have an account, navigate to API Keys in your project dashboard. Click Create API key, give it a name, and copy the generated key. You will not be able to see the key again after closing the dialog.

Create the environment file and start the dev server

Your app reads API keys from a file called .env.local at the project root. This file is automatically ignored by Git, so your keys stay out of version control.

  • Open VS Code. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the doc-rag-copilot folder on your Desktop, then click Open.
  • Create a new file in the project root by clicking the New File icon in the Explorer sidebar.
  • Name the file .env.local and press Enter.
  • Add your three API keys by pasting the following into .env.local:
GOOGLE_GENERATIVE_AI_API_KEY=your_gemini_api_key_here
VISION_AGENT_API_KEY=your_landing_ai_ade_key_here
PINECONE_API_KEY=your_pinecone_api_key_here
  • Replace each placeholder with the real API key you copied earlier.

What do these variable names mean?

  • GOOGLE_GENERATIVE_AI_API_KEY is the variable name the Vercel AI SDK reads automatically to authenticate with Google Gemini.
  • VISION_AGENT_API_KEY is the default variable the landingai-ade SDK reads automatically.
  • PINECONE_API_KEY is the standard variable the Pinecone SDK reads automatically.
  • Save .env.local by pressing Cmd+S (macOS) or Ctrl+S (Windows).

Before you start the dev server, what do you think you will see at localhost:3000?

  • Open the VS Code integrated terminal by pressing Ctrl+` (backtick key, same on macOS and Windows).
  • Start the dev server by running:
npm run dev

The terminal prints a local URL. Open http://localhost:3000 in your browser. You should see the default Next.js starter page with the Next.js logo and links to documentation.

Don't see the starter page?

  • Make sure the terminal shows no errors. If you see a port conflict, stop any other process using port 3000 and try again.
  • Check that you are inside the doc-rag-copilot folder. Run pwd in the terminal to confirm.
  • Verify that .env.local does not have any syntax errors (no quotes around the values, no spaces around the = sign).

Help me troubleshoot my Next.js dev server not starting.

✔️ Awesome, I've got everything!

Great. Double check you have saved .env.local with all three keys filled in, and that the dev server is running at localhost:3000.

ⓧ I'd like to double check the full code

Your .env.local file should look like this (with your real keys replacing the placeholders):

GOOGLE_GENERATIVE_AI_API_KEY=your_gemini_api_key_here
VISION_AGENT_API_KEY=your_landing_ai_ade_key_here
PINECONE_API_KEY=your_pinecone_api_key_here

Your environment is ready and the dev server is running. Next up, you will wire up the CopilotKit sidebar so you have a working AI chat interface to talk to.

Wire Up the CopilotKit Sidebar with Gemini

Your project is scaffolded and your API keys are configured. Now it's time to bring your app to life with an AI chat sidebar powered by Google Gemini.

CopilotKit makes it easy to embed an AI assistant directly into a React app. You'll connect it to Gemini so the sidebar can hold conversations. There's a catch, though. Without any document knowledge, the copilot will confidently make up answers when you ask about your files.

In this step, get ready to:
  • Create the CopilotKit runtime API route with Gemini as the language model.
  • Add the CopilotKit provider and sidebar to your app's UI.
  • Test the sidebar and discover why RAG is needed.
Create the CopilotKit runtime API route

CopilotKit needs a backend API route to process chat messages. This route creates a CopilotRuntime that uses Gemini as its language model and exposes it at /api/copilotkit.

Next.js uses a special folder name [[...slug]] to create a catch-all route. This lets CopilotKit handle multiple endpoints under a single route file.

  • Create the API route directory by running this command in your terminal:
mkdir -p "app/api/copilotkit/[[...slug]]"
  • Confirm the directory was created by running:
ls app/api/copilotkit/

You should see [[...slug]] listed in the output.

  • In VS Code, expand the app/api/copilotkit/[[...slug]] folder in the Explorer sidebar.
  • Right-click the [[...slug]] folder and select New File.
  • Name the file route.ts and paste the following code into it:
import { CopilotRuntime, BuiltInAgent, InMemoryAgentRunner } from '@copilotkit/runtime';
import { google } from '@ai-sdk/google';
import { NextRequest } from 'next/server';

// Create a CopilotKit runtime with a built-in agent powered by Gemini
const runtime = new CopilotRuntime({
  agents: [
    new BuiltInAgent({
      runner: new InMemoryAgentRunner({
        model: google('gemini-1.5-flash'),
      }),
    }),
  ],
});

// Handle incoming chat requests from the CopilotKit frontend
export async function POST(req: NextRequest) {
  return runtime.handle(req);
}

export async function GET(req: NextRequest) {
  return runtime.handle(req);
}

What does this code do?

  • CopilotRuntime is the backend engine that processes chat messages from the sidebar.
  • BuiltInAgent wraps Gemini as the AI model. InMemoryAgentRunner manages conversation state in memory so the copilot can follow a multi-turn chat.
  • google('gemini-1.5-flash') connects to the Gemini API using the GOOGLE_GENERATIVE_AI_API_KEY from your .env.local file.
  • The POST and GET handlers let CopilotKit's frontend communicate with this runtime.
  • Save route.ts.
  • Confirm the file exists by running:
ls app/api/copilotkit/\[\[...slug\]\]/

You should see route.ts in the output.

Don't see route.ts?

  • Make sure you created the file inside the [[...slug]] folder, not in app/api/copilotkit/ directly.
  • Check that the folder name is exactly [[...slug]] with double square brackets. A single pair of brackets creates a different type of route.
  • My route.ts file isn't showing up in the right location. How do I fix my folder structure?

✔️ Awesome, I've got everything!

Great. Make sure you've saved route.ts before moving on.

ⓧ I'd like to double check the full code

Here is the complete route.ts file. Make sure yours matches exactly.

import { CopilotRuntime, BuiltInAgent, InMemoryAgentRunner } from '@copilotkit/runtime';
import { google } from '@ai-sdk/google';
import { NextRequest } from 'next/server';

const runtime = new CopilotRuntime({
  agents: [
    new BuiltInAgent({
      runner: new InMemoryAgentRunner({
        model: google('gemini-1.5-flash'),
      }),
    }),
  ],
});

export async function POST(req: NextRequest) {
  return runtime.handle(req);
}

export async function GET(req: NextRequest) {
  return runtime.handle(req);
}
Add the CopilotKit provider and sidebar

The backend route is ready. Now you need to connect the frontend to it. CopilotKit uses a provider component that wraps your entire app, plus a sidebar component that renders the chat UI.

  • Open app/layout.tsx in VS Code.
  • Replace the entire contents of the file with the following code:
import { CopilotKit } from '@copilotkit/react-core';
import '@copilotkit/react-ui/styles.css';
import './globals.css';

// Wrap the entire app in CopilotKit so the sidebar can access the runtime
export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <CopilotKit runtimeUrl="/api/copilotkit">
          {children}
        </CopilotKit>
      </body>
    </html>
  );
}

What does this code do?

  • The CopilotKit provider wraps your app and connects every page to the backend route at /api/copilotkit.
  • The @copilotkit/react-ui/styles.css import loads the default styling for the sidebar so it looks polished out of the box.
  • Save layout.tsx.

Now add the sidebar component and a placeholder upload area to the main page.

  • Open app/page.tsx in VS Code.
  • Replace the entire contents of the file with the following code:
'use client';
import { CopilotSidebar } from '@copilotkit/react-ui';

// Main page with the CopilotKit sidebar and a placeholder upload area
export default function Home() {
  return (
    <main>
      <CopilotSidebar defaultOpen={true} />
      <div className="upload-area">
        <p>Upload a PDF or image to get started</p>
      </div>
    </main>
  );
}

What does this code do?

  • 'use client' tells Next.js this is a client component. CopilotKit's UI components need to run in the browser, not on the server.
  • CopilotSidebar renders the chat panel on the right side of the page. The defaultOpen={true} prop makes it open automatically so you can see it right away.
  • The upload area is a placeholder for now. You'll add real file upload functionality in the next step.
  • Save page.tsx.
  • Return to your browser at localhost:3000. The page should auto-refresh.

You should see a CopilotKit sidebar panel open on the right side of the page with a text input at the bottom. The main area shows "Upload a PDF or image to get started".

Don't see the sidebar?

  • Check your terminal for errors. If you see a module not found error for @copilotkit/react-ui, run npm install @copilotkit/react-ui and restart the dev server.
  • Make sure layout.tsx imports @copilotkit/react-ui/styles.css. Without the stylesheet, the sidebar renders but is invisible.
  • Confirm that page.tsx starts with 'use client' on the very first line. Without it, Next.js treats the component as a server component and the sidebar fails to render.
  • My CopilotKit sidebar is not appearing on the page. Help me debug my layout.tsx, page.tsx, and route.ts setup.

✔️ Awesome, I've got everything!

Great. Double check that both layout.tsx and page.tsx are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete layout.tsx:

import { CopilotKit } from '@copilotkit/react-core';
import '@copilotkit/react-ui/styles.css';
import './globals.css';

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <CopilotKit runtimeUrl="/api/copilotkit">
          {children}
        </CopilotKit>
      </body>
    </html>
  );
}

And here is the complete page.tsx:

'use client';
import { CopilotSidebar } from '@copilotkit/react-ui';

export default function Home() {
  return (
    <main>
      <CopilotSidebar defaultOpen={true} />
      <div className="upload-area">
        <p>Upload a PDF or image to get started</p>
      </div>
    </main>
  );
}
Test the sidebar and expose the hallucination problem

The sidebar is live and connected to Gemini. Before you move on, try asking it a question about a document you haven't uploaded yet.

Before you type anything, think about this: the copilot has no access to any of your documents. What do you think it will say when you ask about one?

  • In the CopilotKit sidebar, type the following message and press Enter:
What does my document say about revenue?

The copilot responds with a confident-sounding answer about revenue. The problem is that it made the entire answer up. It has no access to any documents, so it fills in the gap with plausible-sounding text.

Why does the copilot hallucinate?

Large language models generate responses based on patterns from training data. When you ask about "your document", Gemini has no actual document to reference. It guesses what a document might say about revenue and presents that guess as fact.

This is exactly the problem that Retrieval-Augmented Generation (RAG) solves. Instead of guessing, RAG retrieves real content from your documents and injects it into the prompt so the model answers from facts, not imagination.

Your AI sidebar is live, but it proves a critical gap. The copilot has no document knowledge. In the next step, you'll fix that by uploading documents and parsing them with Landing AI ADE into structured content the copilot can actually use.

Upload and Parse Documents with Landing AI ADE

Your copilot can chat, but it knows nothing about your documents. When you asked it about revenue in the last step, it made up an answer.

That hallucination is the core problem RAG solves. In this step, you'll add the document intelligence layer: upload a PDF or image, and Landing AI ADE parses it into structured Markdown with semantic chunks. Tables get reconstructed, layouts are understood, and reading order is preserved.

In this step, get ready to:
  • Create a parse API route that sends documents to Landing AI ADE.
  • Build a file upload component that accepts PDFs and images.
  • Display the parsed Markdown and semantic chunks on screen.
Create the parse API route

The frontend needs a backend endpoint to send uploaded files to. This API route receives a file, passes it to Landing AI ADE for parsing, and returns the structured Markdown and chunks.

  • Create a new file at app/api/parse/route.ts by right-clicking the app/api folder in VS Code's Explorer sidebar, selecting New Folder, and naming it parse.
  • Inside the parse folder, create a new file called route.ts.
  • Paste the following code into app/api/parse/route.ts:
import { NextRequest, NextResponse } from 'next/server';
import { AdeClient } from 'landingai-ade';

// Initialize the ADE client with your API key
const client = new AdeClient({ apiKey: process.env.VISION_AGENT_API_KEY! });

export async function POST(req: NextRequest) {
  // Extract the uploaded file from the form data
  const formData = await req.formData();
  const file = formData.get('file') as File;
  // Convert the file to a buffer for the ADE SDK
  const buffer = Buffer.from(await file.arrayBuffer());
  // Send the document to Landing AI ADE for parsing
  const result = await client.v2.parse(buffer, { filename: file.name });
  // Return the structured Markdown and semantic chunks
  return NextResponse.json({
    markdown: result.markdown,
    chunks: result.chunks,
  });
}

What does this code do?

This route handles file uploads sent from the frontend as FormData. It extracts the file, converts it to a buffer, and passes it to Landing AI ADE's client.v2.parse() method.

ADE's DPT-3 model does far more than basic OCR. It understands document layout, reconstructs tables, preserves reading order, and returns semantic chunks with types like paragraph, table, heading, and figure.

✔️ Awesome, I've got everything!

Great. Save app/api/parse/route.ts before moving on.

ⓧ I'd like to double check the full code

import { NextRequest, NextResponse } from 'next/server';
import { AdeClient } from 'landingai-ade';

// Initialize the ADE client with your API key
const client = new AdeClient({ apiKey: process.env.VISION_AGENT_API_KEY! });

export async function POST(req: NextRequest) {
  // Extract the uploaded file from the form data
  const formData = await req.formData();
  const file = formData.get('file') as File;
  // Convert the file to a buffer for the ADE SDK
  const buffer = Buffer.from(await file.arrayBuffer());
  // Send the document to Landing AI ADE for parsing
  const result = await client.v2.parse(buffer, { filename: file.name });
  // Return the structured Markdown and semantic chunks
  return NextResponse.json({
    markdown: result.markdown,
    chunks: result.chunks,
  });
}
Add the file upload handler

The placeholder upload area from the last step needs a real file input and a handler function that sends the selected file to your new parse route.

  • Replace the entire contents of app/page.tsx with the following code:
'use client';
import { CopilotSidebar } from '@copilotkit/react-ui';
import { useState } from 'react';

export default function Home() {
  const [parsedMarkdown, setParsedMarkdown] = useState<string>('');
  const [chunks, setChunks] = useState<any[]>([]);
  const [uploading, setUploading] = useState(false);
  // Send the selected file to the parse API route
  async function handleFileUpload(e: React.ChangeEvent<HTMLInputElement>) {
    const file = e.target.files?.[0];
    if (!file) return;
    setUploading(true);
    const formData = new FormData();
    formData.append('file', file);
    const res = await fetch('/api/parse', { method: 'POST', body: formData });
    const data = await res.json();
    setParsedMarkdown(data.markdown);
    setChunks(data.chunks);
    setUploading(false);
  }

  return (
    <main>
      <CopilotSidebar defaultOpen={true} />
      <div className="upload-area">
        <input type="file" accept=".pdf,image/*" onChange={handleFileUpload} />
        {uploading && <p>Parsing document...</p>}
      </div>
    </main>
  );
}

What does this code do?

  • Three state variables track the parsed Markdown, the chunks array, and whether an upload is in progress.
  • handleFileUpload wraps the selected file in a FormData object and sends it to /api/parse via a POST request.
  • The file input uses accept=".pdf,image/*" to restrict uploads to PDFs and images, which are the formats Landing AI ADE supports.
  • Save app/page.tsx and switch to your browser at localhost:3000.
  • You should see a file input (a "Choose File" button) where the placeholder text used to be.

Don't see the file input?

  • Make sure you saved app/page.tsx. Unsaved changes won't appear in the browser.
  • Check that your dev server is still running in the terminal. If it stopped, run npm run dev again.
  • Confirm you replaced the entire file, including the new useState import on line 3.

Why isn't my file input showing up in the browser?

Display the parsed output and test

The upload handler stores the parsed data in state, but nothing renders it yet. You need to add JSX that shows the Markdown and each chunk's type.

  • In app/page.tsx, find the line that says:
{uploading && <p>Parsing document...</p>}
  • Directly below that line (still inside the <div className="upload-area">), paste the following display code:
        {parsedMarkdown && (
          <div>
            <h2>Parsed Markdown</h2>
            <pre>{parsedMarkdown}</pre>
            <h2>Chunks ({chunks.length})</h2>
            {/* Show each chunk's type and a preview of its content */}
            {chunks.map((chunk, i) => (
              <div key={i}>
                <strong>{chunk.type}</strong>: {chunk.text?.slice(0, 100)}...
              </div>
            ))}
          </div>
        )}

What does this display section do?

  • The parsedMarkdown && guard ensures this section only renders after a document has been parsed.
  • The <pre> tag renders the full Markdown output with whitespace preserved.
  • Each chunk is displayed with its semantic type (paragraph, table, heading, or figure) and a preview of its content.

Time to test the full pipeline. Find any PDF on your computer to upload. A document with a table or mixed layout works best for seeing ADE's capabilities.

Before you upload, what do you expect to see? Think about what the parse route returns and what the display code renders.

  • Save app/page.tsx and refresh your browser.
  • Click the file input and select a PDF or image from your computer.
  • Wait for the "Parsing document..." message to disappear.

You should see two sections appear below the file input: the full parsed Markdown output and a list of chunks, each labeled with its semantic type (paragraph, table, heading, or figure).

What is semantic chunking?

Most document parsers split text by character count or fixed-size windows. Landing AI ADE chunks by meaning instead. Each chunk represents a complete structural element: a full paragraph, an entire table, or a section heading.

This matters for RAG because your vector search will retrieve complete, meaningful units of content rather than fragments that start mid-sentence or split a table in half.

Seeing an error after uploading?

  • Check your terminal for errors. If you see a 401 or authentication error, verify that VISION_AGENT_API_KEY in your .env.local file matches the key from your Landing AI ADE account.
  • If the page shows nothing after the loading message, open your browser's DevTools console (Cmd+Option+J on macOS, Ctrl+Shift+J on Windows) and check for network errors on the /api/parse request.
  • Make sure you restarted the dev server after adding the .env.local file in Step 1. Environment variables only load at server start.

Help me debug my Landing AI ADE parse error

✔️ Awesome, I've got everything!

Great. Double check you've saved both app/api/parse/route.ts and app/page.tsx.

ⓧ I'd like to double check the full code

Your app/page.tsx should contain the two code blocks you pasted above. Scroll up and confirm:

  • The imports at the top include useState from react.
  • Three useState calls for parsedMarkdown, chunks, and uploading.
  • The handleFileUpload function that sends the file to /api/parse.
  • The return JSX includes the <input type="file">, the uploading indicator, and the parsed Markdown and chunks display.

Your app now has a document intelligence layer. Upload any PDF or image and Landing AI ADE transforms it into structured, searchable content. Next up, you'll store those chunks as vectors in Pinecone so the copilot can search through them.

Embed and Store Chunks in Pinecone

In the last step, you gave your app the ability to parse documents into structured Markdown and semantic chunks using Landing AI ADE. The copilot sidebar can chat, and your parse route delivers beautifully structured chunks. But there is a gap: those chunks sit in memory for one page load and then vanish.

This step builds the vector storage layer. Each parsed chunk gets embedded and stored in Pinecone so your copilot can search them later. You will also handle document updates, so re-uploading a changed document replaces the old vectors automatically.

In this step, get ready to:
  • Create a Pinecone index with integrated embeddings that convert text to vectors automatically.
  • Build an ingest API route that stores document chunks with metadata in Pinecone.
  • Wire the upload flow so parsing and ingesting happen in one click, with a success status on the UI.
Create a Pinecone index with integrated embeddings

Before your app can store vectors, you need a place to put them. Pinecone indexes with integrated embeddings let you send raw text and have Pinecone generate the vector embeddings for you.

  • Open the Pinecone console at app.pinecone.io and sign in.
  • Click Create Index.
  • Enter doc-rag-index as the index name.
  • Select the option to create an index with an integrated embedding model.
  • Choose multilingual-e5-large as the embedding model.
  • Set the dimensions to 1024.
  • Set the distance metric to cosine.
  • Set the text field name to data. This tells Pinecone which field in your records contains the text to embed.
  • Click Create Index.

Wait for the index status to show as ready. This usually takes about 30 seconds.

What are integrated embeddings?

Normally, building a vector search pipeline requires a separate embedding service to convert text into vectors before storing them. Pinecone's integrated embedding eliminates that extra step.

When you create an index with an integrated model like multilingual-e5-large, you send raw text and Pinecone generates the 1024-dimension vectors for you. The free Starter plan includes 5 million embedding tokens per month.

Don't see the integrated embedding option?

Make sure you are on the Starter (free) plan and creating a serverless index. Integrated embedding is available on serverless indexes only.

Help me create a Pinecone index with integrated embeddings.

Build the ingest API route

Your index is ready. Now you need an API route that receives parsed chunks from the frontend and stores them as vectors in Pinecone.

  • Create a new file at app/api/ingest/route.ts inside your doc-rag-copilot project and paste in the following code:
import { NextRequest, NextResponse } from 'next/server';
import { Pinecone } from '@pinecone-database/pinecone';

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pinecone.index('doc-rag-index');

export async function POST(req: NextRequest) {
  const { documentId, chunks } = await req.json();
  // Delete existing vectors for this document
  await index.deleteMany({ filter: { documentId: { $eq: documentId } } });
  // Upsert new vectors with metadata
  const vectors = chunks.map((chunk: any, i: number) => ({
    id: `${documentId}-${i}`,
    data: chunk.text,
    metadata: {
      documentId,
      chunkType: chunk.type,
      pageNumber: chunk.page ?? 0,
      text: chunk.text,
    },
  }));
  await index.upsertRecords(vectors);
  return NextResponse.json({ success: true, count: vectors.length });
}

What does this code do?

  • deleteMany removes any existing vectors for this documentId. This is how re-uploading a changed document replaces the old data instead of duplicating it.
  • Each chunk is mapped to a record with a unique id, the raw text in the data field (which Pinecone embeds automatically), and metadata for filtering later.
  • upsertRecords sends the records to Pinecone. Because this is an integrated embedding index, Pinecone generates the vector embeddings from the data field and stores everything in one call.
  • Save app/api/ingest/route.ts.
  • Confirm the file exists by running:
ls app/api/ingest/

You should see route.ts listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your file before moving on.

ⓧ I'd like to double check the full code

import { NextRequest, NextResponse } from 'next/server';
import { Pinecone } from '@pinecone-database/pinecone';

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pinecone.index('doc-rag-index');

export async function POST(req: NextRequest) {
  const { documentId, chunks } = await req.json();
  // Delete existing vectors for this document
  await index.deleteMany({ filter: { documentId: { $eq: documentId } } });
  // Upsert new vectors with metadata
  const vectors = chunks.map((chunk: any, i: number) => ({
    id: `${documentId}-${i}`,
    data: chunk.text,
    metadata: {
      documentId,
      chunkType: chunk.type,
      pageNumber: chunk.page ?? 0,
      text: chunk.text,
    },
  }));
  await index.upsertRecords(vectors);
  return NextResponse.json({ success: true, count: vectors.length });
}
Wire the upload flow end-to-end

The ingest route is ready, but nothing calls it yet. You need to update the frontend so that after parsing a document, the app automatically sends the chunks to Pinecone and shows a success status.

  • Install the uuid package to generate unique document IDs by running:
npm install uuid

Install failed?

Make sure you are running this command from inside the doc-rag-copilot folder. If your dev server is occupying the terminal, open a new terminal tab first.

Help me install uuid in my Next.js project.

Now update app/page.tsx with three changes: a new import, two new state variables, and an updated upload handler.

  • In app/page.tsx, find this line at the top of the file:
import { useState } from 'react';
  • Add this new import directly below it:
import { v4 as uuidv4 } from 'uuid';
  • Next, find this state variable inside the Home component:
const [uploading, setUploading] = useState(false);
  • Add these two new state variables directly below it:
  // Track the ingest status message and a stable document ID
  const [ingestStatus, setIngestStatus] = useState<string>('');
  const [documentId] = useState<string>(uuidv4());

Why generate a document ID on the client?

The documentId is a UUID that stays stable for the lifetime of the page. When you re-upload a document, the ingest route uses this ID to find and delete the old vectors before inserting new ones. This prevents stale data from accumulating in your index.

  • Now replace your entire handleFileUpload function with this updated version that calls the ingest route after parsing:
  async function handleFileUpload(e: React.ChangeEvent<HTMLInputElement>) {
    const file = e.target.files?.[0];
    if (!file) return;
    setUploading(true);
    setIngestStatus('');
    const formData = new FormData();
    formData.append('file', file);
    // Step 1: Parse the document
    const parseRes = await fetch('/api/parse', { method: 'POST', body: formData });
    const parseData = await parseRes.json();
    setParsedMarkdown(parseData.markdown);
    setChunks(parseData.chunks);
    // Step 2: Send parsed chunks to Pinecone
    const ingestRes = await fetch('/api/ingest', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ documentId, chunks: parseData.chunks }),
    });
    const ingestData = await ingestRes.json();
    setIngestStatus(ingestData.success ? `Stored ${ingestData.count} chunks in Pinecone.` : 'Ingest failed.');
    setUploading(false);
  }

What changed in the upload handler?

  • After parsing completes, the function now makes a second fetch call to /api/ingest, sending the parsed chunks and the document ID as JSON.
  • setIngestStatus updates the UI with a success message showing how many chunks were stored, or a failure message if something went wrong.

Finally, update the JSX to show the ingest status.

  • In the return block of your component, find this line:
        {uploading && <p>Parsing document...</p>}
  • Replace it with these two lines to update the loading text and display the ingest status:
        {uploading && <p>Parsing and ingesting document...</p>}
        {ingestStatus && <p>{ingestStatus}</p>}
  • Save app/page.tsx.

Seeing a TypeScript error about uuid?

If your editor shows a type error on the uuid import, you may need to install the type definitions separately by running:

npm install --save-dev @types/uuid

Help me fix a TypeScript error with the uuid import in my Next.js project.

✔️ Awesome, I've got everything!

Great. Double check that you have saved both files before testing.

ⓧ I'd like to double check the full code

Since app/page.tsx is a longer file, scroll through it and confirm these sections are in place, in order:

  • The 'use client' directive at the very top.
  • Three imports: CopilotSidebar, useState, and uuidv4.
  • Five state variables: parsedMarkdown, chunks, uploading, ingestStatus, and documentId.
  • The handleFileUpload function with two fetch calls: one to /api/parse and one to /api/ingest.
  • The JSX with the loading text Parsing and ingesting document... and the ingestStatus display.

Before you test, think about what you expect to happen. When you upload a document now, do you think the chunks will appear in your Pinecone index right away?

  • Return to your browser at localhost:3000.
  • Upload a sample PDF or image using the file input.
  • Watch for the loading text to change to a success message like Stored 5 chunks in Pinecone. (the number depends on your document).

Now confirm the vectors are actually in Pinecone.

  • Switch back to the Pinecone console at app.pinecone.io.
  • Click on your doc-rag-index index.
  • You should see vectors listed with metadata fields including documentId, chunkType, and pageNumber.

Not seeing vectors in the console?

  • Check your terminal for errors. A common issue is a missing or incorrect PINECONE_API_KEY in your .env.local file.
  • Make sure the index name in your code (doc-rag-index) matches the name you gave the index in the Pinecone console exactly.
  • If you see a status message on the page but no vectors in Pinecone, restart your dev server. The .env.local values are read at startup.

Help me debug why my chunks are not appearing in Pinecone.

Your document chunks are now embedded and stored as searchable vectors in Pinecone. Next up, you will connect this vector store to the copilot so it can retrieve relevant chunks and answer questions about your documents.

Connect RAG Retrieval with Reranking to the Copilot

Your document chunks are embedded and stored in Pinecone, complete with metadata for every page and chunk type. The copilot can chat, but it still has no way to access that knowledge.

In this step, you will close the loop. You will build a retrieval function that searches Pinecone for the most relevant chunks, reranks them for precision, and feeds them into the copilot as grounded context.

In this step, get ready to:
  • Build a server-side retrieval function that searches and reranks document chunks.
  • Register a copilot action that calls retrieval on every user question.
  • Test the full RAG pipeline end to end.
Build the server-side retrieval function

The copilot needs a way to search your stored chunks when a user asks a question. You will add a retrieval function directly to the CopilotKit runtime route so it can query Pinecone and rerank the results.

  • In app/api/copilotkit/[[...slug]]/route.ts, find the existing import lines at the top of the file:
import { CopilotRuntime, BuiltInAgent, InMemoryAgentRunner } from '@copilotkit/runtime';
import { google } from '@ai-sdk/google';
import { NextRequest } from 'next/server';
  • Replace those import lines and add the retrieval function below them by copying the following code to the top of the file, above the const runtime line:
import { CopilotRuntime, BuiltInAgent, InMemoryAgentRunner } from '@copilotkit/runtime';
import { google } from '@ai-sdk/google';
import { NextRequest, NextResponse } from 'next/server';
import { Pinecone } from '@pinecone-database/pinecone';

// Connect to the Pinecone index where document chunks are stored
const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pinecone.index('doc-rag-index');

// Search for relevant chunks and rerank them for precision
export async function retrieveChunks(query: string): Promise<string[]> {
  const results = await index.searchRecords({
    query: { data: query, topK: 10 },
  });
  const candidates = results.result.hits.map((hit: any) => ({
    id: hit._id,
    text: hit.fields.text as string,
  }));
  const reranked = await pinecone.inference.rerank(
    'bge-reranker-v2-m3',
    query,
    candidates.map((c) => c.text),
    { topN: 3 }
  );
  return reranked.data.map((r: any) => candidates[r.index].text);
}

What does this code do?

The updated imports add NextResponse for JSON responses and the Pinecone client for connecting to your index.

The retrieveChunks function does two things. First, it calls searchRecords to find the 10 most semantically similar chunks to the user's question. Then it passes those 10 candidates through Pinecone's bge-reranker-v2-m3 reranking model to re-score them and return only the top 3 most relevant results.

This two-stage approach is standard in production RAG systems. Semantic search casts a wide net, then reranking narrows it down to the most precise matches.

Now you need to expose this function as an API endpoint so the frontend can call it.

  • In the same file, scroll down to the POST function. Find this code:
export async function POST(req: NextRequest) {
  return runtime.handle(req);
}
  • Replace the entire POST function with this updated version that handles retrieval requests:
export async function POST(req: NextRequest) {
  const url = new URL(req.url);
  // Handle retrieval requests at /api/copilotkit/retrieve
  if (url.pathname.endsWith('/retrieve')) {
    const { query } = await req.json();
    const chunks = await retrieveChunks(query);
    return NextResponse.json({ chunks });
  }
  return runtime.handle(req);
}

What does this code do?

The updated POST handler checks whether the incoming request URL ends with /retrieve. If it does, the handler extracts the query from the request body, runs the retrieval and reranking pipeline, and returns the top 3 chunks as JSON.

All other POST requests fall through to the CopilotKit runtime as before. This keeps the copilot's chat functionality intact while adding a new retrieval endpoint.

  • Save app/api/copilotkit/[[...slug]]/route.ts.
  • Check the terminal where your dev server is running. You should see no compilation errors.

Seeing a compilation error?

  • Make sure your PINECONE_API_KEY is set correctly in .env.local.
  • Check that the new import line says NextResponse alongside NextRequest in the same import from next/server.
  • Confirm the retrieveChunks function appears above the const runtime line, not below it.

Why is my retrieval route not compiling?

✔️ Awesome, I've got everything!

Great. Double check you have saved route.ts and there are no errors in the terminal.

ⓧ I'd like to double check the full code

Scroll up through your file and confirm these sections appear in order:

  • The four import lines (@copilotkit/runtime, @ai-sdk/google, next/server, @pinecone-database/pinecone).
  • The Pinecone client initialization (const pinecone and const index).
  • The retrieveChunks function.
  • The const runtime block (unchanged from before).
  • The updated POST handler with the /retrieve check.
  • The GET handler (unchanged).
Register the searchDocuments action on the frontend

The retrieval endpoint is ready on the server. Now the copilot needs to know when and how to call it. You will register a useCopilotAction that fires whenever the copilot wants to search your documents.

  • Switch back to app/page.tsx.
  • Add this import line at the top of the file, below the existing CopilotSidebar import:
import { useCopilotAction } from '@copilotkit/react-core';
  • Inside the Home component, add the following useCopilotAction hook above the handleFileUpload function. Copy this code:
  useCopilotAction({
    name: 'searchDocuments',
    description: 'Search the uploaded document for relevant information to answer the user question.',
    parameters: [{ name: 'query', type: 'string', description: 'The user question to search for.' }],
    handler: async ({ query }: { query: string }) => {
      const res = await fetch('/api/copilotkit/retrieve', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ query }),
      });
      const data = await res.json();
      return data.chunks.join('\n\n');
    },
  });

What does this code do?

The useCopilotAction hook registers an action called searchDocuments with the copilot. The copilot can now automatically call this action when it decides it needs document context to answer a question.

When triggered, the handler sends the user's question to your /api/copilotkit/retrieve endpoint. The endpoint searches Pinecone, reranks the results, and returns the top 3 chunks. Those chunks are joined into a single string and passed back to the copilot as grounded context for its response.

  • Save app/page.tsx.
  • Check the terminal. You should see no compilation errors.

Seeing an import error?

  • Make sure the useCopilotAction import is from @copilotkit/react-core, not from @copilotkit/react-ui.
  • Confirm the useCopilotAction call is inside the Home component function, not outside it.

Why is my useCopilotAction not working?

✔️ Awesome, I've got everything!

Great. Double check you have saved page.tsx and there are no errors in the terminal.

ⓧ I'd like to double check the full code

Scroll through your page.tsx and confirm these sections appear in order:

  • The 'use client' directive at the very top.
  • Three import lines: CopilotSidebar from @copilotkit/react-ui, useCopilotAction from @copilotkit/react-core, and useState plus uuidv4.
  • The useCopilotAction block with name: 'searchDocuments', placed before handleFileUpload.
  • The handleFileUpload function (unchanged from before).
  • The return block with the sidebar, upload input, and parsed markdown display (unchanged).
Test the full RAG pipeline

Everything is wired up. The copilot can now search your documents, rerank the results, and use them to answer questions. Time to prove it works.

  • Return to your browser at localhost:3000.
  • Upload a PDF or image that contains some factual content you can ask about.
  • Wait for the status message to confirm chunks were stored in Pinecone.

Before you ask the copilot a question, think back to Step 2. When you asked the copilot about your documents before, it made up an answer because it had no document context. Do you think the answer will be different this time?

  • In the copilot sidebar, type a question about specific content in your uploaded document.

The copilot now responds with a specific, accurate answer drawn from the actual content of your document. Compare this to the hallucinated response from Step 2. The difference is RAG in action: the copilot searched your stored chunks, reranked them for relevance, and used them as grounded context.

What is happening behind the scenes?

When you type a question, the copilot decides to call the searchDocuments action. The action sends your question to the /api/copilotkit/retrieve endpoint.

The endpoint searches Pinecone for the 10 most semantically similar chunks, then reranks them with bge-reranker-v2-m3 to pick the top 3. Those chunks are returned to the copilot as context. The copilot uses them to generate a grounded, accurate response instead of hallucinating.

Now test document updates to confirm the pipeline stays current.

  • Upload a modified version of the same document (or a different document).
  • Wait for the ingest confirmation.
  • Ask the copilot a question about the new content.

The copilot reflects the updated content. The ingest route deletes old vectors by document ID before inserting new ones, so the copilot always answers from the latest version.

Copilot still giving vague answers?

  • Make sure you waited for the "Stored X chunks in Pinecone" confirmation before asking a question. The chunks need to be indexed before they can be searched.
  • Try asking a very specific question that uses exact terms from your document. This helps the retrieval pipeline find relevant chunks.
  • Check the browser console for any network errors when the copilot tries to call /api/copilotkit/retrieve.

Why is my copilot not using my document content in its answers?

Your RAG copilot is fully operational. Documents go in, get parsed by Landing AI ADE, embedded and stored in Pinecone, and retrieved with reranking whenever the copilot needs context. Next up, push your retrieval even further with an optional Secret Mission.

Secret mission

Add Hybrid Search for More Precise Retrieval

Your dense vector search finds semantically similar chunks, but it can miss exact terms like product codes, specific dollar amounts, or model numbers. In this bonus challenge, you will upgrade to hybrid search by combining dense semantic vectors with sparse keyword vectors, so your copilot catches both meaning and exact matches.

Clean Up Your Resources

Clean Up Your Resources

This project uses free-tier services with no ongoing charges. All three services (Pinecone Starter plan, Landing AI ADE Explore plan, and Google Gemini free tier) persist at no cost while under their free-tier limits.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • doc-rag-copilot project folder on your Desktop.
  • Pinecone index in the Pinecone console.
  • Google Gemini API key in Google AI Studio.
  • .env.local file containing your API keys.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this project.

  • Your Pinecone index stays active on the free Starter plan with no charges.
  • Your Landing AI ADE account persists on the free Explore plan. Credits are only consumed when you parse a document.
  • Your Google Gemini API key remains valid on the free tier with rate limits.
  • Your local doc-rag-copilot folder and .env.local file remain on your machine.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and accounts stay intact for when you return.

  • Stop the dev server by pressing Ctrl+C in the terminal where it is running.
  • All three services are on free tiers, so no further action is needed. Your Pinecone index, Landing AI ADE account, and Gemini API key persist at no cost.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Stop the dev server by pressing Ctrl+C in the terminal.
  • Delete the project folder by running the command for your operating system.

macOS

cd ~/Desktop
rm -rf doc-rag-copilot

Windows

cd ~\Desktop
Remove-Item -Recurse -Force doc-rag-copilot
  • Delete your Pinecone index. Open the Pinecone console at app.pinecone.io, find your index, click the three dots to the right of the index name, then click Delete.
  • Delete your Gemini API key. Open the AI Studio API Keys page at aistudio.google.com/app/apikey, find the key you created for this project, and delete it.

Nice Work!

Nice Work!

Nice work! You just built a full Document Intelligence RAG Copilot that parses, embeds, retrieves, and answers questions from any document you upload.

You've learned how to:

  • Built a RAG Copilot with a CopilotKit sidebar powered by Google Gemini that answers questions grounded in your actual document content instead of hallucinating.
  • Used Landing AI ADE to parse PDFs and images into structured Markdown with semantic chunking that reconstructs tables, detects layout, and preserves reading order.
  • Stored document chunks in Pinecone using integrated embeddings with the multilingual-e5-large model, then retrieved and reranked them with bge-reranker-v2-m3 to surface the most relevant matches.
  • Secret Mission: Completed an optional challenge to push your retrieval pipeline even further.

Ready to quiz yourself?