Build a Local RAG API with LangChain.js

Build a local Document Q&A API with vector search and SSE streaming.

Introduction

30 Second Summary

You have a folder full of company documents and someone asks you a question about one of them. You could dig through every file yourself, or you could ask an AI. The problem is, AI chatbots confidently make things up when they don't actually have the information.

In this project, you will build a local API that ingests your documents, stores them as searchable vectors, and answers questions grounded in real content. Answers stream back in real-time with source citations so you can verify them.

What You'll Build

You drop Markdown files into a folder, hit one endpoint to ingest them, then ask questions and watch answers stream back token-by-token in your terminal with exact file references cited at the end.

By the end of this project, you'll have:

  • A document ingestion pipeline that reads Markdown files, splits them into chunks, generates vector embeddings via Ollama, and stores everything in ChromaDB running in Docker.
  • A streaming Q&A endpoint that retrieves relevant chunks, generates an answer using a local LLM, and delivers tokens via Server-Sent Events with source file names cited at the end.
  • A conversation memory system that tracks chat history per session so follow-up questions get coherent answers without re-explaining context.
  • Secret Mission: Extend the system to ingest PDF documents alongside Markdown.

Are there any prerequisites?

You should have basic TypeScript or JavaScript knowledge and familiarity with REST APIs. You'll also need Node.js 22+, Cursor or VS Code, and Docker already installed on your machine.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

You're building something that solves a real problem: large language models are confident but clueless about your private data. By the end of this project, you'll have a local API that grounds every answer in your actual documents.

Set Up Ollama, ChromaDB, and the TypeScript Project

Before you can build a RAG pipeline, you need three building blocks in place: a local LLM to generate answers, a vector database to store and search document chunks, and a TypeScript API to orchestrate everything.

In this step, you'll install Ollama (which serves AI models locally on your machine), start a ChromaDB container with Docker, and scaffold your Express server with TypeScript.

In this step, get ready to:
  • Install Ollama and pull the chat and embedding models.
  • Start ChromaDB in a Docker container.
  • Initialize the TypeScript project with all dependencies.
Install Ollama and pull models

Ollama lets you run large language models entirely on your local machine. You'll need two models: one for generating chat responses and one for creating vector embeddings from text.

  • Open your browser and go to ollama.com/download.
  • Download OllamaSetup.exe and run the installer.

What is Ollama?

Ollama is a tool that downloads and serves open-source AI models on your own machine. Instead of sending data to a cloud API, everything stays local. It exposes models at http://localhost:11434 so other tools (like your TypeScript app) can talk to them.

  • Once the installer finishes, open PowerShell by pressing the Windows key, typing PowerShell, and pressing Enter.
  • Verify Ollama is installed by running:
ollama --version

✔️ I see a version number

Ollama is ready. Continue to pull the models below.

ⓧ Command not found

The installer may not have finished adding Ollama to your PATH. Try closing and reopening PowerShell. If it still fails, restart your computer and try again.

Still not working?

Make sure you ran the OllamaSetup.exe installer completely (not just downloaded it). If you see a security warning, click Run anyway.

Help me fix the Ollama command not found issue

  • Pull the chat model (about 4.7GB download) by running:
ollama pull llama3.1

You'll see a progress bar as the model downloads. This can take a few minutes depending on your internet speed.

  • Pull the embedding model (about 274MB) by running:
ollama pull nomic-embed-text
  • Confirm both models are downloaded by running:
ollama list

You should see both llama3.1 and nomic-embed-text listed with their sizes.

Model download stuck or failing?

If the download hangs, press Ctrl+C to cancel and run the pull command again. Ollama resumes interrupted downloads automatically.

Help me troubleshoot model downloads

Create the Docker Compose file and start ChromaDB

ChromaDB is the vector database that will store your document embeddings and handle similarity search. You'll run it as a Docker container so it stays isolated from the rest of your system.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-api
cd rag-api
  • Open Cursor by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Cursor, and pressing Enter.
  • In Cursor, click File in the top menu bar.
  • Click Open Folder.
  • Select the rag-api folder on your Desktop, then click Open.
  • Create a new file by clicking File then New File.
  • Name the file docker-compose.yml and paste the following code:
services:
  chromadb:
    image: chromadb/chroma
    ports:
      - "8000:8000"
    volumes:
      - chroma-data:/chroma/chroma

volumes:
  chroma-data:

What does this file do?

Docker Compose defines the services your project needs. This file tells Docker to pull the official chromadb/chroma image, map port 8000 on your machine to port 8000 inside the container, and persist data in a named volume so it survives container restarts.

  • Save docker-compose.yml with Ctrl+S (Windows) or Cmd+S (macOS).
  • Open Cursor's integrated terminal by pressing Ctrl+` (Windows) or Cmd+` (macOS).
  • Start ChromaDB in the background by running:
docker compose up -d
  • Verify the container is running by running:
docker ps

You should see a container using the chromadb/chroma image with port 8000 mapped.

Container not showing up?

Make sure Docker Desktop is running before you execute docker compose up -d. If you see a connection error, open Docker Desktop and wait until it shows "Engine running" in the bottom-left corner.

Help me troubleshoot Docker Compose

Initialize the TypeScript project

Now you'll set up the Node.js project with all the LangChain packages, the ChromaDB client, and Express. This gives you everything you need to build the RAG API.

  • Initialize a new Node.js project by running:
npm init -y
  • Install the runtime dependencies by running:
npm install @langchain/ollama @langchain/community @langchain/core @langchain/textsplitters chromadb express
  • Install the development dependencies by running:
npm install -D @types/express @types/node tsx typescript

What are these packages?

The @langchain/ollama package connects LangChain to your local Ollama models. The @langchain/community package provides the ChromaDB vector store integration. The @langchain/textsplitters package handles breaking documents into chunks.

The tsx tool lets you run TypeScript files directly without a separate compile step. The type packages (@types/express, @types/node) provide TypeScript with type information for Express and Node.js.

Next, you need to configure the package.json and tsconfig.json files so TypeScript and ES modules work correctly.

  • Open package.json in Cursor's Explorer sidebar (click the file name).
  • Replace the entire contents of package.json with the following:
{
  "name": "rag-api",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "start": "tsx src/index.ts"
  },
  "dependencies": {
    "@langchain/community": "^0.3.26",
    "@langchain/core": "^0.3.40",
    "@langchain/ollama": "^0.2.0",
    "@langchain/textsplitters": "^0.1.0",
    "chromadb": "^1.9.2",
    "express": "^4.21.2"
  },
  "devDependencies": {
    "@types/express": "^5.0.0",
    "@types/node": "^22.0.0",
    "tsx": "^4.19.0",
    "typescript": "^5.7.0"
  }
}

Why replace the whole file?

The npm init -y command generates a basic package.json but it's missing critical fields. Setting "type": "module" enables ES module imports (the import syntax). The scripts section gives you a dev command that auto-restarts the server when you save changes.

  • Save package.json.
  • Create a new file called tsconfig.json in the project root (File > New File) and paste:
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "esModuleInterop": true,
    "strict": true,
    "outDir": "dist",
    "rootDir": "src",
    "skipLibCheck": true
  },
  "include": ["src"]
}

What does tsconfig.json do?

This file tells TypeScript how to compile your code. The "moduleResolution": "bundler" setting matches how modern tools like tsx resolve imports. The "strict": true flag catches common errors at compile time.

  • Save tsconfig.json.
  • Create a src folder by right-clicking in the Explorer sidebar and selecting New Folder. Name it src.
  • Right-click the src folder and select New File. Name it index.ts.
  • Paste the following code into src/index.ts:
import express from "express";

const app = express();
app.use(express.json());

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

What does this code do?

This creates a minimal Express server that listens on port 3000. The express.json() middleware parses incoming JSON request bodies, which you'll need when your API receives questions and file paths.

  • Save src/index.ts.
  • Re-install dependencies to match the updated package.json by running:
npm install

Before you run the server, what do you think you'll see in the terminal?

  • Start the Express server by running:
npx tsx src/index.ts

You should see Server running on port 3000 printed in the terminal. That confirms Express, TypeScript, and all your dependencies are working together.

  • Press Ctrl+C to stop the server.

Seeing a module or import error?

Make sure your package.json has "type": "module" set. Without it, Node.js tries to use CommonJS and the import statement fails.

Also check that you ran npm install after replacing package.json. The versions from the initial npm init install may differ from what the config now expects.

Help me fix this module error

✔️ Awesome, I've got everything!

Great. Make sure you've saved all three files (docker-compose.yml, package.json, tsconfig.json) and src/index.ts.

ⓧ I'd like to double check the full code

Scroll up and confirm you have these four files in your project, in order:

  • docker-compose.yml (the Docker Compose service definition).
  • package.json (with "type": "module" and both dependency blocks).
  • tsconfig.json (TypeScript compiler options).
  • src/index.ts (the Express server entry point).

Each chunk above shows the exact contents of these files. Make sure they match.

Your environment is fully wired. Ollama is serving models, ChromaDB is storing vectors, and your Express server is ready for routes. Next up, you'll build your first chat endpoint and see firsthand why a plain LLM needs retrieval.

Build a Naive Chat Endpoint

Your environment is ready. Ollama is serving models, ChromaDB is storing vectors, and your Express server starts up clean.

Now you need to see what happens when you ask an LLM a question it has no context for. In this step, you will build a basic chat endpoint and ask it about your company's refund policy. The model has never seen your documents, so it will confidently make up an answer. This hallucination problem is exactly what retrieval-augmented generation solves.

In this step, get ready to:
  • Create a chat endpoint using LangChain's LCEL pipe syntax.
  • Test it with a domain-specific question the model cannot know.
  • Observe the hallucination problem firsthand.
Create the chat route file

The chat endpoint needs its own route file. This keeps your Express server modular as you add more endpoints in later steps.

  • Create the routes directory inside src by running this command:
mkdir src/routes
  • Confirm the directory was created by running:
ls src

You should see routes listed alongside index.ts.

  • Create a new file called chat.ts inside src/routes/ in Cursor, then paste the following code:
import { Router } from "express";
import { ChatOllama } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";

export const chatRouter = Router();

// Initialize the LLM with deterministic output (temperature 0)
const model = new ChatOllama({ model: "llama3.1", temperature: 0 });
// Define system instruction and user message slot
const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant. Answer the user's question."],
  ["human", "{question}"],
]);
// Chain: prompt -> model -> parse output as plain string
const chain = prompt.pipe(model).pipe(new StringOutputParser());

chatRouter.post("/", async (req, res) => {
  const { question } = req.body;
  const answer = await chain.invoke({ question });
  res.json({ answer });
});

What does this code do?

  • ChatOllama connects to your local Ollama server and uses the llama3.1 model you pulled in Step 1.
  • ChatPromptTemplate.fromMessages() defines a system instruction and a user message slot. The {question} placeholder gets filled at runtime.
  • The .pipe() calls chain the components together using LCEL (LangChain Expression Language). Data flows from prompt to model to output parser in sequence.
  • StringOutputParser extracts just the text content from the model's response object, so you get a clean string back.
  • Save src/routes/chat.ts.

✔️ Awesome, I've got everything!

Great. Make sure chat.ts is saved before moving on.

ⓧ I'd like to double check the full code

import { Router } from "express";
import { ChatOllama } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";

export const chatRouter = Router();

const model = new ChatOllama({ model: "llama3.1", temperature: 0 });
const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant. Answer the user's question."],
  ["human", "{question}"],
]);
const chain = prompt.pipe(model).pipe(new StringOutputParser());

chatRouter.post("/", async (req, res) => {
  const { question } = req.body;
  const answer = await chain.invoke({ question });
  res.json({ answer });
});
Wire the route into your server

Your server does not know about the new route yet. You need to import it and mount it at the /chat path.

  • Open src/index.ts in Cursor.
  • Find this line at the top of the file:
import express from "express";
  • Add the chatRouter import directly below it, so the top of the file looks like this:
import express from "express";
import { chatRouter } from "./routes/chat.js";
  • Find this line in the middle of the file:
app.use(express.json());
  • Add the route mount directly below it:
// Mount the chat route
app.use("/chat", chatRouter);

Why the .js extension in the import?

TypeScript with ESM modules requires you to use the .js extension in import paths, even though the source file is .ts. This is because the import resolves at runtime after compilation, when the file will be JavaScript.

  • Save src/index.ts.
  • Start your server by running this command:
npx tsx src/index.ts

You should see Server running on port 3000 in the terminal.

Seeing a TypeScript error?

  • Check that the import path in src/index.ts uses ./routes/chat.js (not .ts).
  • Make sure chatRouter is exported with the export keyword in chat.ts.
  • Confirm both files are saved.

Still stuck? Help me fix a TypeScript error when starting my Express server

✔️ Awesome, I've got everything!

Great. Both files are saved and the server is running.

ⓧ I'd like to double check the full code

import express from "express";
import { chatRouter } from "./routes/chat.js";

const app = express();
app.use(express.json());

app.use("/chat", chatRouter);

app.listen(3000, () => {
  console.log("Server running on port 3000");
});
Test with a question the model cannot know

Your endpoint is wired up and the server is running. Time to see what happens when you ask the LLM a question about documents it has never seen.

  • Open a second terminal. You need it because the server is occupying your first one.

Before you send the request, think about this: you are about to ask the model about your company's refund policy. The model has never seen any of your documents. What do you think it will say?

  • Send a request to the chat endpoint by running this command:
curl http://localhost:3000/chat -X POST -H "Content-Type: application/json" -d "{\"question\": \"What is the refund policy?\"}"

The model responds confidently with a detailed refund policy. It sounds authoritative. It might mention 30-day windows, receipt requirements, or restocking fees.

Every single detail is fabricated. The model has no access to your documents, so it invented plausible-sounding information from its training data. This is called hallucination.

Why is this a problem?

An LLM without context will always generate an answer. It never says "I don't know" unless specifically instructed to. For a document Q&A system, this is dangerous because users trust confident-sounding responses.

This is the core limitation that RAG (Retrieval-Augmented Generation) solves. Instead of letting the model guess, you feed it the actual document content before it answers. You will build that retrieval pipeline in the next steps.

Not getting a response?

  • Make sure Ollama is running. Open a new terminal and run ollama list to confirm llama3.1 is available.
  • Check that your server shows Server running on port 3000 in the first terminal. If not, look for TypeScript errors in the output.
  • The first request may take 10-30 seconds while Ollama loads the model into memory. Wait for it to complete.

Still stuck? Help me debug why my /chat endpoint is not responding

You have seen the problem firsthand. The model sounds convincing but invents everything. In the next step, you will load your actual documents into a vector database so the model has real content to draw from.

Ingest Documents into ChromaDB

In the previous step, you saw the model confidently fabricate answers about documents it has never seen. The fix is straightforward: give the model access to your actual documents.

In this step, you will load Markdown files, split them into retrieval-sized chunks, generate vector embeddings via Ollama, and store everything in ChromaDB. Once your documents are indexed, the retrieval chain in the next step can find the right chunks to ground the model's answers.

In this step, get ready to:
  • Create sample Markdown documents for the knowledge base.
  • Build an ingestion endpoint that chunks and embeds documents into ChromaDB.
  • Verify the documents are stored by checking the ChromaDB collection.
Create sample documents for the knowledge base

Before you can ingest anything, you need documents to ingest. You will create three short Markdown files covering different topics so you can later verify that retrieval finds the right one.

  • Create a docs folder inside your rag-api project by running this command:
mkdir docs
  • In Cursor, create a new file at docs/refund-policy.md and paste the following content:
# Refund Policy

All purchases are eligible for a full refund within 30 days of purchase.

## Conditions

- The product must be unused and in original packaging
- Digital products can be refunded if less than 10% of content has been accessed
- Refund requests must be submitted through our support portal

## Processing Time

Refunds are processed within 5-7 business days after approval. The refund will appear on your original payment method.

## Exceptions

- Sale items are final sale and cannot be refunded
- Custom or personalized products are non-refundable
- Subscriptions can be cancelled but the current billing period is non-refundable
  • Create a second file at docs/features.md and paste this content:
# Product Features

## Core Features

- Real-time collaboration: Multiple users can edit documents simultaneously
- Version history: Every change is tracked with full rollback capability
- Smart search: AI-powered search across all your documents
- Export options: PDF, Word, HTML, and Markdown export

## Premium Features

- Custom branding: Add your company logo and colors
- API access: Full REST API for programmatic access
- Priority support: 24/7 dedicated support team
- Advanced analytics: Usage statistics and engagement metrics

## Integrations

- Slack: Get notifications and share documents directly
- GitHub: Sync documentation with your repositories
- Jira: Link documents to tickets and sprints
  • Create a third file at docs/troubleshooting.md and paste this content:
# Troubleshooting Guide

## Login Issues

If you cannot log in, try the following steps:
1. Clear your browser cache and cookies
2. Reset your password via the "Forgot Password" link
3. Ensure your account has not been deactivated (contact support)

## Sync Problems

Documents not syncing? Check these common causes:
- Ensure you have a stable internet connection
- Check if the service status page shows any outages
- Try logging out and back in to refresh your session token

## Performance Issues

If the application is running slowly:
- Close unnecessary browser tabs
- Disable browser extensions that may interfere
- Try using a different browser (Chrome or Firefox recommended)
- Contact support if issues persist for more than 24 hours
  • Confirm the files exist by running:
ls docs

You should see features.md, refund-policy.md, and troubleshooting.md listed in the output.

Don't see the files?

Make sure you created the docs folder inside your rag-api project root, not inside src. Your terminal should be in the rag-api directory when you run ls docs.

help me find where my docs folder ended up

Build the ingestion endpoint

Now you need an API endpoint that reads those Markdown files, splits them into smaller chunks, and stores their vector embeddings in ChromaDB. Chunking matters because LLMs have limited context windows. Smaller chunks also improve retrieval precision.

  • Create a new file at src/routes/ingest.ts and add the imports and setup:
import { Router } from "express";
import { readdirSync, readFileSync } from "fs";
import { join } from "path";
import { Document } from "@langchain/core/documents";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { OllamaEmbeddings } from "@langchain/ollama";
import { Chroma } from "@langchain/community/vectorstores/chroma";

export const ingestRouter = Router();

// Configure the embedding model for vector generation
const embeddings = new OllamaEmbeddings({ model: "nomic-embed-text" });

What do these imports do?

  • readdirSync and readFileSync read files from disk without needing async callbacks.
  • RecursiveCharacterTextSplitter breaks large documents into smaller, overlapping chunks for better retrieval.
  • OllamaEmbeddings generates vector embeddings locally using the nomic-embed-text model you pulled earlier.
  • Chroma is LangChain's integration for storing and querying vectors in ChromaDB.
  • Below the embeddings line, add the POST route handler that reads, chunks, and stores documents:
ingestRouter.post("/", async (req, res) => {
  const { directory } = req.body;

  // Read all Markdown files from the specified directory
  const files = readdirSync(directory).filter((f) => f.endsWith(".md"));
  const docs: Document[] = [];

  for (const file of files) {
    const content = readFileSync(join(directory, file), "utf-8");
    docs.push(new Document({ pageContent: content, metadata: { source: file } }));
  }

  // Split documents into chunks for better retrieval accuracy
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 1000,
    chunkOverlap: 200,
  });
  const chunks = await splitter.splitDocuments(docs);

  // Generate embeddings and store in ChromaDB
  await Chroma.fromDocuments(chunks, embeddings, {
    collectionName: "documents",
    url: "http://localhost:8000",
  });

  res.json({ message: `Ingested ${chunks.length} chunks from ${files.length} files` });
});

What does this route do?

The handler receives a directory path in the request body, reads every .md file from it, and wraps each in a LangChain Document with the filename stored as metadata.

RecursiveCharacterTextSplitter breaks each document into 1000-character chunks with 200-character overlap. The overlap ensures that sentences at chunk boundaries are not cut off and lost.

Chroma.fromDocuments() takes all chunks, sends them through nomic-embed-text to generate vectors, and stores everything in the ChromaDB collection named documents.

  • Save src/routes/ingest.ts.

Now you need to mount this router in your Express app so it's accessible at /ingest.

  • In src/index.ts, find this line near the top:
import { chatRouter } from "./routes/chat.js";
  • Add the ingest import directly below it, so the imports look like this:
import { chatRouter } from "./routes/chat.js";
import { ingestRouter } from "./routes/ingest.js";
  • Then find this line:
app.use("/chat", chatRouter);
  • Add the ingest route directly below it:
app.use("/chat", chatRouter);
app.use("/ingest", ingestRouter);
  • Save src/index.ts.

What should I see?

If you have npm run dev running, the server auto-restarts via tsx watch and prints "Server running on port 3000" again. If you stopped it, restart it now with npm run dev.

Seeing a TypeScript or import error?

Make sure the import uses ./routes/ingest.js (with the .js extension), not .ts. This is required for ESM module resolution in TypeScript projects using the bundler module resolution strategy.

help me fix my import error in index.ts

Test ingestion and verify documents are stored

The endpoint is wired up. Time to feed it your sample documents and confirm they land in ChromaDB.

  • Make sure your server is running. If it is not, start it by running:
npm run dev
  • In a separate terminal, trigger the ingestion by running this curl command:
curl http://localhost:3000/ingest -X POST -H "Content-Type: application/json" -d "{\"directory\": \"./docs\"}"

What should I see?

This may take 30-60 seconds the first time while Ollama generates embeddings for each chunk. You should see a JSON response like:

{"message":"Ingested 7 chunks from 3 files"}

The exact chunk count may vary slightly depending on how the splitter divides your documents. Any number above 0 means it worked.

Before you check ChromaDB directly, what do you think you will find when you query its REST API for the collection?

  • Open your browser and navigate to:
http://localhost:8000/api/v1/collections

You should see a JSON array containing a collection named documents. This confirms your chunks were embedded and stored successfully.

Seeing an empty array or connection error?

If the array is empty, the curl command may not have completed successfully. Check the terminal where your server is running for error messages.

If you get a connection refused error in the browser, make sure your ChromaDB container is still running by checking docker ps. If it stopped, run docker compose up -d from your project folder to restart it.

help me debug why my documents aren't showing up in ChromaDB

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/routes/ingest.ts and src/index.ts.

ⓧ I'd like to double check the full code

Your src/routes/ingest.ts should contain these two sections in order:

  • The imports and embeddings setup (7 import lines, router export, embeddings config).
  • The ingestRouter.post("/", ...) handler that reads files, splits them, and stores them in ChromaDB.

Your src/index.ts should look like this:

import express from "express";
import { chatRouter } from "./routes/chat.js";
import { ingestRouter } from "./routes/ingest.js";

const app = express();
app.use(express.json());

app.use("/chat", chatRouter);
app.use("/ingest", ingestRouter);

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

Your documents are now chunked, embedded, and stored in ChromaDB. In the next step, you will build a retrieval chain that queries these vectors and feeds the relevant chunks to the LLM, finally fixing that hallucination problem.

Build the Retrieval Chain with Source Citations

In Step 2, you saw the model confidently fabricate a refund policy it never had access to. The fix is retrieval: before the LLM generates a response, you fetch the actual document chunks that contain the answer and pass them as context.

In this step, you will build a /ask endpoint that connects to your ChromaDB collection, retrieves the most relevant chunks for a question, and feeds them into the LLM via a RAG chain. The response includes both the answer and the source files it drew from.

In this step, get ready to:
  • Connect to the existing ChromaDB collection and create a retriever.
  • Build a RAG chain using LCEL pipe syntax.
  • Create a /ask endpoint that returns grounded answers with source citations.
Create the ask route with the RAG prompt

The key to fixing hallucination is a prompt that constrains the model. You will create a system prompt that instructs the LLM to answer ONLY from provided context, refusing to guess when it has no relevant information.

  • Create a new file src/routes/ask.ts and add the following imports, model configuration, and prompt template:
import { Router } from "express";
import { ChatOllama } from "@langchain/ollama";
import { OllamaEmbeddings } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
import { Chroma } from "@langchain/community/vectorstores/chroma";
import { Document } from "@langchain/core/documents";

export const askRouter = Router();

// Same models as the ingest step
const model = new ChatOllama({ model: "llama3.1", temperature: 0 });
const embeddings = new OllamaEmbeddings({ model: "nomic-embed-text" });

What does this code do?

The imports bring in the same ChatOllama model and OllamaEmbeddings you used in earlier steps. You also import Chroma (to connect to the existing collection) and Document (for the TypeScript type that represents retrieved chunks).

Setting temperature: 0 makes the model deterministic. It will give the same answer for the same context every time, which is important for a factual Q&A system.

  • Below the embeddings line, add the RAG prompt and a helper function to format retrieved documents:
// Instruct the model to ONLY use provided context
const RAG_PROMPT = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are a helpful assistant. Answer the question based ONLY on the following context. If the context does not contain the answer, say "I don't have that information in my documents."

Context:
{context}`,
  ],
  ["human", "{question}"],
]);

// Join retrieved document chunks into a single context string
function formatDocs(docs: Document[]): string {
  return docs.map((doc) => doc.pageContent).join("\n\n");
}

What does this code do?

The RAG_PROMPT is a prompt template with two messages. The system message tells the LLM to only answer from the {context} placeholder (which you will fill with retrieved chunks). If the context lacks the answer, it must say so instead of guessing.

The formatDocs helper takes an array of retrieved documents and joins their text content with double newlines. This creates one string that the prompt template consumes as context.

  • Save src/routes/ask.ts.
Add the retriever and route handler

The retriever connects to your existing ChromaDB collection and performs a similarity search to find the 3 most relevant chunks for any question. The route handler ties everything together: retrieve, generate, and return sources.

  • Add the following code below the formatDocs function in src/routes/ask.ts:
// Connect to existing ChromaDB collection
async function getRetriever() {
  const vectorStore = await Chroma.fromExistingCollection(embeddings, {
    collectionName: "documents",
    url: "http://localhost:8000",
  });
  return vectorStore.asRetriever({ k: 3 });
}

askRouter.post("/", async (req, res) => {
  const { question } = req.body;

  // Retrieve the 3 most relevant chunks
  const retriever = await getRetriever();
  const retrievedDocs = await retriever.invoke(question);

  // Build the LCEL chain: prompt -> model -> output
  const chain = RAG_PROMPT.pipe(model).pipe(new StringOutputParser());
  const answer = await chain.invoke({
    context: formatDocs(retrievedDocs),
    question,
  });

  // Extract source metadata from retrieved docs
  const sources = retrievedDocs.map((doc) => ({
    source: doc.metadata.source,
    content: doc.pageContent.slice(0, 150) + "...",
  }));

  res.json({ answer, sources });
});

What does this code do?

The getRetriever() function connects to the "documents" collection you created in Step 3. The k: 3 setting means it returns the 3 chunks whose vector embeddings are most similar to the question's embedding.

The LCEL pipe syntax chains three steps: the prompt template formats the context and question into messages, ChatOllama generates the response, and StringOutputParser extracts the text. Each .pipe() call connects one step's output to the next step's input.

The sources array extracts the original file name from each chunk's metadata, plus a 150-character content preview. This lets the API consumer verify where the answer came from.

  • Save src/routes/ask.ts.

Seeing a red underline on the Chroma import?

  • Make sure you have the chromadb npm package installed. It is a peer dependency of @langchain/community for the Chroma integration.
  • Confirm the import path is exactly @langchain/community/vectorstores/chroma with no typos.

Help me fix a Chroma import error in my LangChain.js project.

Mount the route and test the endpoint

The route file is ready, but the server does not know about it yet. You need to import and mount it in your main entry point.

  • In src/index.ts, find this line:
import { ingestRouter } from "./routes/ingest.js";
  • Add the following import directly below it:
import { askRouter } from "./routes/ask.js";
  • Then find this line:
app.use("/ingest", ingestRouter);
  • Add the following line directly below it:
app.use("/ask", askRouter);
  • Save src/index.ts.
  • Restart your server by running:
npx tsx src/index.ts

You should see Server running on port 3000 with no TypeScript errors.

Seeing a TypeScript error on startup?

  • Check that the import path in src/index.ts uses ./routes/ask.js (with the .js extension). TypeScript with ESM resolution requires the .js extension even for .ts source files.
  • Verify that src/routes/ask.ts exports askRouter (not a default export).
  • Make sure your ChromaDB container is still running. Run docker ps to confirm.

Help me debug my Express server startup error after adding the ask router.

Now for the real test. Before you run this, think: do you expect the model to hallucinate like it did with the /chat endpoint, or give a grounded answer this time?

  • Test the /ask endpoint with the same refund policy question by running:
curl http://localhost:3000/ask -X POST -H "Content-Type: application/json" -d "{\"question\": \"What is the refund policy?\"}"

This time, the answer matches your actual document content. You should see a response mentioning a 30-day refund window, along with a sources array listing refund-policy.md as the source file.

Why is this answer different from the /chat endpoint?

The /chat endpoint gives the LLM zero context. It can only guess. The /ask endpoint retrieves the actual document chunks from ChromaDB and passes them into the prompt. The model reads your real content and summarizes it, instead of fabricating an answer from its training data.

The sources array proves provenance. A consumer of this API can verify exactly which files contributed to the answer.

Getting an empty answer or connection error?

  • If you see a connection refused error, make sure both Ollama and the ChromaDB Docker container are running.
  • If the response says it has no information, re-run the ingest step from Step 3 to repopulate the collection.
  • If the response takes a long time, that is normal. The llama3.1 model processes retrieved context and generates a full answer locally.

Help me troubleshoot my /ask endpoint returning an error or empty response.

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/routes/ask.ts and src/index.ts.

ⓧ I'd like to double check the full code

Here is the complete src/routes/ask.ts file split into three sections. Scroll through and confirm each section is present in order: imports and setup, RAG_PROMPT and formatDocs, then getRetriever and the route handler.

import { Router } from "express";
import { ChatOllama } from "@langchain/ollama";
import { OllamaEmbeddings } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
import { Chroma } from "@langchain/community/vectorstores/chroma";
import { Document } from "@langchain/core/documents";

export const askRouter = Router();

const model = new ChatOllama({ model: "llama3.1", temperature: 0 });
const embeddings = new OllamaEmbeddings({ model: "nomic-embed-text" });
const RAG_PROMPT = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are a helpful assistant. Answer the question based ONLY on the following context. If the context does not contain the answer, say "I don't have that information in my documents."

Context:
{context}`,
  ],
  ["human", "{question}"],
]);

function formatDocs(docs: Document[]): string {
  return docs.map((doc) => doc.pageContent).join("\n\n");
}
async function getRetriever() {
  const vectorStore = await Chroma.fromExistingCollection(embeddings, {
    collectionName: "documents",
    url: "http://localhost:8000",
  });
  return vectorStore.asRetriever({ k: 3 });
}

askRouter.post("/", async (req, res) => {
  const { question } = req.body;

  const retriever = await getRetriever();
  const retrievedDocs = await retriever.invoke(question);

  const chain = RAG_PROMPT.pipe(model).pipe(new StringOutputParser());
  const answer = await chain.invoke({
    context: formatDocs(retrievedDocs),
    question,
  });

  const sources = retrievedDocs.map((doc) => ({
    source: doc.metadata.source,
    content: doc.pageContent.slice(0, 150) + "...",
  }));

  res.json({ answer, sources });
});

And here is the updated src/index.ts:

import express from "express";
import { chatRouter } from "./routes/chat.js";
import { ingestRouter } from "./routes/ingest.js";
import { askRouter } from "./routes/ask.js";

const app = express();
app.use(express.json());

app.use("/chat", chatRouter);
app.use("/ingest", ingestRouter);
app.use("/ask", askRouter);

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

Your RAG pipeline is working. The model now grounds every answer in real document content and tells you exactly where it found the information. Next up, you will add real-time streaming so answers appear token-by-token, plus conversation memory so users can ask follow-up questions.

Stream Responses with SSE and Conversation Memory

Your /ask endpoint works, but it makes the user wait for the entire response to generate before anything appears. For longer answers, this means staring at a blank screen for several seconds.

Server-Sent Events let you push tokens to the client the instant the model generates them. On top of that, conversation memory lets users ask follow-up questions without repeating context every time.

In this step, get ready to:
  • Create a streaming endpoint that delivers tokens one at a time via SSE.
  • Add in-memory conversation history so follow-up questions work.
  • Test multi-turn conversation with the same session ID.
Create the streaming endpoint with SSE headers

Server-Sent Events work by keeping an HTTP connection open and sending data frames as the server produces them. Express supports this natively through res.setHeader and res.write.

  • In src/routes/ask.ts, find the line that says const embeddings = new OllamaEmbeddings({ model: "nomic-embed-text" });.
  • Directly below it, add the conversation history store and the first half of the streaming handler:
const chatHistory: Map<string, { role: string; content: string }[]> = new Map();

askRouter.post("/stream", async (req, res) => {
  const { question, sessionId = "default" } = req.body;

  // Tell the client to expect a stream, not a single JSON response
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  // Retrieve relevant chunks from ChromaDB
  const retriever = await getRetriever();
  const retrievedDocs = await retriever.invoke(question);

  // Load any prior conversation for this session
  const history = chatHistory.get(sessionId) || [];
  const historyText = history
    .map((msg) => `${msg.role}: ${msg.content}`)
    .join("\n");

  // Build a prompt that includes context AND chat history
  const promptWithHistory = ChatPromptTemplate.fromMessages([
    [
      "system",
      `You are a helpful assistant. Answer the question based ONLY on the following context. If the context does not contain the answer, say "I don't have that information in my documents."

Context:
{context}

Chat History:
{history}`,
    ],
    ["human", "{question}"],
  ]);

What does this code do?

The three setHeader calls tell the browser (or curl) that this response is a stream, not a single JSON payload. The connection stays open so the server can push data as it becomes available.

The chatHistory Map stores past Q&A pairs keyed by sessionId. When a session has prior messages, they get formatted into the prompt so the model knows what was discussed before.

Add the streaming loop and memory storage

Now you need the code that actually streams tokens to the client and saves the conversation for future turns.

  • Directly below the code you just added (still inside the handler), paste the streaming and memory logic:
  const chain = promptWithHistory.pipe(model).pipe(new StringOutputParser());

  // Stream tokens one at a time to the client
  let fullAnswer = "";
  const stream = await chain.stream({
    context: formatDocs(retrievedDocs),
    question,
    history: historyText,
  });

  for await (const chunk of stream) {
    fullAnswer += chunk;
    res.write(`data: ${JSON.stringify({ token: chunk })}\n\n`);
  }

  // Send sources as a final SSE event and close the connection
  const sources = retrievedDocs.map((doc) => ({
    source: doc.metadata.source,
    content: doc.pageContent.slice(0, 150) + "...",
  }));

  res.write(`data: ${JSON.stringify({ done: true, sources })}\n\n`);
  res.end();

  // Store this exchange for future follow-up questions
  history.push({ role: "human", content: question });
  history.push({ role: "assistant", content: fullAnswer });
  chatHistory.set(sessionId, history);
});

What does this code do?

  • chain.stream() returns an async iterable. Each iteration yields a small text chunk (often a single word or token) from the LLM.
  • res.write() pushes each chunk as an SSE data: frame. The double newline (\n\n) is required by the SSE protocol to mark the end of each event.
  • After the stream finishes, a final event carries the sources array so the client knows where the answer came from.
  • The last three lines store this Q&A pair so the next request with the same sessionId gets the prior conversation prepended into the prompt.
  • Save src/routes/ask.ts.

Seeing a TypeScript error?

Make sure the new code sits below the existing askRouter.post("/", ...) handler and above nothing else. The chatHistory Map must be declared at module level (not inside a function).

If you see an error about chatHistory not being found, check that the declaration sits between the embeddings line and the RAG_PROMPT definition.

help me debug a TypeScript error in my ask.ts streaming endpoint

Test streaming and multi-turn conversation

Time to see tokens flow in real-time. You will hit the new /ask/stream endpoint and watch the response build token by token in your terminal.

  • Restart your server by running:
npx tsx src/index.ts

Before you run the next command, predict what you expect to see. Will the response appear all at once like the /ask endpoint, or differently?

  • Open a second terminal and test the streaming endpoint by running:
curl -N http://localhost:3000/ask/stream -X POST -H "Content-Type: application/json" -d "{\"question\": \"What is the refund policy?\", \"sessionId\": \"abc123\"}"

You should see data: lines appearing one at a time as the model generates each token. The final line contains {"done":true,"sources":[...]} with the source file names.

Why the -N flag?

The -N flag tells curl to disable output buffering. Without it, curl may wait for the entire response before printing anything, defeating the purpose of streaming.

  • Now test conversation memory by asking a follow-up with the same sessionId:
curl -N http://localhost:3000/ask/stream -X POST -H "Content-Type: application/json" -d "{\"question\": \"How long is the window?\", \"sessionId\": \"abc123\"}"

The model should respond with "30 days" (referencing the refund policy from your first question) without you having to mention refunds again. It knows the context because the prior exchange is stored under sessionId abc123.

Not seeing tokens stream in?

  • Make sure you included the -N flag in your curl command. Without it, output may buffer until the stream ends.
  • If you get a connection refused error, confirm your server is running on port 3000 in the other terminal.
  • If the follow-up question does not reference your earlier question, verify both requests use the same sessionId value (abc123).

help me debug my SSE streaming endpoint that is not sending tokens incrementally

✔️ Awesome, I've got everything!

Great. Double check you have saved src/routes/ask.ts with both the chatHistory Map and the full /stream handler.

ⓧ I'd like to double check the full code

Here is the complete src/routes/ask.ts file. Scroll through and confirm your code matches.

Because this file exceeds 30 lines, verify each section in order:

  • Imports at the top (Router, ChatOllama, OllamaEmbeddings, ChatPromptTemplate, StringOutputParser, RunnablePassthrough, RunnableSequence, Chroma, Document).
  • Module-level declarations: askRouter, model, embeddings, chatHistory Map, RAG_PROMPT, formatDocs, and getRetriever.
  • The POST / handler (unchanged from Step 4).
  • The new POST /stream handler with SSE headers, retrieval, history, streaming loop, sources event, and history storage.

Your API now streams answers token by token and remembers previous questions within a session. You have a complete local RAG system: ingest documents, ask questions with grounded answers and source citations, stream responses in real-time, and maintain conversation context across turns.

Secret mission

Add PDF Document Support

Your RAG pipeline only handles Markdown files right now. Real-world document systems need to handle PDFs too. In this bonus challenge, you'll install a PDF parsing library and extend your ingest endpoint to automatically detect and process PDF files alongside Markdown.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing cloud costs.

Resources you used:

  • ChromaDB Docker container running on port 8000 (stores vector data in a Docker volume).
  • Ollama models downloaded locally (llama3.1 at ~4.7GB, nomic-embed-text at ~274MB).
  • rag-api project folder on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building on this project or want to keep the RAG API available for local development.

  • Your ChromaDB container continues running in the background and retains all ingested documents.
  • Ollama stays available in your system tray, ready to serve model requests.
  • Your project folder remains on your Desktop for further development.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping all your data and files intact for later.

  • Stop the ChromaDB container by running this in your terminal:
  • Stop your Express server with Ctrl+C in the terminal where it is running.
  • Quit Ollama from the system tray (right-click the Ollama icon and select Quit).
docker compose stop

When you come back, start everything up again with docker compose start and relaunch Ollama from your Start menu.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your ingested documents and downloaded models.

  • Stop and remove the ChromaDB container along with its stored data by running:
docker compose down -v
  • Delete the Ollama models by running these commands:
ollama rm llama3.1
ollama rm nomic-embed-text
  • Delete the project folder. On macOS or Linux:
cd ~/Desktop
rm -rf rag-api

On Windows (PowerShell):

cd ~\Desktop
Remove-Item -Recurse -Force rag-api
  • Delete the sample docs folder if it exists outside your project directory.

Nice Work!

Nice Work!

You just built a fully local RAG API from scratch. Your system ingests documents, stores vector embeddings, retrieves relevant context, and streams grounded answers back to the client with source citations.

You've learned how to:

  • Build a complete RAG pipeline end-to-end: loading documents, chunking them with RecursiveCharacterTextSplitter, generating embeddings via Ollama, and storing them in ChromaDB for similarity search.
  • Compose a streaming retrieval chain using LangChain.js LCEL pipe syntax with ChatOllama, delivering tokens in real-time via Server-Sent Events and returning source citations so users can verify answers.
  • Implement conversation memory for multi-turn Q&A and use Docker Compose to containerize your vector database for clean, reproducible infrastructure.
  • Secret Mission: Extended the ingest pipeline to handle PDF documents alongside Markdown using pdf-parse.

Ready to quiz yourself?