Build a Local RAG API with Node.js

Build a REST API that ingests documents and answers questions using local LLMs.

Introduction

30 Second Summary

Every AI chatbot has the same blind spot. It can only answer from what it was trained on, not from your specific documents, notes, or files.

In this project, you will build a local RAG API that answers questions using your own documents. You'll feed it text, it will break it into searchable pieces, and when you ask a question, it will find the most relevant pieces and generate a grounded answer.

What You'll Build

You'll have a running API on your machine where you can POST any text document, then ask questions in plain English and get back answers that cite specific passages from your files.

By the end of this project, you'll have:

  • A document ingestion pipeline where you POST text and watch it get chunked and embedded into searchable vector embeddings.
  • A semantic search endpoint where you ask a question and see ranked results showing which document chunks are most relevant, scored by cosine similarity.
  • A query endpoint that generates natural language answers grounded in your documents using a local LLM, with source citations and relevance scores.
  • Secret Mission: A confidence gate that makes your API refuse to answer when it doesn't have enough relevant information.

Do I need any prior experience?

This project assumes you're comfortable writing JavaScript and working with Node.js. You should also have a basic understanding of how REST APIs work (making requests and handling JSON responses).

Before We Start

You're about to build a local RAG API that answers questions using your own documents. This teaches you how AI-powered document Q&A systems work from the ground up.

You need an embedding model and a chat model running locally before you can build the pipeline that uses them. In this step, you'll pull both models and set up your Node.js project with the right dependencies.

In this step, get ready to:
  • Pull the all-minilm and llama3.1 models in Ollama.
  • Initialize a Node.js project with ES modules and install dependencies.
Pull the required Ollama models

Your RAG pipeline needs two models: one to convert text into vector embeddings, and one to generate natural language answers. Both run entirely on your machine through Ollama.

  • Press the Windows key to open your search bar.
  • Type Terminal and press Enter to open it.
  • Pull the embedding model by running this command:
ollama pull all-minilm

What is all-minilm?

This is a small embedding model that converts text into 384-dimension vectors. These vectors capture the meaning of text so you can compare how similar two pieces of writing are.

You should see a download progress bar complete in a few seconds (the model is about 46MB).

  • Pull the chat model by running this command:
ollama pull llama3.1

What is llama3.1?

This is an 8-billion parameter language model that generates natural language answers. At approximately 4.7GB, this download will take a few minutes depending on your connection speed.

Wait for the download to finish completely before continuing.

Before you run the next command, what do you expect to see in the output?

  • Confirm both models are available by running:
ollama list

You should see both all-minilm and llama3.1 listed in the output with their sizes.

Don't see both models listed?

  • Make sure Ollama is running. Open a new terminal and run ollama serve if it is not already active.
  • If a download failed partway through, re-run the pull command for that model.
  • help me troubleshoot why ollama list is not showing my models
Initialize the Node.js project

With both models ready, you can set up the Node.js project that will become your RAG API.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-api
cd rag-api
  • Initialize a new Node.js project by running:
npm init -y

You should see a JSON printout of your new package.json file in the terminal.

  • Set the project to use ES modules by running:
npm pkg set type=module

Why ES modules?

This command adds "type": "module" to your package.json. It tells Node.js to use modern import/export syntax instead of the older require() pattern.

  • Install the project dependencies by running:
npm install express@4.21.2 ollama@0.5.14

What are these packages?

  • express@4.21.2 is a web framework that handles HTTP requests for your API endpoints.
  • ollama@0.5.14 is the JavaScript client that talks to your local Ollama models for embedding and chat.

You should see npm download and add both packages. The terminal prints a summary like "added 2 packages" when it finishes.

  • Confirm your package.json is correct by running:
cat package.json

You should see "type": "module" and both express and ollama listed under dependencies.

Missing dependencies or type field?

  • If "type": "module" is missing, re-run npm pkg set type=module inside your rag-api folder.
  • If dependencies are missing, make sure you ran the install command from inside the rag-api folder (not from Desktop).
  • help me fix my package.json for this RAG API project

Your local environment is ready. You have both AI models available and a configured Node.js project. Next up, you'll build the ingestion pipeline that breaks documents into searchable chunks.

Set Up the Express Server

Your models are ready, but you need a way to communicate with them over HTTP. An Express server gives you clean API endpoints to send documents in and get answers back.

In this step, you'll create a minimal server file with a health check endpoint. This confirms that Express and Ollama are both connected before you build the full RAG pipeline on top.

In this step, get ready to:
  • Create a server.js file with Express configured for JSON payloads.
  • Define model constants and an in-memory vector store.
  • Add a health check endpoint and start the server.
Create the server file

Your project folder already has package.json and node_modules from the previous step. Now you need a single entry point that wires everything together.

  • Create a new file called server.js in your project folder and add the following code:
import express from 'express';
import ollama from 'ollama';

const app = express();
app.use(express.json({ limit: '10mb' }));

const EMBED_MODEL = 'all-minilm';
const CHAT_MODEL = 'llama3.1';

// In-memory vector store
const vectorStore = [];

// Health check
app.get('/health', (req, res) => {
  res.json({ status: 'ok', totalChunks: vectorStore.length });
});

app.listen(3000, () => {
  console.log('RAG API running at http://localhost:3000');
});

What does this code do?

  • The imports pull in Express for the HTTP server and the Ollama client library for talking to your local models.
  • express.json({ limit: '10mb' }) lets the server accept JSON request bodies up to 10MB, which is important because you'll be sending entire documents.
  • EMBED_MODEL and CHAT_MODEL are constants so you can swap models in one place later.
  • vectorStore is the array that will hold your document chunks and their embeddings in memory.
  • The /health endpoint returns how many chunks are currently stored, giving you a quick sanity check.
  • Save server.js.

File not saving?

Make sure you created server.js in the same folder as your package.json. If you're using VS Code, check the Explorer sidebar to confirm both files are at the same level.

help me figure out why my server.js file isn't in the right place

✔️ Awesome, I've got everything!

Great. Make sure you've saved server.js before moving on to start the server.

ⓧ I'd like to double check the full code

Here's the complete server.js file. Compare it against yours character for character.

import express from 'express';
import ollama from 'ollama';

const app = express();
app.use(express.json({ limit: '10mb' }));

const EMBED_MODEL = 'all-minilm';
const CHAT_MODEL = 'llama3.1';

// In-memory vector store
const vectorStore = [];

// Health check
app.get('/health', (req, res) => {
  res.json({ status: 'ok', totalChunks: vectorStore.length });
});

app.listen(3000, () => {
  console.log('RAG API running at http://localhost:3000');
});
Start the server and test the health endpoint

Before you run the server, what do you think will appear in the terminal?

  • Start the server by running this command:
node server.js

You should see this output in your terminal:

RAG API running at http://localhost:3000

Your terminal is now occupied

The server keeps running until you stop it. Open a second terminal window to run the next command.

When you need to stop the server later, press Ctrl+C in the terminal where it's running.

  • In your new terminal, test the health endpoint by running:
curl http://localhost:3000/health

You should see this JSON response:

{"status":"ok","totalChunks":0}

The totalChunks is 0 because you haven't ingested any documents yet. That changes in the next step.

Seeing 'Cannot find module' or 'ERR_MODULE_NOT_FOUND'?

  • Confirm your package.json contains "type": "module". Without this, Node.js treats your file as CommonJS and the import syntax fails.
  • Check that node_modules exists in the same folder as server.js. If not, run npm install again.
  • Make sure you're running the command from inside your project folder where package.json lives.

help me debug my Node.js module import error

Curl not working on Windows PowerShell?

In PowerShell, curl is an alias for a different command. Use curl.exe instead (with the .exe extension) to call the real curl.

Your server is alive and responding. Next up, you'll build the ingestion pipeline that breaks documents into chunks and converts them into vector embeddings.

Build the Ingestion Pipeline

Your server is running and responding to health checks. Now it needs a way to accept documents and prepare them for search.

Raw text documents are too large to search meaningfully. If you feed an entire article to a search algorithm, it cannot pinpoint which paragraph answers your question. In this step, you will break documents into small, overlapping chunks and convert each chunk into a vector embedding that captures its meaning.

In this step, get ready to:
  • Build a text chunking function that splits documents into overlapping segments.
  • Create an ingestion endpoint that embeds chunks and stores them in memory.
Implement text chunking

A single document might be thousands of characters long. Searching it as one giant block would lose precision. Chunking splits the text into smaller segments, each short enough to embed and compare individually.

Why overlap chunks?

If you split at hard boundaries, a sentence that spans two chunks gets cut in half. Overlap means the end of one chunk repeats at the start of the next, so context at boundaries is never lost.

  • In server.js, find the line that says const vectorStore = [];.
  • Add the following chunking function below it:
// Chunk text into overlapping segments
function chunkText(text, chunkSize = 500, overlap = 50) {
  const chunks = [];
  let start = 0;
  while (start < text.length) {
    const end = Math.min(start + chunkSize, text.length);
    chunks.push(text.slice(start, end));
    start += chunkSize - overlap;
  }
  return chunks;
}

What does this code do?

  • chunkSize = 500 sets each chunk to roughly 500 characters.
  • overlap = 50 repeats the last 50 characters of each chunk at the start of the next one.
  • The while loop slides a window across the text, pushing each segment into the chunks array until the entire document is covered.
  • Save server.js.

The function is ready but has no caller yet. The next chunk builds the endpoint that uses it.

Create the ingestion endpoint

Chunking alone produces plain text segments. To search by meaning rather than keywords, each chunk needs a vector embedding that represents its semantic content as an array of numbers.

Ollama's embed() function converts text into these vectors using the all-minilm model. You can pass all chunks as an array in one batch call, which is faster than embedding them one by one.

  • In server.js, find the health check endpoint (the app.get('/health', ...) block).
  • Add the following /ingest endpoint below it:
// POST /ingest - Ingest a document
app.post('/ingest', async (req, res) => {
  const { text, source } = req.body;

  if (!text) {
    return res.status(400).json({ error: 'text field is required' });
  }

  const chunks = chunkText(text);
  const sourceName = source || 'unknown';

  const response = await ollama.embed({
    model: EMBED_MODEL,
    input: chunks,
  });

  for (let i = 0; i < chunks.length; i++) {
    vectorStore.push({
      text: chunks[i],
      embedding: response.embeddings[i],
      source: sourceName,
      chunkIndex: i,
    });
  }

  res.json({
    message: `Ingested ${chunks.length} chunks from "${sourceName}"`,
    totalChunks: vectorStore.length,
  });
});

What does this code do?

  • It destructures text and source from the request body, returning a 400 error if text is missing.
  • chunkText(text) splits the document into overlapping segments.
  • ollama.embed() sends all chunks as a batch to the all-minilm model and returns one embedding vector per chunk.
  • The for loop stores each chunk's text, its embedding, the source name, and its index in the vectorStore array.
  • The response tells the caller how many chunks were created and the total stored so far.
  • Save server.js.
  • Switch to the terminal where your server is running and press Ctrl+C to stop it.
  • Restart the server by running:
node server.js

You should see RAG API running at http://localhost:3000 again.

Before you test, think about this: you are about to send a block of text to your endpoint. What do you expect the response to contain?

  • Open a second terminal and send a test document by running:

macOS

curl -X POST http://localhost:3000/ingest \
  -H "Content-Type: application/json" \
  -d '{"text": "Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval with text generation. Instead of relying solely on a language model training data, RAG systems first search a knowledge base for relevant documents, then use those documents as context when generating a response. This approach reduces hallucination because the model answers are grounded in actual source material. RAG pipelines typically involve several stages including document ingestion, text chunking, embedding generation, vector storage, retrieval, and finally generation with a language model. The chunking stage determines retrieval granularity.", "source": "rag-overview"}'

Windows

curl.exe -X POST http://localhost:3000/ingest -H "Content-Type: application/json" -d "{\"text\": \"Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval with text generation. Instead of relying solely on a language model training data, RAG systems first search a knowledge base for relevant documents, then use those documents as context when generating a response. This approach reduces hallucination because the model answers are grounded in actual source material. RAG pipelines typically involve several stages including document ingestion, text chunking, embedding generation, vector storage, retrieval, and finally generation with a language model. The chunking stage determines retrieval granularity.\", \"source\": \"rag-overview\"}"

You should see a JSON response confirming the ingestion:

{
  "message": "Ingested 2 chunks from \"rag-overview\"",
  "totalChunks": 2
}

Why 2 chunks?

The sample text is roughly 600 characters. The first chunk takes characters 0 through 499, and the second chunk picks up from character 450 onward. With a longer document, you would see more chunks.

Not seeing a response?

  • Make sure Ollama is running in the background. The server needs it to generate embeddings.
  • If you see a connection error, confirm your server is still running in the first terminal.
  • On Windows, use curl.exe (not curl) in PowerShell. PowerShell aliases curl to a different command.

Help me debug my /ingest endpoint

✔️ Awesome, I've got everything!

Great. Double check you have saved server.js with both the chunkText function and the /ingest endpoint.

ⓧ I'd like to double check the full code

Your server.js should contain these blocks in order. Scroll up and confirm each one is present:

  • Imports (express and ollama).
  • App setup with express.json({ limit: '10mb' }).
  • Constants EMBED_MODEL and CHAT_MODEL.
  • The vectorStore array.
  • The chunkText function (10 lines, with chunkSize = 500 and overlap = 50).
  • The GET /health endpoint.
  • The POST /ingest endpoint (chunks text, embeds, stores in vectorStore).
  • app.listen(3000, ...) at the bottom.

Your API can now swallow documents and store their meaning as vectors. Next up, you will build the retrieval layer that searches those vectors to find the chunks most relevant to a question.

Build Retrieval and Semantic Search

Your ingestion pipeline stores chunked documents as vectors. But right now there is no way to ask a question and find which chunks are relevant.

In this step, you will build the retrieval layer. You will implement cosine similarity to compare vectors mathematically, wire up a retrieval function that ranks stored chunks by relevance, and expose a /search endpoint to test it all.

In this step, get ready to:
  • Implement a cosine similarity function that measures how semantically close two pieces of text are.
  • Build a retrieval function that embeds a question and scores it against all stored chunks.
  • Create a POST /search endpoint that returns the most relevant chunks with their similarity scores.
Implement cosine similarity

Cosine similarity measures the angle between two vectors. Two texts with similar meaning produce vectors pointing in nearly the same direction, giving a score close to 1. Unrelated texts point in different directions, scoring closer to 0.

  • In server.js, find the closing brace of your chunkText function.
  • Add the following function directly below it:
// Compute cosine similarity between two vectors
function cosineSimilarity(a, b) {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < a.length; i++) {
    dotProduct += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

What does this code do?

This function takes two number arrays (vectors) and computes how similar they are.

  • The dotProduct captures how much the vectors point in the same direction.
  • The normA and normB values measure each vector's length, so the result is normalized between -1 and 1 regardless of vector magnitude.
  • A return value near 1 means the texts are semantically similar. Near 0 means unrelated.
  • Save server.js.
Build the retrieval function

The retrieval function takes a question, converts it into a vector using the same embedding model, then scores every stored chunk against it. The highest-scoring chunks are the most relevant to the question.

  • In server.js, find the closing brace of your new cosineSimilarity function.
  • Add the following function directly below it:
// Retrieve top-k relevant chunks
async function retrieve(question, topK = 3) {
  const response = await ollama.embed({
    model: EMBED_MODEL,
    input: question,
  });
  const questionEmbedding = response.embeddings[0];

  const scored = vectorStore.map((item) => ({
    text: item.text,
    source: item.source,
    chunkIndex: item.chunkIndex,
    score: cosineSimilarity(questionEmbedding, item.embedding),
  }));

  scored.sort((a, b) => b.score - a.score);
  return scored.slice(0, topK);
}

What does this code do?

This function chains three operations to find the best chunks for a given question.

  • It embeds the question with the same all-minilm model used during ingestion. Using the same model means the question vector lives in the same space as the document vectors.
  • It maps over every item in vectorStore and calls cosineSimilarity to score each chunk against the question.
  • It sorts by descending score and returns only the top-k results. The default is 3.
  • Save server.js.
  • Stop your running server with Ctrl+C, then restart it by running:
node server.js

You should see RAG API running at http://localhost:3000 printed in the terminal. This confirms both new functions have no syntax errors.

Seeing a syntax error on startup?

  • Check that both functions are placed between the closing brace of chunkText and the app.get('/health', ...) line. They must not be inside another function.
  • Confirm every opening brace { has a matching closing brace }. The retrieve function has nested objects inside .map() that are easy to mis-close.

Help me debug a syntax error when starting my Node.js server.

Create the POST /search endpoint

The /search endpoint lets you test retrieval in isolation. You send it a question, and it returns the most relevant chunks with their similarity scores. No LLM generation happens here.

  • In server.js, find the closing }); of your /ingest route (the line after res.json({ message: ...).
  • Add the following endpoint below it:
// POST /search - Search for relevant chunks
app.post('/search', async (req, res) => {
  const { question, topK } = req.body;

  if (!question) {
    return res.status(400).json({ error: 'question field is required' });
  }

  if (vectorStore.length === 0) {
    return res.status(400).json({ error: 'No documents ingested yet.' });
  }

  const results = await retrieve(question, topK || 3);
  res.json({
    results: results.map((r) => ({
      text: r.text,
      source: r.source,
      similarity: parseFloat(r.score.toFixed(4)),
    })),
  });
});

What does this code do?

  • It validates that a question field is present and that the vector store is not empty.
  • It calls retrieve() with the question and an optional topK parameter (defaulting to 3).
  • It formats each result with the chunk text, its source document name, and the similarity score rounded to 4 decimal places.
  • Save server.js.
  • Stop your server with Ctrl+C and restart it:
node server.js

Now test the full retrieval flow. First, ingest a sample document so there is something to search. Run this command:

curl -X POST http://localhost:3000/ingest -H "Content-Type: application/json" -d "{\"text\": \"Retrieval-Augmented Generation (RAG) is a technique that enhances large language models by providing them with relevant context from external documents. Instead of relying solely on training data, a RAG system retrieves specific information related to the user question and includes it in the prompt sent to the language model. This approach reduces hallucinations because the model can ground its answers in actual source material. The RAG pipeline typically involves several stages: document ingestion, text chunking, vector embedding, similarity search, and finally generation with retrieved context. Vector embeddings convert text into numerical representations that capture semantic meaning. Similar concepts end up as nearby points in vector space, enabling mathematical comparison of text similarity.\", \"source\": \"rag-overview\"}"

You should see a response confirming chunks were ingested.

Before you run the next command, what do you think the /search endpoint will return? A full generated answer, or something else?

  • Search for relevant chunks by running:
curl -X POST http://localhost:3000/search -H "Content-Type: application/json" -d "{\"question\": \"What does RAG do?\"}"

You will see the top chunks with their similarity scores. Something like this:

{
  "results": [
    {
      "text": "Retrieval-Augmented Generation (RAG) is a technique...",
      "source": "rag-overview",
      "similarity": 0.7842
    },
    {
      "text": "...document ingestion, text chunking, vector embedding...",
      "source": "rag-overview",
      "similarity": 0.6531
    }
  ]
}

You get back raw text snippets ranked by relevance. The search found the chunks most related to your question. But notice what is missing: there is no synthesized answer. Just fragments.

Why only raw chunks?

This is the retrieval half of RAG working in isolation. It proves that cosine similarity correctly identifies relevant text. But a user does not want to read three disconnected chunks.

They want a single coherent answer grounded in those chunks. That is the generation half, which you will build in the next step.

Getting an empty results array?

  • Make sure you ran the ingest command first. The server's in-memory store is empty each time you restart.
  • Confirm Ollama is still running in the background. The embed call needs Ollama's all-minilm model to be available.
  • Check the terminal where your server is running for any error messages.

Help me troubleshoot why my /search endpoint returns empty results.

✔️ Awesome, I've got everything!

Great. Double check you have saved server.js and your server restarts cleanly.

ⓧ I'd like to double check the full code

Your server.js should now contain these pieces in order. Scroll up and confirm each one is present:

  • Imports (express and ollama) and constants (EMBED_MODEL, CHAT_MODEL, vectorStore).
  • The chunkText function (from the previous step).
  • The cosineSimilarity function (added in this step).
  • The retrieve function (added in this step).
  • The /health GET endpoint.
  • The /ingest POST endpoint (from the previous step).
  • The /search POST endpoint (added in this step).
  • The app.listen(3000, ...) call at the very end.

Your retrieval layer is working. You can find relevant chunks from any ingested document using nothing but vector math. Next, you will close the gap between raw snippets and a real answer by wiring a local LLM into the pipeline.

Add LLM Generation with Retrieved Context

Your /search endpoint proves that semantic retrieval works. You can find the most relevant chunks for any question. But raw text snippets are not what a user expects when they ask a question.

A coherent, natural-language answer is the goal. In this step, you'll send the retrieved chunks as context to the llama3.1 model and have it generate a grounded response that cites your documents instead of hallucinating.

In this step, get ready to:
  • Build a prompt template that injects retrieved chunks as numbered context.
  • Call the local LLM with the augmented prompt and return the answer.
  • Test the full RAG pipeline end to end.
Build the prompt template and start the /query endpoint

The "augment" in RAG means injecting retrieved context into the LLM's prompt. You'll format chunks with index numbers so the model can reference them, and write a system prompt that constrains the LLM to answer only from the provided context.

  • In server.js, find the closing }); of the /search endpoint (the line just above app.listen).
  • Add the following code below it to start the /query endpoint:
// POST /query - Full RAG pipeline
app.post('/query', async (req, res) => {
  const { question, topK } = req.body;

  if (!question) {
    return res.status(400).json({ error: 'question field is required' });
  }

  if (vectorStore.length === 0) {
    return res.status(400).json({ error: 'No documents ingested yet. POST to /ingest first.' });
  }

  const topChunks = await retrieve(question, topK || 3);

  const context = topChunks
    .map((chunk, i) => `[${i + 1}] ${chunk.text}`)
    .join('\n\n');

  const systemPrompt = `You are a helpful assistant. Answer the user's question using ONLY the context provided below. If the answer is not in the context, say "I don't have enough information to answer that."

Context:
${context}`;

What does this code do?

  • The endpoint validates the request and checks that documents have been ingested before doing any work.
  • retrieve(question, topK || 3) reuses the function you built in the previous step to find the top matching chunks.
  • The context variable formats each chunk with a number prefix like [1], [2], [3] so the LLM can reference specific chunks in its answer.
  • The systemPrompt constrains the model. It tells the LLM to answer ONLY from the provided context. This is what prevents hallucination.
Add the LLM call and complete the endpoint

With the prompt template ready, you pass it to ollama.chat() along with the user's question. The model generates an answer grounded in your chunks rather than its general training data.

  • Directly below the systemPrompt variable you just added, paste this code to complete the endpoint:
  const chatResponse = await ollama.chat({
    model: CHAT_MODEL,
    messages: [
      { role: 'system', content: systemPrompt },
      { role: 'user', content: question },
    ],
  });

  res.json({
    answer: chatResponse.message.content,
    sources: topChunks.map((chunk) => ({
      text: chunk.text.slice(0, 100) + '...',
      source: chunk.source,
      similarity: parseFloat(chunk.score.toFixed(4)),
    })),
  });
});

What does this code do?

  • ollama.chat() sends the system prompt (with injected context) and the user's question to the local llama3.1 model.
  • The response includes the generated answer from chatResponse.message.content plus a sources array showing which chunks informed the answer and how relevant each one was.
  • Truncating chunk text to the first 100 characters keeps the response payload small while still letting the user identify each source.
  • Save server.js.

Seeing a syntax error on save?

  • Make sure the systemPrompt template literal ends with a backtick (`) followed by a semicolon on the line that reads ${context}`;.
  • Confirm the closing }); of the /query endpoint appears before app.listen.
  • help me fix a syntax error in my /query endpoint

✔️ Awesome, I've got everything!

Great. Make sure you've saved server.js before moving on to the test.

ⓧ I'd like to double check the full code

Your server.js is now long enough that inlining it here would create a wall of code. Scroll up and confirm these sections appear in order:

  • Imports and constants (EMBED_MODEL, CHAT_MODEL, vectorStore).
  • chunkText function.
  • cosineSimilarity function.
  • retrieve function.
  • GET /health endpoint.
  • POST /ingest endpoint.
  • POST /search endpoint.
  • POST /query endpoint (the one you just added).
  • app.listen(3000, ...) at the very end.
Test the full RAG pipeline

Your server's in-memory store clears each time it restarts. You'll need to restart the server, ingest a document, and then query it.

  • Stop the running server with Ctrl+C in your terminal.
  • Restart the server by running:
node server.js
  • Ingest a sample document by running this in a second terminal:
curl -X POST http://localhost:3000/ingest -H "Content-Type: application/json" -d "{\"text\": \"Node.js is a JavaScript runtime built on Chrome's V8 engine. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. Node.js package ecosystem, npm, is the largest ecosystem of open source libraries in the world. Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.\", \"source\": \"node-intro\"}"

You should see a response confirming chunks were ingested.

Before you run the next command, predict: will you get back raw text chunks or a full natural-language answer?

  • Query the full RAG pipeline by running:
curl -X POST http://localhost:3000/query -H "Content-Type: application/json" -d "{\"question\": \"What is Express?\"}"

You get back a complete JSON response with a generated answer field containing a natural-language explanation, plus a sources array showing which chunks informed the answer with their similarity scores.

Why is the response slow?

The llama3.1 model generates text token by token on your local hardware. The first request may take 10-30 seconds depending on your CPU/GPU. Subsequent requests are faster because the model stays loaded in memory.

Getting a connection error or empty response?

  • Make sure Ollama is running in the background. Open a new terminal and run ollama list to confirm it responds.
  • Confirm your server is running on port 3000. You should see RAG API running at http://localhost:3000 in the terminal where you started it.
  • If you see a JSON error about text field is required on the ingest step, check that your curl command has the correct JSON structure with a text field.
  • help me debug why my /query endpoint is not returning an answer

Your full RAG pipeline is working. You can ingest any text, ask questions about it, and get grounded answers from a local LLM. Next up, you'll make this system smarter by adding confidence gating in the Secret Mission.

Secret mission

Add Confidence Gating

Your RAG API always answers, even when it has zero relevant context. That means it can hallucinate confidently about topics your documents never mention. Can you add a relevance threshold that makes the system refuse to answer when retrieval scores are too low?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The only resources using disk space are the Ollama models you pulled and your project folder.

Decide whether to keep your resources on disk for future use, or remove them to reclaim space.

Resources you used:

  • Ollama models stored locally (all-minilm ~26MB, llama3.1 ~4.7GB).
  • Project folder containing server.js, package.json, and node_modules.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your RAG API or want to build on it further.

  • Your Ollama models stay cached on disk and are ready to use anytime with zero ongoing cost.
  • Your project folder stays on your Desktop.
  • To start the server again later, open your terminal, navigate to your project folder, and run node server.js.

Pause - I'll come back to this later

Stop the running server to free up the terminal, but keep all files and models in place so you can resume later.

  • In the terminal where your server is running, press Ctrl+C to stop the server.
  • Your models and project files remain on disk, ready to use when you come back.

Delete - I don't want to use this again

Remove all project resources to reclaim disk space. This deletes the models and all project files permanently.

  • In the terminal where your server is running, press Ctrl+C to stop the server.
  • Remove the Ollama models by running these commands:
ollama rm all-minilm
ollama rm llama3.1
  • Delete your project folder. Run the command for your operating system:

macOS

cd ~/Desktop
rm -rf rag-api

Windows

cd ~\Desktop
rmdir /s /q rag-api
  • Confirm the folder is gone by running ls (macOS) or dir (Windows) on your Desktop. Your project folder should no longer appear.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete Retrieval-Augmented Generation pipeline from scratch using Node.js and Ollama, running entirely on your own machine with zero API costs.

You've learned how to:

  • Build a complete RAG pipeline from scratch in JavaScript: ingest, chunk, embed, retrieve, augment, generate.
  • Use Ollama to run both an embedding model and a chat model entirely locally with zero cost.
  • Implement cosine similarity search for semantic document retrieval.
  • Construct grounded prompts that prevent LLM hallucination by anchoring answers in retrieved context.
  • Secret Mission: Added confidence gating so the system refuses to answer when retrieval scores are too low.

Ready to quiz yourself?