Build a Document Search Engine

Build an inverted index and TF-IDF search engine from scratch in TypeScript.

Introduction

30 Second Summary

Every time you type something into a search bar, an engine somewhere decides which results matter most and which to bury. That ranking decision happens in milliseconds, powered by data structures and algorithms you can build yourself.

In this project, you will build a command-line search engine from scratch that indexes text files, looks up matching documents instantly, and ranks results by relevance using the same algorithm that powered early Google.

What You'll Build

You run a single command, point it at a folder of documents and a search query, and get back a ranked list of the most relevant files with relevance scores and content snippets.

By the end of this project, you'll have:

  • A working CLI search tool where you point it at any folder of .txt files, type a query, and see ranked results instantly.
  • A TF-IDF scoring system you built from first principles, where a document mentioning a rare term like "quantum" visibly outranks one mentioning a common term like "data."
  • An inverted index you can inspect directly to see exactly how the engine finds documents without scanning every file.
  • Secret Mission: Add boolean AND/OR query support using posting list intersection, a classic interview algorithm.

Are there any prerequisites?

You should have basic TypeScript or JavaScript knowledge (variables, functions, loops). You also need Node.js installed on your machine.

Before We Start

Before we dive in, let's get clear on what you're about to build and why it matters for your engineering toolkit.

Set Up the Project

Your search engine is a TypeScript project that runs directly in the terminal. To start building algorithms, you first need a project folder with the right dependencies installed.

You'll use tsx to run TypeScript files without a separate compile step. This means every change you make runs instantly. No waiting for builds.

In this step, get ready to:
  • Initialize the project and install TypeScript tooling.
  • Create the project structure and entry point.
  • Run the entry point to confirm everything works.
Initialize the project and install TypeScript tooling

Every Node.js project starts with a folder and a package.json file. You'll create both, then install the two packages your project needs: TypeScript for type checking and tsx for running .ts files directly.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir search-engine
cd search-engine
  • Initialize a new Node.js project by running:
npm init -y

You should see a JSON object printed in the terminal showing your new package.json file contents.

  • Install TypeScript and tsx as development dependencies by running:
npm install --save-dev tsx typescript

You should see output confirming that packages were added. Two packages now live in your node_modules folder.

Why tsx?

Node.js recommends tsx for full TypeScript support including tsconfig.json handling. It uses esbuild under the hood, so your files run instantly without a separate compile step.

Seeing permission errors or command not found?

Make sure Node.js is installed by running node --version in your terminal. If nothing appears, install Node.js from the official website before continuing.

On macOS, if you see a permission error, try prefixing the install command with sudo.

Help me fix npm install errors in my search engine project.

Create the project structure and entry point

The auto-generated package.json has fields you don't need. You'll replace its contents with a clean configuration, then create the TypeScript config and your first source file.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the search-engine folder on your Desktop, then click Open.
  • In the Explorer sidebar, click package.json to open it.
  • Select all the existing content and replace it with this configuration:
{
  "name": "search-engine",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "search": "tsx src/index.ts"
  },
  "devDependencies": {
    "typescript": "^7.0.2",
    "tsx": "^4.23.1"
  }
}

What does this configuration do?

"type": "module" tells Node.js to use ES module syntax (import/export) instead of the older require() style.

The search script gives you a shortcut to run your search engine later with npm run search.

  • Save package.json with Cmd+S (macOS) or Ctrl+S (Windows).
  • Create a new file in the search-engine folder by clicking the new file icon in the Explorer sidebar.
  • Name it tsconfig.json and add this configuration:
{
  "compilerOptions": {
    "target": "ESNext",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

What does tsconfig.json do?

This file tells TypeScript how to check your code. "strict": true enables all type-checking rules so you catch bugs early.

"noEmit": true means TypeScript only checks types. The actual running is handled by tsx, not the TypeScript compiler.

  • Save tsconfig.json.
  • Confirm both config files exist by switching to your terminal and running:
ls

You should see node_modules, package.json, package-lock.json, and tsconfig.json listed in the output.

  • Create the src folder and your entry point file by running:
mkdir src
  • Back in VS Code, expand the new src folder in the Explorer sidebar.
  • Create a new file inside src called index.ts and add this single line:
console.log("Search engine starting...");
  • Save index.ts.
Run the entry point to confirm everything works

Your project structure is in place. Time to prove that tsx can find and run your TypeScript file.

  • Before you run this, what do you expect to see printed in the terminal?
  • Run your entry point by executing this command in your terminal:
npx tsx src/index.ts

You should see Search engine starting... printed to your terminal. That confirms TypeScript is running without any compile step.

Not seeing the output?

Make sure you're in the search-engine directory. Run pwd to check your current location.

If you see a "Cannot find module" error, confirm the src folder exists and contains index.ts by running ls src.

Help me fix my tsx command that isn't running my TypeScript file.

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files: package.json, tsconfig.json, and src/index.ts.

ⓧ I'd like to double check the full code

Here are all three files you should have at this point. Compare each one to yours.

{
  "name": "search-engine",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "search": "tsx src/index.ts"
  },
  "devDependencies": {
    "typescript": "^7.0.2",
    "tsx": "^4.23.1"
  }
}
{
  "compilerOptions": {
    "target": "ESNext",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}
console.log("Search engine starting...");

Your environment is ready. Next up, you'll build the tokenizer that breaks raw text into searchable terms and construct the inverted index data structure.

Build the Tokenizer and Index Your First Document

Your environment is set up and running TypeScript with zero config friction. Now you face the core problem: how does a search engine find a document without reading every single one?

The answer is an inverted index. Instead of scanning every document for a query term, you pre-build a map from every term to the documents that contain it. Lookup becomes O(1) instead of O(n). But before you can build that map, you need to break raw text into individual terms. That process is called tokenization.

In this step, get ready to:
  • Implement a tokenizer that breaks text into searchable terms.
  • Build an inverted index data structure that maps terms to posting lists.
  • Index sample documents and print the index to verify it works.
Implement the tokenizer

Raw text like "Machine learning is great" contains noise that would pollute search results. Words like "is" and "the" appear in nearly every document and add no search value. A tokenizer strips this noise by lowercasing everything, splitting on punctuation, and filtering out common stop words.

  • Create a new file called src/tokenizer.ts in VS Code.
  • Paste in the following code:
const STOP_WORDS = new Set([
  "the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
  "have", "has", "had", "do", "does", "did", "will", "would", "could",
  "should", "may", "might", "shall", "can", "need", "dare", "ought",
  "used", "to", "of", "in", "for", "on", "with", "at", "by", "from",
  "as", "into", "through", "during", "before", "after", "above", "below",
  "between", "out", "off", "over", "under", "again", "further", "then",
  "once", "here", "there", "when", "where", "why", "how", "all", "each",
  "every", "both", "few", "more", "most", "other", "some", "such", "no",
  "not", "only", "own", "same", "so", "than", "too", "very", "just",
  "because", "but", "and", "or", "if", "while", "about", "this", "that",
  "these", "those", "it", "its", "he", "she", "they", "them", "his",
  "her", "their", "what", "which", "who", "whom"
]);

export function tokenize(text: string): string[] {
  return text
    .toLowerCase()
    .split(/[^a-z0-9]+/)
    .filter(token => token.length > 1 && !STOP_WORDS.has(token));
}

What does this code do?

STOP_WORDS is a set of common English words that appear in almost every document. They add noise to search results because they carry no meaning for ranking.

The tokenize() function does three things in sequence: converts text to lowercase (so "Machine" matches "machine"), splits on any non-alphanumeric character, and filters out single-character tokens plus stop words.

Now test that the tokenizer works before building anything on top of it.

  • Open src/index.ts and replace its contents with this test code:
import { tokenize } from "./tokenizer.ts";

// Quick test: see what tokens come out of a sentence
console.log(tokenize("Machine learning is a subset of artificial intelligence. Algorithms learn from data."));
  • Save both files.
  • Run the tokenizer test by running this command:
npx tsx src/index.ts

You should see an array of meaningful terms with stop words like "is", "a", and "of" stripped out:

[
  'machine',     'learning',
  'subset',      'artificial',
  'intelligence', 'algorithms',
  'learn',       'data'
]

Seeing an import error?

Make sure your tokenizer.ts file is inside the src/ folder, not in the project root. The import path ./tokenizer.ts is relative to index.ts.

Help me debug my tokenizer import error

Build the inverted index data structure

With tokenization working, you can now build the data structure that makes search fast. An inverted index maps every unique term to a list of "postings." Each posting records which document contains the term and how many times it appears there.

  • Create a new file called src/index-builder.ts in VS Code.
  • Start by defining the type interfaces at the top of the file:
import { tokenize } from "./tokenizer.ts";

// Each posting records: which document has this term, and how often
export interface Posting {
  docId: number;
  termFrequency: number;
}

// Metadata we store for each document in the index
export interface DocumentEntry {
  id: number;
  name: string;
  totalTerms: number;
  content: string;
}

// The complete index: a map of terms to postings, plus the doc list
export interface InvertedIndex {
  postings: Map<string, Posting[]>;
  documents: DocumentEntry[];
}

What do these interfaces represent?

Posting is one entry in a posting list. It says "document #2 contains this term 3 times."

DocumentEntry stores metadata about each indexed document. The totalTerms field is important later for calculating how significant a term is relative to the document's length.

InvertedIndex ties it all together. The postings map is the core lookup structure. Given a term, you get back every document that contains it in O(1) time.

  • Now add the buildIndex function below the interfaces in the same file:
export function buildIndex(docs: { name: string; content: string }[]): InvertedIndex {
  const postings = new Map<string, Posting[]>();
  const documents: DocumentEntry[] = [];

  for (let i = 0; i < docs.length; i++) {
    // Tokenize the document content into searchable terms
    const tokens = tokenize(docs[i].content);
    documents.push({
      id: i,
      name: docs[i].name,
      totalTerms: tokens.length,
      content: docs[i].content,
    });

    // Count how many times each term appears in this document
    const termCounts = new Map<string, number>();
    for (const token of tokens) {
      termCounts.set(token, (termCounts.get(token) || 0) + 1);
    }

    // Add a posting for each term found in this document
    for (const [term, count] of termCounts) {
      if (!postings.has(term)) {
        postings.set(term, []);
      }
      postings.get(term)!.push({ docId: i, termFrequency: count });
    }
  }

  return { postings, documents };
}

What does buildIndex do?

For each document, it tokenizes the content, counts term frequencies, then adds a posting entry for each term. After processing all documents, the postings map lets you look up any term and instantly get back every document that contains it plus how many times.

This is the same structure that powers real search engines. Google's original index was an inverted index stored across thousands of machines.

  • Save src/index-builder.ts.
  • Confirm the new file is in place by running:
ls src/

You should see index-builder.ts, index.ts, and tokenizer.ts listed.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Your src/index-builder.ts should have two sections in this order:

  • The import and three interface definitions (Posting, DocumentEntry, InvertedIndex) at the top.
  • The buildIndex function below them.

Scroll up and confirm both chunks are present and in order.

Index sample documents and print the index

The tokenizer and index builder are ready. Now you wire them together with sample data to see the inverted index in action.

  • Open src/index.ts and replace its entire contents with this code:
import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";

// Three sample documents covering different topics
const sampleDocs = [
  { name: "doc1.txt", content: "Machine learning is a subset of artificial intelligence. Algorithms learn from data." },
  { name: "doc2.txt", content: "Quantum computing uses quantum mechanics to process information faster than classical computers." },
  { name: "doc3.txt", content: "Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths." }
];

// Build the index and print every term with its posting list
const index = buildIndex(sampleDocs);
console.log("Inverted Index:");
for (const [term, postings] of index.postings) {
  console.log(`  ${term}:`, postings);
}

What does this code do?

It feeds three short documents into buildIndex(), then iterates over the resulting inverted index to print every term alongside its posting list. Each posting shows which document contains the term and how many times.

  • Save src/index.ts.

Before you run this, what do you expect the output to look like? Think about which terms will map to multiple documents versus just one.

  • Run the indexer by running this command:
npx tsx src/index.ts

You should see the inverted index printed with terms mapped to their posting lists. Notice that "algorithms" appears in two posting entries (doc 0 and doc 2) because both documents contain it. Meanwhile "quantum" appears only in doc 1 with a frequency of 2.

Why does this matter?

Without an inverted index, searching for "quantum" means scanning every document. With the index, you look up "quantum" and instantly get back [{ docId: 1, termFrequency: 2 }]. No scanning required.

Seeing an empty index or errors?

If you see Cannot find module, check that the import path in index.ts includes the .ts extension: ./index-builder.ts not ./index-builder.

If the index prints but looks empty, make sure you saved tokenizer.ts with the full STOP_WORDS set and the export keyword on the tokenize function.

Help me debug my inverted index output

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved.

ⓧ I'd like to double check the full code

Here is the complete src/index.ts at this point:

import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";

const sampleDocs = [
  { name: "doc1.txt", content: "Machine learning is a subset of artificial intelligence. Algorithms learn from data." },
  { name: "doc2.txt", content: "Quantum computing uses quantum mechanics to process information faster than classical computers." },
  { name: "doc3.txt", content: "Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths." }
];

const index = buildIndex(sampleDocs);
console.log("Inverted Index:");
for (const [term, postings] of index.postings) {
  console.log(`  ${term}:`, postings);
}

Your inverted index is working. You can look up any term and instantly find which documents contain it. Next up, you will use this index to search for documents and see why unranked results are a problem.

Search the Index with Naive Matching

In Step 2 you built an inverted index that maps every term to the documents it appears in. That's the hard part done.

Now you can look up any word and instantly find which documents contain it. But there's a problem. If you search for "computer", every matching document looks equal. A document mentioning "computer" once appears the same as one mentioning it ten times. You cannot tell which result is actually most relevant.

In this step, get ready to:
  • Build a search function that looks up query terms in the inverted index.
  • Return matching documents with their raw term frequency as a score.
  • Observe why raw frequency alone fails to produce meaningful rankings.
Implement the search function

The search function takes a list of query terms, looks each one up in the inverted index, and collects every document that matches. For now, it uses raw term frequency as the score. This is the simplest possible approach.

  • Create a new file called src/scorer.ts in your src folder.
  • Add the import and the SearchResult interface at the top of the file:
import type { InvertedIndex } from "./index-builder.ts";

// Shape of each result returned by the search function
export interface SearchResult {
  docId: number;
  name: string;
  score: number;
  snippet: string;
}

What does this interface define?

Each SearchResult carries everything the caller needs to display a hit: which document matched, a relevance score, the document's name, and a short text snippet for preview.

  • Below the interface, add the search function:
export function searchWithTfIdf(
  queryTerms: string[],
  index: InvertedIndex
): SearchResult[] {
  // Accumulate raw term frequency per document
  const scores = new Map<number, number>();

  for (const term of queryTerms) {
    const postingList = index.postings.get(term);
    if (!postingList) continue;

    for (const posting of postingList) {
      // Add raw count — no weighting yet
      scores.set(posting.docId, (scores.get(posting.docId) || 0) + posting.termFrequency);
    }
  }

  // Build result objects from accumulated scores
  const results: SearchResult[] = [];
  for (const [docId, score] of scores) {
    const doc = index.documents[docId];
    const snippet = doc.content.substring(0, 100).trim();
    results.push({ docId, name: doc.name, score, snippet });
  }

  return results;
}

What does this function do?

  • For each query term, it looks up the posting list in the inverted index. If the term doesn't exist, it skips it.
  • For each posting (document that contains the term), it adds the raw term frequency to that document's running score.
  • Finally, it builds an array of SearchResult objects with a 100-character snippet from each matching document.

Notice: the results are NOT sorted. They come out in whatever order the Map happens to iterate.

  • Save src/scorer.ts.
Wire up the search in index.ts

The search function exists, but nothing calls it yet. You need to import it in src/index.ts and replace the index-printing code with actual search queries.

  • In src/index.ts, find the import lines at the top of the file:
import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";
  • Add a new import for searchWithTfIdf so the imports look like this:
import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";
import { searchWithTfIdf } from "./scorer.ts";
  • Now find the code at the bottom of the file that prints the inverted index:
console.log("Inverted Index:");
for (const [term, postings] of index.postings) {
  console.log(`  ${term}:`, postings);
}
  • Replace that entire block with the following search loop:
// Run multiple queries to test the search
const queries = ["computer", "algorithm", "quantum"];
for (const query of queries) {
  const queryTerms = tokenize(query);
  const results = searchWithTfIdf(queryTerms, index);
  console.log(`\nQuery: "${query}"`);
  for (const r of results) {
    console.log(`  [${r.score}] ${r.name}: ${r.snippet}`);
  }
}

What does this search loop do?

  • It runs three queries against the index: "computer", "algorithm", and "quantum".
  • Each query is tokenized first (lowercased, stopwords removed), then passed to searchWithTfIdf().
  • Results print with the raw score in brackets, the document name, and a snippet. This lets you compare how each query performs.
  • Save src/index.ts.
Test with multiple queries and observe the problem

Before you run this, think about what you expect. If you search for "algorithm", which document do you think should rank first? doc3.txt mentions "algorithms" and "algorithm" (two hits). doc1.txt mentions "algorithms" (one hit). Do you think the output will reflect that difference clearly?

  • Run the search by executing this command in your terminal:
npx tsx src/index.ts

You should see output like this:

Query: "computer"
  [1] doc2.txt: Quantum computing uses quantum mechanics to process information faster than classical computers.
  [1] doc3.txt: Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths.

Query: "algorithm"
  [1] doc1.txt: Machine learning is a subset of artificial intelligence. Algorithms learn from data.
  [2] doc3.txt: Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths.

Query: "quantum"
  [2] doc2.txt: Quantum computing uses quantum mechanics to process information faster than classical computers.

Look at the "computer" query. Both doc2.txt and doc3.txt have a score of [1]. They look equally relevant. But "computer" is a more specific, defining topic in doc3.txt (computer science) than in doc2.txt (which is really about quantum mechanics). The search cannot tell the difference.

Why is this a problem?

Raw term frequency treats every word as equally important. The word "computer" carries the same weight whether it appears in a document about computers or in a document that just mentions "computers" in passing.

Worse, the results are not even sorted by score. They come out in arbitrary Map iteration order. A document scoring [2] can appear after one scoring [1]. This is the gap that proper relevance scoring needs to close.

Seeing "Cannot find module" or no output?

  • Check that src/scorer.ts exists and its filename matches the import in index.ts exactly: ./scorer.ts.
  • Confirm you removed the old inverted-index printing code. If both blocks exist, you may see duplicate output or errors.
  • Make sure the function is exported: export function searchWithTfIdf. A missing export keyword will cause an import error.

Help me debug my search function

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/scorer.ts and src/index.ts.

ⓧ I'd like to double check the full code

Here is the full src/scorer.ts:

import type { InvertedIndex } from "./index-builder.ts";

export interface SearchResult {
  docId: number;
  name: string;
  score: number;
  snippet: string;
}

export function searchWithTfIdf(
  queryTerms: string[],
  index: InvertedIndex
): SearchResult[] {
  const scores = new Map<number, number>();

  for (const term of queryTerms) {
    const postingList = index.postings.get(term);
    if (!postingList) continue;

    for (const posting of postingList) {
      scores.set(posting.docId, (scores.get(posting.docId) || 0) + posting.termFrequency);
    }
  }

  const results: SearchResult[] = [];
  for (const [docId, score] of scores) {
    const doc = index.documents[docId];
    const snippet = doc.content.substring(0, 100).trim();
    results.push({ docId, name: doc.name, score, snippet });
  }

  return results;
}

Here is the full src/index.ts:

import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";
import { searchWithTfIdf } from "./scorer.ts";

const sampleDocs = [
  { name: "doc1.txt", content: "Machine learning is a subset of artificial intelligence. Algorithms learn from data." },
  { name: "doc2.txt", content: "Quantum computing uses quantum mechanics to process information faster than classical computers." },
  { name: "doc3.txt", content: "Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths." }
];

const index = buildIndex(sampleDocs);

const queries = ["computer", "algorithm", "quantum"];
for (const query of queries) {
  const queryTerms = tokenize(query);
  const results = searchWithTfIdf(queryTerms, index);
  console.log(`\nQuery: "${query}"`);
  for (const r of results) {
    console.log(`  [${r.score}] ${r.name}: ${r.snippet}`);
  }
}

You now have a working search function. It finds the right documents. But it treats all matches as roughly equal and doesn't sort results by relevance. Next up, you'll fix this with TF-IDF scoring, the algorithm that makes rare, specific terms score higher than common ones.

Add TF-IDF Ranking

In Step 3, you searched the index and got results. But you also noticed the problem: every match looks the same.

A document mentioning "quantum" twice should rank higher than one casually mentioning "data". Right now, raw term counts cannot distinguish them. TF-IDF fixes this by weighting both how often a term appears in a document AND how rare that term is across the entire corpus.

In this step, get ready to:
  • Implement term frequency (TF) to normalize raw counts by document length.
  • Implement inverse document frequency (IDF) to boost rare terms and penalize common ones.
  • Combine TF-IDF into a final relevance score and sort results.
Implement term frequency (TF)

Term frequency measures how important a word is within a single document.

The formula is simple: divide the number of times a term appears by the total number of terms in that document.

Why divide by total terms?

A word appearing 3 times in a 10-word document is far more significant than the same word appearing 3 times in a 100-word document. Dividing by document length normalizes the count so short and long documents are compared fairly.

In code, this becomes: tf = termFrequency / totalTerms.

Implement inverse document frequency (IDF)

IDF measures how rare or common a term is across the entire corpus.

The formula takes the logarithm of total documents divided by the number of documents containing the term.

Why does rarity matter?

A term like "quantum" that appears in only 1 of 3 documents is highly specific. It tells you something meaningful about that document. A term like "data" appearing in 2 of 3 documents is much less distinctive.

The logarithm scales this down to a usable range. In code: idf = Math.log(totalDocs / docsContainingTerm). Rare terms get a high IDF. Common terms get a low IDF (approaching zero).

Combine TF-IDF scores and sort results

Now you have both pieces. TF tells you how relevant a term is within one document. IDF tells you how rare that term is overall. Multiplying them gives the final score: frequent in THIS document AND rare across ALL documents.

  • In src/scorer.ts, replace the entire searchWithTfIdf function with this updated version:
export function searchWithTfIdf(
  queryTerms: string[],
  index: InvertedIndex
): SearchResult[] {
  const totalDocs = index.documents.length;
  const scores = new Map<number, number>();

  for (const term of queryTerms) {
    const postingList = index.postings.get(term);
    if (!postingList) continue;

    // IDF: rare terms score higher
    const idf = Math.log(totalDocs / postingList.length);

    for (const posting of postingList) {
      const doc = index.documents[posting.docId];
      // TF: normalize by document length
      const tf = posting.termFrequency / doc.totalTerms;
      const tfidf = tf * idf;

      scores.set(posting.docId, (scores.get(posting.docId) || 0) + tfidf);
    }
  }

  const results: SearchResult[] = [];
  for (const [docId, score] of scores) {
    const doc = index.documents[docId];
    const snippet = doc.content.substring(0, 100).trim();
    results.push({ docId, name: doc.name, score, snippet });
  }

  return results.sort((a, b) => b.score - a.score);
}

What does this code do?

  • const totalDocs = index.documents.length stores the corpus size, which IDF needs to measure term rarity.
  • const idf = Math.log(totalDocs / postingList.length) computes how rare the term is. A term in 1 of 3 docs gets log(3/1) = 1.09. A term in all 3 gets log(3/3) = 0.
  • const tf = posting.termFrequency / doc.totalTerms normalizes term count by document length.
  • const tfidf = tf * idf produces the final score for one term in one document.
  • results.sort((a, b) => b.score - a.score) ensures the most relevant document appears first.

Since scores are now decimal values, update the output formatting in src/index.ts to display them cleanly.

  • In src/index.ts, find this line inside the for loop:
    console.log(`  [${r.score}] ${r.name}: ${r.snippet}`);
  • Replace r.score with r.score.toFixed(4) so the line looks like this:
    console.log(`  [${r.score.toFixed(4)}] ${r.name}: ${r.snippet}`);
  • Save both src/scorer.ts and src/index.ts.

Before you run this, think about the "quantum" query. In Step 3, results came back in arbitrary order. With TF-IDF, do you expect "quantum" to score higher or lower than "algorithm"? Why?

  • Run the search engine by executing:
npx tsx src/index.ts

You should see output like this, with decimal scores and results sorted by relevance:

Query: "computer"
  [0.1221] doc3.txt: Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths.

Query: "algorithm"
  [0.1221] doc3.txt: Graph algorithms are fundamental in computer science. Dijkstra algorithm finds shortest paths.

Query: "quantum"
  [0.2197] doc2.txt: Quantum computing uses quantum mechanics to process information faster than classical luftco...

Notice that "quantum" scores highest (0.2197). It appears twice in a short document AND in only 1 of 3 documents. That double boost from TF and IDF is exactly why TF-IDF works.

Why did the ranking change?

In Step 3, raw term frequency gave you just a count. "quantum" appearing twice scored 2, but so would any term appearing twice in any document.

TF-IDF now captures two dimensions: frequency relative to document size (TF) and rarity across the corpus (IDF). A term that is both frequent in its document and rare overall gets the highest score.

Seeing NaN or Infinity in your scores?

  • Check that you used index.documents.length (not index.postings.size) for totalDocs. The postings map counts unique terms, not documents.
  • Confirm doc.totalTerms is not zero. If your tokenizer filters out all words in a document, dividing by zero produces Infinity.
  • Make sure the division in Math.log(totalDocs / postingList.length) uses the posting list length for the current term, not the total number of postings.

Why are my TF-IDF scores showing NaN or Infinity?

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/scorer.ts and src/index.ts.

ⓧ I'd like to double check the full code

Here is the complete src/scorer.ts file:

import type { InvertedIndex } from "./index-builder.ts";

export interface SearchResult {
  docId: number;
  name: string;
  score: number;
  snippet: string;
}

export function searchWithTfIdf(
  queryTerms: string[],
  index: InvertedIndex
): SearchResult[] {
  const totalDocs = index.documents.length;
  const scores = new Map<number, number>();
  for (const term of queryTerms) {
    const postingList = index.postings.get(term);
    if (!postingList) continue;

    const idf = Math.log(totalDocs / postingList.length);

    for (const posting of postingList) {
      const doc = index.documents[posting.docId];
      const tf = posting.termFrequency / doc.totalTerms;
      const tfidf = tf * idf;

      scores.set(posting.docId, (scores.get(posting.docId) || 0) + tfidf);
    }
  }
  const results: SearchResult[] = [];
  for (const [docId, score] of scores) {
    const doc = index.documents[docId];
    const snippet = doc.content.substring(0, 100).trim();
    results.push({ docId, name: doc.name, score, snippet });
  }

  return results.sort((a, b) => b.score - a.score);
}

And here is the updated line in src/index.ts (only this line changed):

    console.log(`  [${r.score.toFixed(4)}] ${r.name}: ${r.snippet}`);

Your search engine now ranks results by relevance. Documents with rare, specific terms rise to the top. Next up, you will swap out the hardcoded sample strings for real files from the filesystem and build a proper CLI interface.

Load Real Documents and Build the CLI

Your search engine now ranks documents by relevance using TF-IDF. But it still operates on hardcoded strings baked into your source code.

A real search engine reads from the filesystem and accepts queries from the user. In this step, you'll wire up file loading and build a CLI that makes your engine usable on any folder of text documents.

In this step, get ready to:
  • Load .txt files from any directory on your filesystem.
  • Accept a directory path and search query as command-line arguments.
  • Display ranked results with relevance scores and content previews.
Create a test corpus of documents

Your search engine needs real text files to index. You'll create a corpus/ directory with several documents covering different topics so TF-IDF scoring produces meaningful, differentiated results.

  • Create the corpus directory and five sample documents by running these commands in your terminal:
mkdir corpus

cat > corpus/algorithms.txt << 'EOF'
Graph algorithms are fundamental to computer science. Dijkstra's algorithm finds the shortest path in weighted graphs. Trees and hash tables provide efficient data structures for organizing information. Sorting algorithms like quicksort and mergesort divide problems into smaller subproblems.
EOF

cat > corpus/machine-learning.txt << 'EOF'
Machine learning algorithms learn patterns from data without explicit programming. Neural networks are inspired by biological neurons in the brain. Deep learning has revolutionized computer vision and natural language processing. Training models requires large datasets and significant compute.
EOF

cat > corpus/quantum-computing.txt << 'EOF'
Quantum computing harnesses quantum mechanics to solve problems classical computers cannot. Qubits can exist in superposition, representing both zero and one simultaneously. Quantum entanglement enables faster communication between particles. Error correction remains a major challenge.
EOF

cat > corpus/web-development.txt << 'EOF'
Modern web development relies on frameworks like React and Vue for building user interfaces. JavaScript runs in the browser and on the server with Node.js. APIs connect frontend applications to backend databases and services. Performance optimization requires understanding of network latency.
EOF

cat > corpus/databases.txt << 'EOF'
Relational databases store data in tables with rows and columns. SQL provides a standard language for querying structured data. NoSQL databases offer flexible schemas for unstructured data like documents and graphs. Indexing speeds up query performance significantly.
EOF

What are these files for?

Each file covers a distinct topic (algorithms, ML, quantum, web dev, databases). When you search for "quantum", the engine should rank quantum-computing.txt highest because TF-IDF scores rare, topic-specific terms more heavily than common ones.

Windows users

The heredoc syntax above works in macOS/Linux terminals. On Windows, create the corpus folder manually in VS Code's Explorer sidebar. Then create each .txt file inside it, pasting the text content between the EOF markers.

  • Confirm the files were created by running:
ls corpus

You should see five files listed: algorithms.txt, databases.txt, machine-learning.txt, quantum-computing.txt, and web-development.txt.

Don't see all five files?

Make sure each cat command ran successfully. If one failed, re-run just that command. Check that the EOF line at the end has no leading spaces.

Help me create my corpus text files.

Add filesystem reading and CLI argument parsing

The current src/index.ts uses hardcoded document arrays. You'll replace the entire file with code that reads real files from disk and accepts command-line arguments.

  • In VS Code, open src/index.ts.
  • Select all the existing code (Cmd+A on macOS, Ctrl+A on Windows) and delete it.
  • Paste the following code, which adds filesystem imports, a document-loading function, and argument validation:
import { readFileSync, readdirSync } from "node:fs";
import { join, extname } from "node:path";
import { tokenize } from "./tokenizer.ts";
import { buildIndex } from "./index-builder.ts";
import { searchWithTfIdf } from "./scorer.ts";

// Read all .txt files from a directory into indexable documents
function loadDocuments(dirPath: string): { name: string; content: string }[] {
  const files = readdirSync(dirPath).filter(f => extname(f) === ".txt");
  return files.map(file => ({
    name: file,
    content: readFileSync(join(dirPath, file), "utf-8"),
  }));
}

function main() {
  const args = process.argv.slice(2);

  // Require both a directory and a query
  if (args.length < 2) {
    console.log("Usage: npx tsx src/index.ts <directory> <query>");
    console.log("Example: npx tsx src/index.ts ./corpus \"machine learning\"");
    process.exit(1);
  }
}

main();

What does this code do?

The loadDocuments function uses Node.js built-in readdirSync to list files, filters for .txt extensions, then reads each one into memory. It returns the same { name, content } shape that buildIndex already expects.

The process.argv.slice(2) grabs command-line arguments after the script name. If the user doesn't provide both a directory and a query, the script prints usage instructions and exits.

  • Save src/index.ts.
  • Test the argument validation by running the script without any arguments:
npx tsx src/index.ts

You should see the usage message printed: Usage: npx tsx src/index.ts <directory> <query>.

Seeing an error instead of the usage message?

Check that you deleted ALL the old code before pasting. Leftover lines from the previous version (like the old sampleDocs array) will cause syntax or import conflicts.

Help me fix my index.ts rewrite.

Wire up the full search pipeline with formatted output

The argument validation works. Now you'll expand main() to actually load documents, build the index, run the search, and display ranked results with scores and content snippets.

  • In src/index.ts, find the closing } of the main() function and the main(); call at the bottom.
  • Replace both of those lines with the following code, which completes the search pipeline:
  // Parse directory path and join remaining args as query
  const dirPath = args[0];
  const query = args.slice(1).join(" ");

  // Load and index documents from the filesystem
  console.log(`Indexing documents from: ${dirPath}`);
  const docs = loadDocuments(dirPath);
  console.log(`Indexed ${docs.length} documents.\n`);

  const index = buildIndex(docs);
  console.log(`Index contains ${index.postings.size} unique terms.\n`);

  // Tokenize query and search the index
  const queryTerms = tokenize(query);
  console.log(`Searching for: "${query}" (terms: ${queryTerms.join(", ")})\n`);

  const results = searchWithTfIdf(queryTerms, index);

  if (results.length === 0) {
    console.log("No matching documents found.");
    return;
  }

  // Display ranked results with scores and snippets
  console.log("Results:");
  console.log("\u2500".repeat(60));
  for (let i = 0; i < Math.min(results.length, 10); i++) {
    const r = results[i];
    console.log(`${i + 1}. [${r.score.toFixed(4)}] ${r.name}`);
    console.log(`   ${r.snippet}...`);
    console.log();
  }
}

main();

What does this code do?

  • It reads args[0] as the directory path and joins remaining arguments as a multi-word query.
  • It calls loadDocuments to read real files, then buildIndex and searchWithTfIdf from the modules you already built.
  • Results are capped at 10 and displayed with their TF-IDF score and a 100-character content snippet.
  • Save src/index.ts.

Before you run this, predict: if you search for "quantum", which document do you think will rank highest and why?

  • Run your search engine on the corpus with a test query:
npx tsx src/index.ts ./corpus "quantum"

You should see quantum-computing.txt ranked first with the highest score. The term "quantum" appears multiple times in that document and rarely elsewhere, so TF-IDF pushes it to the top.

  • Try a multi-word query to see how scores combine across terms:
npx tsx src/index.ts ./corpus "machine learning"

The results now show machine-learning.txt ranked highest because both query terms concentrate in that document.

Seeing "No matching documents found"?

Check that your corpus/ directory is in the same folder where you're running the command. Run ls corpus first to confirm the files are accessible from your current working directory.

Also verify that your query terms aren't all in the stopwords list. Single common words like "the" or "is" get filtered out by the tokenizer.

Help me debug my search engine CLI.

✔️ Awesome, I've got everything!

Great. Confirm you've saved src/index.ts and that searches return ranked results.

ⓧ I'd like to double check the full code

Your src/index.ts should contain these sections in order:

  • Imports from node:fs, node:path, and your three local modules (tokenizer, index-builder, scorer).
  • The loadDocuments function that reads .txt files from a directory.
  • The main() function with argument validation (requiring 2+ args), document loading, index building, query tokenization, TF-IDF search, and result display capped at 10.
  • A main(); call at the very end of the file.

Scroll through your file and verify each section is present. The other files (tokenizer.ts, index-builder.ts, scorer.ts) are unchanged from the previous steps.

Your search engine is complete. It reads real files, indexes them, and returns ranked results using the same algorithm that powered early web search. You've gone from hardcoded test strings to a working CLI tool that can index any folder of documents.

Secret mission

Boolean Query Support

Your search engine treats every query as a bag of words. Real search engines let you say "machine AND learning" to find only documents containing both terms, or "python OR javascript" to find documents about either. Can you implement posting list intersection and union using the two-pointer algorithm?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your project files for future reference or delete them.

Resources you used:

  • The search-engine project folder on your Desktop (contains source code, node_modules, and your corpus/ documents).

Keep everything running

No action needed. Choose this if you want to keep experimenting with your search engine, add more documents to your corpus, or extend the project further.

  • Your project folder stays on your Desktop ready to use anytime.
  • Run searches anytime with npx tsx src/index.ts ./corpus "your query".

Pause - I'll come back to this later

Since this is a local project with no running processes, there is nothing to pause. Your files sit on disk without consuming memory or CPU.

  • Close the project in VS Code to free up editor memory.
  • Your project folder remains on your Desktop. Open it again in VS Code whenever you want to resume.

Delete - I don't want to use this again

Remove all project files from your machine. This cannot be undone.

  • Close the project in VS Code.
  • Delete the project folder by running this command in your terminal:

macOS

rm -rf ~/Desktop/search-engine

Windows

rmdir /s /q %USERPROFILE%\Desktop\search-engine
  • Confirm the folder is gone by checking your Desktop.

That's a Wrap!

That's a Wrap!

Nice work. You just built a search engine from scratch, implementing the same data structure and ranking algorithm that powered early Google.

You've learned how to:

  • Build an inverted index from scratch that maps terms to document locations in O(1) lookup time, turning brute-force scanning into instant retrieval.
  • Implement TF-IDF scoring that ranks documents by relevance, weighting rare terms higher than common ones so results reflect what actually matters.
  • Create a working CLI search tool that indexes real .txt files from the filesystem and returns ranked results with relevance scores.
  • Secret Mission: Added boolean AND/OR query support using posting list intersection and union algorithms with the two-pointer merge technique.

Ready to quiz yourself?