Build a Reddit Job Scraper

Scrape Reddit hiring subreddits with OAuth2, TypeScript, and Next.js.

Introduction

30 Second Summary

Job hunting on Reddit means scrolling through the same listings cross-posted to five different subreddits. You end up wasting time re-reading posts you have already seen.

In this project, you will build a personal job-scraping dashboard that authenticates with Reddit's API, fetches listings from hiring subreddits, filters them by your tech stack keywords, and removes duplicates using a three-layer deduplication system.

What You'll Build

A Next.js web app that connects to Reddit's OAuth2 API, scrapes job posts from multiple subreddits, and displays only fresh, relevant results in a clean feed.

By the end of this project, you'll have:

  • A Next.js App Router project with a server-side API route that handles Reddit OAuth2 authentication using native fetch.
  • A three-layer deduplication engine that eliminates cross-posted and copy-pasted job listings using post IDs, crosspost metadata, and author+content hashing.
  • A feed UI with keyword filtering and a persistent JSON store that tracks which posts you have already seen between scrapes.
  • Secret Mission: Add keyword highlighting in results and a "mark as applied" tracking feature for a complete job-hunting workflow.

Are there any prerequisites?

You should be familiar with TypeScript and React basics. You also need a Reddit account to register an API application during setup.

Set Up the Project

You have your Reddit API credentials ready. Now you need a project to house the scraper itself.

In this step, you'll scaffold a Next.js app with TypeScript and connect it to Reddit through environment variables.

In this step, get ready to:
  • Scaffold a Next.js project with TypeScript and App Router.
  • Create environment variables for your Reddit API credentials.
  • Verify the dev server starts successfully.
Scaffold the Next.js project

Next.js uses an official scaffolding tool called create-next-app that generates a ready-to-run project with TypeScript, a router, and styling preconfigured.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Scaffold the Next.js project by running this command:
npx create-next-app@latest reddit-job-scraper --yes

What does this command do?

  • npx create-next-app@latest runs the latest version of the official Next.js scaffolding tool.
  • reddit-job-scraper is the name of the folder it creates for your project.
  • --yes accepts all defaults automatically: TypeScript, App Router, Tailwind CSS, ESLint, and Turbopack.

npx asks to install a package

If this is your first time running npx create-next-app, npm may prompt you with "Need to install the following packages. Ok to proceed?" Type y and press Enter to continue.

You should see the project creation complete with a success message. A new reddit-job-scraper folder now exists on your Desktop.

Seeing a "command not found" error?

Make sure Node.js is installed by running node --version in your terminal. You need version 20.9.0 or higher for Next.js.

If npx is not recognized, try closing and reopening your terminal so your system picks up the Node.js installation path.

Help me fix this npx error

  • Move into the new project folder by running:
cd reddit-job-scraper
Create your environment variables

Next.js reads environment variables from a file called .env.local at the project root. Your Reddit credentials go here so the app can authenticate without hardcoding secrets into source code.

  • Open the reddit-job-scraper project in your code editor.
  • Create a new file in the project root called .env.local and add the following content:
REDDIT_CLIENT_ID=your-client-id-here
REDDIT_CLIENT_SECRET=your-client-secret-here
REDDIT_USERNAME=your-reddit-username-here
REDDIT_PASSWORD=your-reddit-password-here

What do these variables do?

  • REDDIT_CLIENT_ID is the ID shown under your app name on the Reddit apps page.
  • REDDIT_CLIENT_SECRET is the secret generated when you created the app.
  • REDDIT_USERNAME and REDDIT_PASSWORD are your Reddit login credentials. The OAuth2 password grant uses these to authenticate on your behalf.
  • Replace each placeholder with your actual values. For example, replace your-client-id-here with the client ID you noted from the Reddit app registration.
  • Save .env.local.

Keep your secrets safe

The .env.local file is already included in the default .gitignore that create-next-app generated. Your credentials will never be committed to version control.

  • Verify the dev server starts by running this command in your terminal:
npm run dev

You should see output showing the local server URL at http://localhost:3000. This confirms your project is set up correctly.

Dev server not starting?

If you see a "module not found" or dependency error, run npm install first to make sure all packages are installed.

If port 3000 is already in use, the terminal will show an error. Close any other servers running on that port, then try npm run dev again.

Help me fix my dev server

✔️ Awesome, I've got everything!

Great. Double check you've saved your .env.local file with your real credentials filled in.

ⓧ I'd like to double check the full code

Here's what your .env.local file should look like (with your own values replacing the placeholders):

REDDIT_CLIENT_ID=your-client-id-here
REDDIT_CLIENT_SECRET=your-client-secret-here
REDDIT_USERNAME=your-reddit-username-here
REDDIT_PASSWORD=your-reddit-password-here

Your project is scaffolded and connected to Reddit. Next up, you'll build the API client that uses these credentials to authenticate and fetch job posts.

Build the Reddit API Client

Your environment is set up with credentials ready to go. Now it's time to connect your app to Reddit's data.

Reddit requires OAuth2 authentication before you can access its API. In this step, you'll build a typed client that handles the entire auth flow and fetches job posts from multiple subreddits.

In this step, get ready to:
  • Build a RedditClient class that authenticates with Reddit's OAuth2 API.
  • Add a method to fetch new posts from any subreddit.
  • Create an API route that pulls posts from four hiring subreddits.
Create the Reddit API client with OAuth2 authentication

Reddit's API uses a password grant flow for script-type apps. Your client will send your credentials to Reddit's token endpoint, receive a short-lived access token, and use that token to make authenticated requests.

  • In your code editor, create a new folder called lib inside the app folder of your reddit-job-scraper project.
  • Inside app/lib/, create a new file called reddit-client.ts.
  • Define the TypeScript interfaces for the Reddit API response by pasting this code into the file:
// Shape of the token response from Reddit's OAuth2 endpoint
interface RedditTokenResponse {
  access_token: string;
  token_type: string;
  expires_in: number;
  scope: string;
}

// Shape of a single Reddit post with fields we need
export interface RedditPost {
  id: string;
  title: string;
  selftext: string;
  author: string;
  permalink: string;
  created_utc: number;
  subreddit: string;
  crosspost_parent_list?: RedditPost[];
}

// Shape of Reddit's listing response (wraps an array of posts)
interface RedditListing {
  kind: string;
  data: {
    children: Array<{
      kind: string;
      data: RedditPost;
    }>;
  };
}

What do these interfaces do?

TypeScript interfaces define the expected shape of data your code will work with. RedditTokenResponse describes what Reddit sends back when you authenticate. RedditPost describes a single post's fields. RedditListing describes how Reddit wraps posts in a listing envelope.

The crosspost_parent_list field is optional because only cross-posted content includes it. You'll use this field later for deduplication.

  • Below the interfaces, add the RedditClient class with its constructor and authentication method:
export class RedditClient {
  private accessToken: string | null = null;
  private tokenExpiry: number = 0;
  private clientId: string;
  private clientSecret: string;
  private username: string;
  private password: string;
  private userAgent: string;

  constructor() {
    // Pull credentials from environment variables
    this.clientId = process.env.REDDIT_CLIENT_ID!;
    this.clientSecret = process.env.REDDIT_CLIENT_SECRET!;
    this.username = process.env.REDDIT_USERNAME!;
    this.password = process.env.REDDIT_PASSWORD!;
    // Reddit requires a descriptive User-Agent to avoid throttling
    this.userAgent = "web:reddit-job-scraper:v1.0.0 (by /u/" + this.username + ")";
  }

  private async authenticate(): Promise<void> {
    // Skip auth if we already have a valid token
    if (this.accessToken && Date.now() < this.tokenExpiry) {
      return;
    }

    // Encode client_id:client_secret as Base64 for Basic auth
    const auth = Buffer.from(`${this.clientId}:${this.clientSecret}`).toString("base64");

    // Build the form body for password grant
    const params = new URLSearchParams();
    params.append("grant_type", "password");
    params.append("username", this.username);
    params.append("password", this.password);

    const response = await fetch("https://www.reddit.com/api/v1/access_token", {
      method: "POST",
      headers: {
        Authorization: `Basic ${auth}`,
        "User-Agent": this.userAgent,
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: params,
    });

    if (!response.ok) {
      throw new Error(`Reddit auth failed: ${response.status} ${response.statusText}`);
    }

    const data: RedditTokenResponse = await response.json();
    this.accessToken = data.access_token;
    // Expire 1 minute early to avoid edge-case failures
    this.tokenExpiry = Date.now() + data.expires_in * 1000 - 60000;
  }
}

How does the OAuth2 password grant work here?

The authenticate() method sends a POST to Reddit's token endpoint with your client ID and secret as Basic auth, plus your Reddit username and password in the body. Reddit responds with an access token that lasts 3600 seconds (1 hour).

The class caches this token and only re-authenticates when it expires. Subtracting 60000 milliseconds from the expiry gives a 1-minute safety buffer so requests never hit Reddit with a just-expired token.

Why use a descriptive User-Agent?

Reddit's API requires a unique, descriptive User-Agent string in the format <platform>:<app ID>:<version> (by /u/<username>). Generic or missing User-Agents get rate-limited or blocked entirely.

  • Save reddit-client.ts.
  • Confirm the file appears in the app/lib/ folder in your editor's file explorer sidebar.

Don't see the lib folder?

Make sure you created the lib folder inside app/ (not at the project root). The path should be reddit-job-scraper/app/lib/reddit-client.ts.

If your editor shows a red underline on process.env.REDDIT_CLIENT_ID!, that's normal. TypeScript doesn't know about your .env.local file at compile time, but Next.js loads it at runtime.

help me fix a file path issue with my reddit-client.ts

Add the method to fetch subreddit posts

The client can authenticate, but it can't fetch any data yet. You need a method that takes a subreddit name and returns the newest posts from it.

  • In reddit-client.ts, find the closing } brace of the RedditClient class (the last line of the file).
  • Add the following method just above that closing brace:
  async fetchSubredditPosts(subreddit: string, limit: number = 25): Promise<RedditPost[]> {
    // Ensure we have a valid token before making the request
    await this.authenticate();

    const response = await fetch(
      `https://oauth.reddit.com/r/${subreddit}/new?limit=${limit}`,
      {
        headers: {
          Authorization: `Bearer ${this.accessToken}`,
          "User-Agent": this.userAgent,
        },
      }
    );

    if (!response.ok) {
      throw new Error(`Failed to fetch r/${subreddit}: ${response.status}`);
    }

    // Unwrap the listing envelope to return just the post data
    const listing: RedditListing = await response.json();
    return listing.data.children.map((child) => child.data);
  }

What does this method do?

The fetchSubredditPosts method calls authenticate() first to ensure a valid token exists. Then it hits Reddit's authenticated data endpoint at https://oauth.reddit.com to fetch the 25 newest posts from the given subreddit.

Reddit returns posts wrapped in a listing structure. The .map((child) => child.data) call unwraps each post from its envelope so you get clean RedditPost objects.

  • Save reddit-client.ts.

You won't see a visible change yet. The next substep creates a route that uses this client so you can test it end-to-end.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete app/lib/reddit-client.ts file:

interface RedditTokenResponse {
  access_token: string;
  token_type: string;
  expires_in: number;
  scope: string;
}

export interface RedditPost {
  id: string;
  title: string;
  selftext: string;
  author: string;
  permalink: string;
  created_utc: number;
  subreddit: string;
  crosspost_parent_list?: RedditPost[];
}

interface RedditListing {
  kind: string;
  data: {
    children: Array<{
      kind: string;
      data: RedditPost;
    }>;
  };
}

export class RedditClient {
  private accessToken: string | null = null;
  private tokenExpiry: number = 0;
  private clientId: string;
  private clientSecret: string;
  private username: string;
  private password: string;
  private userAgent: string;

  constructor() {
    this.clientId = process.env.REDDIT_CLIENT_ID!;
    this.clientSecret = process.env.REDDIT_CLIENT_SECRET!;
    this.username = process.env.REDDIT_USERNAME!;
    this.password = process.env.REDDIT_PASSWORD!;
    this.userAgent = "web:reddit-job-scraper:v1.0.0 (by /u/" + this.username + ")";
  }

  private async authenticate(): Promise<void> {
    if (this.accessToken && Date.now() < this.tokenExpiry) {
      return;
    }

    const auth = Buffer.from(`${this.clientId}:${this.clientSecret}`).toString("base64");

    const params = new URLSearchParams();
    params.append("grant_type", "password");
    params.append("username", this.username);
    params.append("password", this.password);

    const response = await fetch("https://www.reddit.com/api/v1/access_token", {
      method: "POST",
      headers: {
        Authorization: `Basic ${auth}`,
        "User-Agent": this.userAgent,
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: params,
    });

    if (!response.ok) {
      throw new Error(`Reddit auth failed: ${response.status} ${response.statusText}`);
    }

    const data: RedditTokenResponse = await response.json();
    this.accessToken = data.access_token;
    this.tokenExpiry = Date.now() + data.expires_in * 1000 - 60000;
  }

  async fetchSubredditPosts(subreddit: string, limit: number = 25): Promise<RedditPost[]> {
    await this.authenticate();

    const response = await fetch(
      `https://oauth.reddit.com/r/${subreddit}/new?limit=${limit}`,
      {
        headers: {
          Authorization: `Bearer ${this.accessToken}`,
          "User-Agent": this.userAgent,
        },
      }
    );

    if (!response.ok) {
      throw new Error(`Failed to fetch r/${subreddit}: ${response.status}`);
    }

    const listing: RedditListing = await response.json();
    return listing.data.children.map((child) => child.data);
  }
}
Create the API route handler

Your client module is ready, but you need a way to trigger it. A Next.js Route Handler gives you a server-side endpoint that your frontend can call later.

  • Inside the app folder, create a new folder called api.
  • Inside api, create another folder called scrape.
  • Inside app/api/scrape/, create a new file called route.ts.
  • Paste the following code into route.ts:
import { NextResponse } from "next/server";
import { RedditClient } from "@/app/lib/reddit-client";

// The four hiring subreddits to scrape
const TARGET_SUBREDDITS = ["forhire", "hiring", "remotejobs", "SoftwareEngineerJobs"];

export async function POST() {
  try {
    const client = new RedditClient();
    const allPosts = [];

    // Fetch 25 newest posts from each subreddit
    for (const subreddit of TARGET_SUBREDDITS) {
      const posts = await client.fetchSubredditPosts(subreddit);
      allPosts.push(...posts);
    }

    return NextResponse.json({ posts: allPosts });
  } catch (error) {
    const message = error instanceof Error ? error.message : "Unknown error";
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

What does this route do?

This file exports a POST function, which Next.js automatically turns into a POST endpoint at /api/scrape. When called, it creates a fresh RedditClient, loops through all four target subreddits, and combines the results into one array.

The try/catch block ensures that if Reddit's API returns an error (bad credentials, rate limit), your route responds with a helpful error message instead of crashing.

  • Save route.ts.
  • Start your dev server by running this command in your terminal:
npm run dev
  • Open your browser and navigate to http://localhost:3000.
  • Open the browser's Developer Console by pressing Ctrl+Shift+J (Windows) or Cmd+Option+J (macOS).
  • Test your route by pasting this into the console and pressing Enter:
fetch('/api/scrape', { method: 'POST' }).then(r => r.json()).then(console.log)

After a few seconds, you should see a JSON object logged in the console containing a posts array filled with Reddit post objects. Each post has fields like title, author, selftext, and subreddit.

Seeing an error in the console?

  • If you see Reddit auth failed: 401, double check that REDDIT_CLIENT_ID and REDDIT_CLIENT_SECRET in your .env.local file match what Reddit shows on your app's page at https://www.reddit.com/prefs/apps.
  • If you see Reddit auth failed: 403, your Reddit username or password in .env.local may be incorrect. Re-enter them and restart the dev server.
  • If you see a 404 or network error, make sure npm run dev is still running in your terminal and you're on http://localhost:3000.
  • After changing .env.local, you must restart the dev server (stop with Ctrl+C, then run npm run dev again) for changes to take effect.

help me debug my Reddit API authentication error

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

Here is the complete app/api/scrape/route.ts file:

import { NextResponse } from "next/server";
import { RedditClient } from "@/app/lib/reddit-client";

const TARGET_SUBREDDITS = ["forhire", "hiring", "remotejobs", "SoftwareEngineerJobs"];

export async function POST() {
  try {
    const client = new RedditClient();
    const allPosts = [];

    for (const subreddit of TARGET_SUBREDDITS) {
      const posts = await client.fetchSubredditPosts(subreddit);
      allPosts.push(...posts);
    }

    return NextResponse.json({ posts: allPosts });
  } catch (error) {
    const message = error instanceof Error ? error.message : "Unknown error";
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

Your API client is live and pulling real job posts from Reddit. Next up, you'll add keyword filtering and a deduplication engine to surface only relevant, unique listings.

Implement Filtering and Deduplication

Your Reddit API client can now pull 100 posts across four subreddits in a single scrape. The problem is that most of those posts are irrelevant to your tech stack, and many are duplicates cross-posted across multiple subreddits.

In this step, you will build a keyword filter to surface only relevant posts and a three-layer deduplication engine to eliminate repeats. You will also add a JSON-based store so the app remembers what it has already seen between scrapes.

In this step, get ready to:
  • Filter posts by your tech stack keywords.
  • Deduplicate posts using post IDs, crosspost metadata, and content hashing.
  • Persist seen-post state to a local JSON file.
Create the keyword filter

Hiring subreddits contain posts for every tech stack imaginable. A keyword filter narrows results to only the jobs that match your skills.

  • Create a new file at app/lib/filter.ts inside your reddit-job-scraper project.
  • Add the following code to define a keyword filter function:
import { RedditPost } from "./reddit-client";

// Keywords matching your tech stack
const DEFAULT_KEYWORDS = [
  "nextjs", "next.js",
  "nestjs", "nest.js",
  "express",
  "node", "nodejs", "node.js",
  "shopify",
  "remote",
  "freelance",
];

export function filterByKeywords(
  posts: RedditPost[],
  keywords: string[] = DEFAULT_KEYWORDS
): RedditPost[] {
  // Check if any keyword appears in the title or body text
  return posts.filter((post) => {
    const searchText = `${post.title} ${post.selftext}`.toLowerCase();
    return keywords.some((keyword) => searchText.includes(keyword.toLowerCase()));
  });
}

What does this code do?

The DEFAULT_KEYWORDS array holds the tech terms you want to match against. You can customize this list to fit your own stack.

The filterByKeywords function combines each post's title and body into one string, lowercases it, and checks whether any keyword appears. Posts with zero keyword matches are dropped.

  • Save filter.ts.
  • You won't see a change yet. The next file (deduplication.ts) uses this filter alongside its own logic.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import { RedditPost } from "./reddit-client";

const DEFAULT_KEYWORDS = [
  "nextjs", "next.js",
  "nestjs", "nest.js",
  "express",
  "node", "nodejs", "node.js",
  "shopify",
  "remote",
  "freelance",
];

export function filterByKeywords(
  posts: RedditPost[],
  keywords: string[] = DEFAULT_KEYWORDS
): RedditPost[] {
  return posts.filter((post) => {
    const searchText = `${post.title} ${post.selftext}`.toLowerCase();
    return keywords.some((keyword) => searchText.includes(keyword.toLowerCase()));
  });
}
Build the three-layer deduplication engine

Job posters on Reddit frequently cross-post the same listing to multiple subreddits. A simple post-ID check catches some, but not all. You need three detection layers to cover every pattern.

Why three layers?

Layer 1 (post ID) catches exact reposts. Layer 2 (crosspost metadata) catches posts shared using Reddit's built-in crosspost feature. Layer 3 (content hash) catches manual copy-pastes where the same author posts identical text with a new post ID.

  • Create a new file at app/lib/deduplication.ts.
  • Add the state interface and helper functions by pasting this code:
import { createHash } from "crypto";
import { RedditPost } from "./reddit-client";

// Shape of the data we persist between scrapes
export interface DeduplicationState {
  seenPostIds: string[];
  seenCrosspostParentIds: string[];
  seenContentHashes: string[];
}

// Strip common tags and normalize whitespace for consistent hashing
function normalizeContent(text: string): string {
  return text
    .toLowerCase()
    .replace(/\[hiring\]/gi, "")
    .replace(/\[for hire\]/gi, "")
    .replace(/\[forhire\]/gi, "")
    .replace(/\s+/g, " ")
    .trim();
}

// Combine author + normalized content into a SHA-256 hash
function generateContentHash(post: RedditPost): string {
  const normalized = normalizeContent(`${post.title} ${post.selftext}`);
  const input = `${post.author}:${normalized}`;
  return createHash("sha256").update(input).digest("hex");
}

What does this code do?

The DeduplicationState interface defines three arrays that track everything the app has already seen: post IDs, crosspost parent IDs, and content hashes.

The normalizeContent function strips common tags like [Hiring] and collapses whitespace so that minor formatting differences do not produce different hashes.

The generateContentHash function uses Node's built-in crypto module to produce a SHA-256 hash from the author name plus the normalized post content. Including the author prevents different companies with similar job descriptions from being falsely deduplicated.

  • Now add the main deduplication function below the helpers in the same file:
export function deduplicatePosts(
  posts: RedditPost[],
  state: DeduplicationState
): { newPosts: RedditPost[]; updatedState: DeduplicationState } {
  const seenIds = new Set(state.seenPostIds);
  const seenParentIds = new Set(state.seenCrosspostParentIds);
  const seenHashes = new Set(state.seenContentHashes);

  const newPosts: RedditPost[] = [];

  for (const post of posts) {
    // Layer 1: Skip if we have already seen this post ID
    if (seenIds.has(post.id)) {
      continue;
    }

    // Layer 2: Check crosspost metadata for a known parent
    if (post.crosspost_parent_list && post.crosspost_parent_list.length > 0) {
      const parentId = post.crosspost_parent_list[0].id;
      if (seenIds.has(parentId) || seenParentIds.has(parentId)) {
        seenIds.add(post.id);
        continue;
      }
      seenParentIds.add(parentId);
    }

    // Layer 3: Compare content hash to catch manual copy-pastes
    const contentHash = generateContentHash(post);
    if (seenHashes.has(contentHash)) {
      seenIds.add(post.id);
      continue;
    }

    // Post passed all three layers — it is genuinely new
    seenIds.add(post.id);
    seenHashes.add(contentHash);
    newPosts.push(post);
  }

  return {
    newPosts,
    updatedState: {
      seenPostIds: Array.from(seenIds),
      seenCrosspostParentIds: Array.from(seenParentIds),
      seenContentHashes: Array.from(seenHashes),
    },
  };
}

How does deduplicatePosts work?

The function converts the persisted arrays into Sets for O(1) lookups, then runs every post through the three layers in sequence. If ANY layer flags a post as seen, it is skipped.

At the end, it returns both the genuinely new posts and an updated state object (converting Sets back to arrays for JSON serialization).

  • Save deduplication.ts.
  • You won't see a change yet. The store module you create next will handle persisting this state to disk.

Seeing a red underline on the crypto import?

The crypto module is built into Node.js. If your editor flags it, make sure you have @types/node installed (Next.js includes this by default). The error should disappear after the dev server restarts.

Help me fix a crypto import error in my deduplication file.

✔️ Awesome, I've got everything!

Great. Make sure the file is saved before continuing.

ⓧ I'd like to double check the full code

import { createHash } from "crypto";
import { RedditPost } from "./reddit-client";

export interface DeduplicationState {
  seenPostIds: string[];
  seenCrosspostParentIds: string[];
  seenContentHashes: string[];
}

function normalizeContent(text: string): string {
  return text
    .toLowerCase()
    .replace(/\[hiring\]/gi, "")
    .replace(/\[for hire\]/gi, "")
    .replace(/\[forhire\]/gi, "")
    .replace(/\s+/g, " ")
    .trim();
}

function generateContentHash(post: RedditPost): string {
  const normalized = normalizeContent(`${post.title} ${post.selftext}`);
  const input = `${post.author}:${normalized}`;
  return createHash("sha256").update(input).digest("hex");
}

export function deduplicatePosts(
  posts: RedditPost[],
  state: DeduplicationState
): { newPosts: RedditPost[]; updatedState: DeduplicationState } {
  const seenIds = new Set(state.seenPostIds);
  const seenParentIds = new Set(state.seenCrosspostParentIds);
  const seenHashes = new Set(state.seenContentHashes);

  const newPosts: RedditPost[] = [];

  for (const post of posts) {
    if (seenIds.has(post.id)) {
      continue;
    }

    if (post.crosspost_parent_list && post.crosspost_parent_list.length > 0) {
      const parentId = post.crosspost_parent_list[0].id;
      if (seenIds.has(parentId) || seenParentIds.has(parentId)) {
        seenIds.add(post.id);
        continue;
      }
      seenParentIds.add(parentId);
    }

    const contentHash = generateContentHash(post);
    if (seenHashes.has(contentHash)) {
      seenIds.add(post.id);
      continue;
    }

    seenIds.add(post.id);
    seenHashes.add(contentHash);
    newPosts.push(post);
  }

  return {
    newPosts,
    updatedState: {
      seenPostIds: Array.from(seenIds),
      seenCrosspostParentIds: Array.from(seenParentIds),
      seenContentHashes: Array.from(seenHashes),
    },
  };
}
Create the store and wire everything together

The deduplication engine needs to remember what it has seen between scrapes. A local JSON file is a lightweight solution for a personal tool running on your machine.

  • Create a new file at app/lib/store.ts and add the following code:
import { readFile, writeFile } from "fs/promises";
import { existsSync } from "fs";
import path from "path";
import { DeduplicationState } from "./deduplication";

// Store the seen-posts file in the project root
const STORE_PATH = path.join(process.cwd(), "seen-posts.json");

// Default empty state for a fresh project
const EMPTY_STATE: DeduplicationState = {
  seenPostIds: [],
  seenCrosspostParentIds: [],
  seenContentHashes: [],
};

export async function loadState(): Promise<DeduplicationState> {
  // If the file does not exist yet, return empty state
  if (!existsSync(STORE_PATH)) {
    return EMPTY_STATE;
  }

  try {
    const data = await readFile(STORE_PATH, "utf-8");
    return JSON.parse(data) as DeduplicationState;
  } catch {
    return EMPTY_STATE;
  }
}

export async function saveState(state: DeduplicationState): Promise<void> {
  await writeFile(STORE_PATH, JSON.stringify(state, null, 2), "utf-8");
}

What does this code do?

The loadState function checks if seen-posts.json exists. If it does, it reads and parses the file. If it does not (or parsing fails), it returns an empty state so the app still works on the very first run.

The saveState function writes the updated deduplication state back to disk as pretty-printed JSON.

  • Save store.ts.

✔️ Awesome, I've got everything!

Great. Save the file and keep going.

ⓧ I'd like to double check the full code

import { readFile, writeFile } from "fs/promises";
import { existsSync } from "fs";
import path from "path";
import { DeduplicationState } from "./deduplication";

const STORE_PATH = path.join(process.cwd(), "seen-posts.json");

const EMPTY_STATE: DeduplicationState = {
  seenPostIds: [],
  seenCrosspostParentIds: [],
  seenContentHashes: [],
};

export async function loadState(): Promise<DeduplicationState> {
  if (!existsSync(STORE_PATH)) {
    return EMPTY_STATE;
  }

  try {
    const data = await readFile(STORE_PATH, "utf-8");
    return JSON.parse(data) as DeduplicationState;
  } catch {
    return EMPTY_STATE;
  }
}

export async function saveState(state: DeduplicationState): Promise<void> {
  await writeFile(STORE_PATH, JSON.stringify(state, null, 2), "utf-8");
}

Now you need to wire all three modules into your existing API route so the scrape endpoint filters, deduplicates, and persists state in a single request.

  • Open app/api/scrape/route.ts and replace the entire file contents with the updated code below:
import { NextResponse } from "next/server";
import { RedditClient } from "@/app/lib/reddit-client";
import { filterByKeywords } from "@/app/lib/filter";
import { deduplicatePosts } from "@/app/lib/deduplication";
import { loadState, saveState } from "@/app/lib/store";

const TARGET_SUBREDDITS = ["forhire", "hiring", "remotejobs", "SoftwareEngineerJobs"];

export async function POST() {
  try {
    const client = new RedditClient();
    const allPosts = [];

    // Fetch posts from all target subreddits
    for (const subreddit of TARGET_SUBREDDITS) {
      const posts = await client.fetchSubredditPosts(subreddit);
      allPosts.push(...posts);
    }

    // Filter to only posts matching your keywords
    const filtered = filterByKeywords(allPosts);

    // Load previous state, deduplicate, and save updated state
    const state = await loadState();
    const { newPosts, updatedState } = deduplicatePosts(filtered, state);
    await saveState(updatedState);

    // Return new posts along with stats
    return NextResponse.json({
      newPosts,
      stats: {
        totalScanned: allPosts.length,
        afterFilter: filtered.length,
        newPostsCount: newPosts.length,
        duplicatesRemoved: filtered.length - newPosts.length,
      },
    });
  } catch (error) {
    const message = error instanceof Error ? error.message : "Unknown error";
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

What changed in the route?

The route now imports your three new modules and chains them in sequence: fetch all posts, filter by keywords, deduplicate against stored state, save the new state, then return only the genuinely new posts.

It also returns a stats object so you can see exactly how many posts were scanned, how many matched your keywords, how many are new, and how many duplicates were removed.

  • Save route.ts.
  • Open your browser and navigate to http://localhost:3000/api/scrape.
  • Since the route now uses POST, open your terminal and test it by running:
curl -X POST http://localhost:3000/api/scrape
  • You should see a JSON response containing a newPosts array and a stats object with counts for totalScanned, afterFilter, newPostsCount, and duplicatesRemoved.
  • Check your project root. A seen-posts.json file should now exist with the deduplication state.
  • Run the same curl command again. This time newPostsCount should be 0 (or very low) because those posts are now in the seen store.

Getting an error response?

  • If you see a 500 error mentioning authentication, double check your .env.local credentials. Make sure REDDIT_CLIENT_ID and REDDIT_CLIENT_SECRET match what Reddit shows under your app.
  • If you see a module-not-found error, confirm all four files (reddit-client.ts, filter.ts, deduplication.ts, store.ts) are saved inside app/lib/.
  • If you see "Method Not Allowed", you may have navigated to the URL in the browser (which sends a GET). Use the curl -X POST command instead.

Help me debug my scrape route error.

✔️ Awesome, I've got everything!

Great. Double check that all four files are saved.

ⓧ I'd like to double check the full code

import { NextResponse } from "next/server";
import { RedditClient } from "@/app/lib/reddit-client";
import { filterByKeywords } from "@/app/lib/filter";
import { deduplicatePosts } from "@/app/lib/deduplication";
import { loadState, saveState } from "@/app/lib/store";

const TARGET_SUBREDDITS = ["forhire", "hiring", "remotejobs", "SoftwareEngineerJobs"];

export async function POST() {
  try {
    const client = new RedditClient();
    const allPosts = [];

    for (const subreddit of TARGET_SUBREDDITS) {
      const posts = await client.fetchSubredditPosts(subreddit);
      allPosts.push(...posts);
    }

    const filtered = filterByKeywords(allPosts);

    const state = await loadState();
    const { newPosts, updatedState } = deduplicatePosts(filtered, state);
    await saveState(updatedState);

    return NextResponse.json({
      newPosts,
      stats: {
        totalScanned: allPosts.length,
        afterFilter: filtered.length,
        newPostsCount: newPosts.length,
        duplicatesRemoved: filtered.length - newPosts.length,
      },
    });
  } catch (error) {
    const message = error instanceof Error ? error.message : "Unknown error";
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

Your scraper now fetches, filters, and deduplicates job posts in a single request. Next up, you will build the front-end feed UI so you can see these results in the browser instead of raw JSON.

Build the Feed UI

Your backend pipeline is complete. It authenticates with Reddit, fetches posts, filters by keywords, and deduplicates across subreddits.

Now you need a way to trigger that pipeline and see the results. In this step, you'll build a feed page with a button that kicks off a scrape and displays the filtered job posts as cards.

In this step, get ready to:
  • Create the main page component with a scrape button and loading state.
  • Build a reusable JobCard component to display each post.
  • Add stats tracking and empty state messaging to complete the feed.
Create the main page component

The page needs to do three things: give the user a button to trigger a scrape, show a loading indicator while the API works, and render the resulting posts. You'll wire it to the /api/scrape route you built in the previous step.

  • Replace the contents of app/page.tsx with the following code:
"use client";

import { useState } from "react";
import { RedditPost } from "@/app/lib/reddit-client";
import { JobCard } from "@/app/components/job-card";

interface ScrapeStats {
  totalScanned: number;
  afterFilter: number;
  newPostsCount: number;
  duplicatesRemoved: number;
}

export default function Home() {
  const [posts, setPosts] = useState<RedditPost[]>([]);
  const [stats, setStats] = useState<ScrapeStats | null>(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);

  async function handleScrape() {
    setLoading(true);
    setError(null);

    try {
      const response = await fetch("/api/scrape", { method: "POST" });
      const data = await response.json();

      if (!response.ok) {
        throw new Error(data.error || "Scrape failed");
      }

      setPosts(data.newPosts);
      setStats(data.stats);
    } catch (err) {
      setError(err instanceof Error ? err.message : "Something went wrong");
    } finally {
      setLoading(false);
    }
  }

  return (
    <main className="max-w-3xl mx-auto px-4 py-8">
      <h1 className="text-3xl font-bold text-gray-900">Reddit Job Scraper</h1>
      <p className="mt-2 text-gray-600">
        Scrapes r/forhire, r/hiring, r/remotejobs, and r/SoftwareEngineerJobs for
        tech roles matching your stack.
      </p>

      <button
        onClick={handleScrape}
        disabled={loading}
        className="mt-4 px-6 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed"
      >
        {loading ? "Scraping..." : "Scrape Now"}
      </button>

      {error && (
        <div className="mt-4 p-3 bg-red-50 text-red-700 rounded-lg">{error}</div>
      )}

      <div className="mt-6 space-y-4">
        {posts.map((post) => (
          <JobCard key={post.id} post={post} />
        ))}
      </div>
    </main>
  );
}

What does this code do?

This is a client component (marked with "use client") because it needs browser interactivity like button clicks and state updates.

  • handleScrape sends a POST request to your /api/scrape route and stores the response in state.
  • The loading state disables the button and swaps its text to "Scraping..." while the request runs.
  • The error state catches failures and displays them in a red banner.

Why does this import JobCard before it exists?

The page imports JobCard from a file you haven't created yet. Your dev server will show a compile error until you create that component in the next substep.

  • Save app/page.tsx.
Create the JobCard component

Each job post needs a clear, scannable card layout. The card shows the post title as a clickable link to the original Reddit thread, the subreddit it came from, who posted it, how long ago, and a short preview of the body text.

  • Create a new folder called components inside your app folder.
  • Create a new file called job-card.tsx inside app/components and add the following code:
"use client";

import { RedditPost } from "@/app/lib/reddit-client";

interface JobCardProps {
  post: RedditPost;
}

export function JobCard({ post }: JobCardProps) {
  const timeAgo = getRelativeTime(post.created_utc);
  const preview = post.selftext.slice(0, 200) + (post.selftext.length > 200 ? "..." : "");

  return (
    <div className="border border-gray-200 rounded-lg p-4 hover:shadow-md transition-shadow">
      <div className="flex items-start justify-between gap-2">
        <a
          href={`https://reddit.com${post.permalink}`}
          target="_blank"
          rel="noopener noreferrer"
          className="text-lg font-semibold text-blue-600 hover:underline"
        >
          {post.title}
        </a>
        <span className="shrink-0 text-xs bg-orange-100 text-orange-700 px-2 py-1 rounded-full">
          r/{post.subreddit}
        </span>
      </div>
      <div className="mt-1 text-sm text-gray-500">
        Posted by u/{post.author} &middot; {timeAgo}
      </div>
      {preview && (
        <p className="mt-2 text-sm text-gray-700 leading-relaxed">{preview}</p>
      )}
    </div>
  );
}

What does this code do?

  • The title renders as a link pointing to https://reddit.com{permalink} so users can click through to the full post.
  • The subreddit badge (orange pill) tells you which subreddit the post came from at a glance.
  • The preview truncates the selftext to 200 characters so cards stay compact.

The component references a getRelativeTime helper that converts Unix timestamps into human-readable strings like "5m ago" or "2h ago". Add this function below the JobCard component in the same file:

function getRelativeTime(utcSeconds: number): string {
  const now = Date.now();
  const diffMs = now - utcSeconds * 1000;
  const diffMins = Math.floor(diffMs / 60000);

  if (diffMins < 60) return `${diffMins}m ago`;
  const diffHours = Math.floor(diffMins / 60);
  if (diffHours < 24) return `${diffHours}h ago`;
  const diffDays = Math.floor(diffHours / 24);
  return `${diffDays}d ago`;
}

What does this function do?

Reddit stores post timestamps as Unix seconds (seconds since January 1, 1970). This helper subtracts the post time from the current time and formats the difference as minutes, hours, or days.

  • Save app/components/job-card.tsx.
  • Return to your browser at http://localhost:3000. You should see the "Reddit Job Scraper" heading, a description line, and a blue "Scrape Now" button.

Seeing a compile error?

  • Check that your file is saved at exactly app/components/job-card.tsx. The import in page.tsx expects this exact path.
  • Make sure getRelativeTime is defined in the same file, below the JobCard component.
  • Verify that the function is exported as a named export (export function JobCard), not a default export.

Help me debug my JobCard component compile error.

Add stats tracking and empty state

After a scrape runs, the user has no idea how effective the pipeline was. You'll add a stats bar showing how many posts were scanned, how many matched your keywords, how many are genuinely new, and how many duplicates were removed.

  • In app/page.tsx, find the closing )} of the error display block:
      {error && (
        <div className="mt-4 p-3 bg-red-50 text-red-700 rounded-lg">{error}</div>
      )}
  • Add the following stats display directly below that error block, before the <div className="mt-6 space-y-4"> line:
      {stats && (
        <div className="mt-4 flex gap-4 text-sm text-gray-600">
          <span>Scanned: {stats.totalScanned}</span>
          <span>Matched: {stats.afterFilter}</span>
          <span>New: {stats.newPostsCount}</span>
          <span>Duplicates removed: {stats.duplicatesRemoved}</span>
        </div>
      )}

What does this code do?

The stats bar only renders after the first scrape completes (when stats is no longer null). Each span shows one metric from the API response so you can see exactly how the pipeline processed the posts.

Next, you need an empty state message for when the scrape finds no new posts. This happens when every post has already been seen.

  • Inside the <div className="mt-6 space-y-4"> block, add this empty state message above the posts.map line:
        {posts.length === 0 && stats && (
          <p className="text-center text-gray-500 py-8">
            No new jobs found. Try again later!
          </p>
        )}

Why check both conditions?

The condition posts.length === 0 && stats ensures the empty message only shows after a scrape has actually run. Without the stats check, the message would show on first page load before the user clicks anything.

  • Save app/page.tsx.
  • Return to your browser and click the Scrape Now button.
  • You should see the button text change to "Scraping...", then job post cards appear along with a stats bar showing counts like "Scanned: 100", "Matched: 12", "New: 8", "Duplicates removed: 4".
  • Click Scrape Now a second time. You should see "New: 0" and the empty state message "No new jobs found. Try again later!" because the deduplication engine recognizes every post has already been seen.

Not seeing any posts after clicking Scrape?

  • Open your browser's developer console (press F12) and check the **Network** tab for the POST to /api/scrape. If the response is a 500 error, check that your .env.local credentials are correct.
  • If you see an error about RedditClient authentication failing, double-check that your Reddit app type is set to "script" (not "web app").
  • If you previously ran the scrape via the browser URL bar (GET request), those posts are already in seen-posts.json. Delete that file and try again to get a fresh slate.

Help me debug why my Reddit job scraper isn't returning posts.

✔️ Awesome, I've got everything!

Great. Double check that both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete app/components/job-card.tsx:

"use client";

import { RedditPost } from "@/app/lib/reddit-client";

interface JobCardProps {
  post: RedditPost;
}

export function JobCard({ post }: JobCardProps) {
  const timeAgo = getRelativeTime(post.created_utc);
  const preview = post.selftext.slice(0, 200) + (post.selftext.length > 200 ? "..." : "");

  return (
    <div className="border border-gray-200 rounded-lg p-4 hover:shadow-md transition-shadow">
      <div className="flex items-start justify-between gap-2">
        <a
          href={`https://reddit.com${post.permalink}`}
          target="_blank"
          rel="noopener noreferrer"
          className="text-lg font-semibold text-blue-600 hover:underline"
        >
          {post.title}
        </a>
        <span className="shrink-0 text-xs bg-orange-100 text-orange-700 px-2 py-1 rounded-full">
          r/{post.subreddit}
        </span>
      </div>
      <div className="mt-1 text-sm text-gray-500">
        Posted by u/{post.author} &middot; {timeAgo}
      </div>
      {preview && (
        <p className="mt-2 text-sm text-gray-700 leading-relaxed">{preview}</p>
      )}
    </div>
  );
}

function getRelativeTime(utcSeconds: number): string {
  const now = Date.now();
  const diffMs = now - utcSeconds * 1000;
  const diffMins = Math.floor(diffMs / 60000);

  if (diffMins < 60) return `${diffMins}m ago`;
  const diffHours = Math.floor(diffMins / 60);
  if (diffHours < 24) return `${diffHours}h ago`;
  const diffDays = Math.floor(diffHours / 24);
  return `${diffDays}d ago`;
}

And the complete app/page.tsx:

"use client";

import { useState } from "react";
import { RedditPost } from "@/app/lib/reddit-client";
import { JobCard } from "@/app/components/job-card";

interface ScrapeStats {
  totalScanned: number;
  afterFilter: number;
  newPostsCount: number;
  duplicatesRemoved: number;
}

export default function Home() {
  const [posts, setPosts] = useState<RedditPost[]>([]);
  const [stats, setStats] = useState<ScrapeStats | null>(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);

  async function handleScrape() {
    setLoading(true);
    setError(null);

    try {
      const response = await fetch("/api/scrape", { method: "POST" });
      const data = await response.json();

      if (!response.ok) {
        throw new Error(data.error || "Scrape failed");
      }

      setPosts(data.newPosts);
      setStats(data.stats);
    } catch (err) {
      setError(err instanceof Error ? err.message : "Something went wrong");
    } finally {
      setLoading(false);
    }
  }

  return (
    <main className="max-w-3xl mx-auto px-4 py-8">
      <h1 className="text-3xl font-bold text-gray-900">Reddit Job Scraper</h1>
      <p className="mt-2 text-gray-600">
        Scrapes r/forhire, r/hiring, r/remotejobs, and r/SoftwareEngineerJobs for
        tech roles matching your stack.
      </p>

      <button
        onClick={handleScrape}
        disabled={loading}
        className="mt-4 px-6 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed"
      >
        {loading ? "Scraping..." : "Scrape Now"}
      </button>

      {error && (
        <div className="mt-4 p-3 bg-red-50 text-red-700 rounded-lg">{error}</div>
      )}

      {stats && (
        <div className="mt-4 flex gap-4 text-sm text-gray-600">
          <span>Scanned: {stats.totalScanned}</span>
          <span>Matched: {stats.afterFilter}</span>
          <span>New: {stats.newPostsCount}</span>
          <span>Duplicates removed: {stats.duplicatesRemoved}</span>
        </div>
      )}

      <div className="mt-6 space-y-4">
        {posts.length === 0 && stats && (
          <p className="text-center text-gray-500 py-8">
            No new jobs found. Try again later!
          </p>
        )}
        {posts.map((post) => (
          <JobCard key={post.id} post={post} />
        ))}
      </div>
    </main>
  );
}

Your job scraper is fully functional. You have a pipeline that authenticates with Reddit, fetches from four subreddits, filters by your keywords, deduplicates across sources, and displays the results in a clean feed. Next up, take on the Secret Mission to add keyword highlighting and application tracking.

Secret mission

Keyword Highlighting and Applied Tracking

Your scraper finds jobs, but scanning through results still takes effort. What if matched keywords jumped out visually, and you could track which posts you have already applied to? In this mission, you will add keyword highlighting, a "Mark as Applied" button with persistent tracking, and a filter toggle to switch between All, New, and Applied posts.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • A seen-posts.json file storing deduplication state (local file, no cost).
  • An applied-posts.json file storing applied post IDs if you completed the Secret Mission (local file, no cost).
  • A Reddit API app registration (free for non-commercial personal projects).
  • The reddit-job-scraper project folder on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or plan to use the scraper regularly.

  • Your project runs locally with zero ongoing cost.
  • Reddit's free tier allows 100 queries per minute, which is far more than this tool needs.
  • The seen-posts.json and applied-posts.json files will continue to grow as you scrape, but they stay small.

Pause - I'll come back to this later

Shut down any running processes to free up memory. Your files will remain on disk for when you return.

  • If the dev server is still running, press Ctrl+C in the terminal to stop it.
  • Your reddit-job-scraper folder, JSON data files, and Reddit API app will remain intact.
  • When you come back, run npm run dev inside the project folder to start the server again.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below.

  • Delete your Reddit API app by visiting https://www.reddit.com/prefs/apps. Find your app (named "job-scraper" or whatever you chose) and click delete.
  • Stop the dev server if it is still running by pressing Ctrl+C in the terminal.
  • Delete the project folder. On macOS or Linux, run:
cd ~/Desktop
rm -rf reddit-job-scraper

On Windows, run:

cd ~\Desktop
rmdir /s /q reddit-job-scraper

This removes the entire project including .env.local, seen-posts.json, applied-posts.json, and all source code.

Nice Work!

Nice Work!

Nice work! You just built a full-stack job-hunting dashboard that pulls real listings from Reddit, filters them by your tech stack, and intelligently removes duplicates across multiple subreddits.

You've learned how to:

  • Built a Next.js web app that authenticates with Reddit's OAuth2 API using native fetch and TypeScript with no wrapper libraries.
  • Engineered a three-layer deduplication system using post IDs, crosspost metadata, and author+content hashing to eliminate duplicate listings across subreddits.
  • Implemented keyword-based filtering and a persistent JSON store that tracks seen posts between scrapes so you only ever see fresh results.
  • Secret Mission: Added keyword highlighting and "mark as applied" tracking for a complete job-hunting workflow.

Ready to quiz yourself?