Build a GitHub AI Research Crawler

Crawl GitHub repos and organize AI discoveries into an Obsidian second brain.

Introduction

30 Second Summary

Keeping up with the open source AI ecosystem is overwhelming. New frameworks, agent libraries, and LLM tools appear on GitHub every week, and the important projects slip through the cracks before you even know they exist.

In this project, you will build a Python crawler that searches the GitHub API for AI-related repositories across multiple topic categories. It enriches each result with metadata and README excerpts, then feeds structured Markdown files into an Obsidian vault powered by Claude Code.

What You'll Build

A research pipeline that uses the GitHub REST API to discover AI repositories, enrich them with full metadata and README excerpts, and organize them into an Obsidian knowledge base with cross-linked wiki pages.

By the end of this project, you'll have:

  • A topics.yaml config file defining your AI research categories and GitHub search queries.
  • A Python crawler that searches, enriches, and exports GitHub repositories as Markdown files with YAML frontmatter.
  • An Obsidian vault with a raw/wiki architecture and a /discover Claude Code slash command that builds cross-linked wiki pages.
  • Secret Mission: A freshness radar that tracks star growth between crawl runs and surfaces fast-rising repos in your wiki.

Are there any prerequisites?

You should have basic Python knowledge (variables, functions, loops, dictionaries). Familiarity with the command line is helpful but not required.

The project walks through everything else, including setting up Obsidian, Claude Code, and your GitHub token.

Install Your Tools

Your research pipeline connects four tools together. Before you can write any code, each one needs to be installed and working on your machine.

Obsidian is where your knowledge base lives as plain Markdown files. Claude Code is the AI assistant that organizes those files into a wiki. Python powers the crawler script, and VS Code is where you edit the code.

In this step, get ready to:
  • Install Obsidian and create your second brain vault.
  • Install Claude Code and verify it is working.
  • Confirm Python 3 and VS Code are ready to use.
Download and install Obsidian

Obsidian is the app that stores your entire knowledge base. It reads and writes plain Markdown files inside a regular folder on your computer called a vault.

macOS

  • Click the download button for macOS.
  • Open the downloaded .dmg file.
  • Drag Obsidian into your Applications folder.
  • Close the .dmg window.

Windows

  • Click the download button for Windows.
  • Run the downloaded .exe installer.
  • Follow the installer prompts to complete the installation.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Obsidian and press Enter to launch it.

When Obsidian opens for the first time, you will see a vault selector screen.

  • Click Create new vault.
  • Enter second-brain as the vault name.
  • Click Browse next to the folder location and select your Documents folder.
  • Click Create.

Obsidian opens to an empty vault. You should see a blank editor with the vault name second-brain displayed in the sidebar.

What is an Obsidian vault?

A vault is just a regular folder on your computer. Obsidian reads and writes Markdown files inside it.

This means other tools, like Claude Code, can access the exact same files directly. Your vault at ~/Documents/second-brain is just a folder that both Obsidian and your terminal can work with.

Obsidian not opening?

  • On macOS, make sure you dragged Obsidian into your Applications folder. Running it directly from the .dmg mount will not install it properly.
  • On Windows, try right-clicking the installer and selecting Run as administrator.
  • Help me troubleshoot my Obsidian installation.
Install Claude Code

Claude Code is the AI coding assistant that will organize your crawled data into wiki pages later in the project. It runs directly in your terminal.

Claude Code requires a paid account

Claude Code requires a Pro, Max, Team, or Enterprise Claude account. Free Claude accounts will not work. Make sure you have a paid account before continuing.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Install Claude Code by running this command:
curl -fsSL https://claude.ai/install.sh | bash

What does this command do?

curl downloads the installer script from Claude's website. The | bash part pipes that script directly into your shell to run the installation.

  • Verify the installation by running:
claude --version

You should see a version number printed in your terminal output.

Install with Homebrew instead

If you use Homebrew on macOS, you can also install Claude Code by running brew install --cask claude-code in your terminal.

Command not found after installing?

  • Close and reopen your terminal, then try claude --version again. Some installs need a fresh terminal session to register the new command.
  • On Windows, the curl command requires Git Bash or WSL. Try running it in Git Bash instead of Command Prompt.
  • Help me troubleshoot my Claude Code installation.
Verify Python 3 and VS Code

Python is the language you will write the crawler in. VS Code is the editor where you will build and edit your scripts.

  • Check if Python 3 is installed by running this command in your terminal:
python3 --version

✔️ I see a version number

Python 3 is ready to go. Move on to the VS Code check below.

ⓧ Command not found

  • Open your browser and go to python.org/downloads.
  • Click the download button for your operating system.
  • Run the installer.
  • On Windows, check the box that says Add Python to PATH before clicking Install Now.
  • Close and reopen your terminal after the installation finishes.
  • Run python3 --version again to confirm it works.

Still not seeing a version number?

  • On Windows, try python --version (without the 3). Some Windows installers register the command as python instead of python3.
  • Help me get Python 3 working on my machine.

Now check for VS Code.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and look for the app in the results.

✔️ I see Visual Studio Code

VS Code is ready. You do not need to open it right now.

ⓧ It's not installed

  • Open your browser and go to code.visualstudio.com.
  • Click the download button for your operating system.
  • Run the installer and follow the prompts to complete the installation.
  • Search for Visual Studio Code again to confirm it appears in your search results.

Your tools are all set. Next up, you will create a GitHub Personal Access Token so the crawler can authenticate with the GitHub API.

Create a GitHub Personal Access Token

Your tools are installed and ready to go. Now your Python crawler needs a way to authenticate with GitHub.

GitHub's REST API limits unauthenticated requests to just 60 per hour. A Personal Access Token raises that limit to 5,000 requests per hour, giving your crawler plenty of room to search and fetch repository data.

In this step, get ready to:
  • Create a Fine-grained Personal Access Token on GitHub.
  • Set up your project folder with a Python virtual environment.
  • Store your token securely in a .env file.
Navigate to GitHub's token settings
  • Open github.com in your browser and sign in to your GitHub account.
  • Click your profile picture in the top-right corner.
  • Click Settings.
  • In the left sidebar, scroll down and click Developer settings.
  • If GitHub asks you to re-enter your password, enter it to continue.
  • Under Personal access tokens, click Fine-grained tokens.
  • Click Generate new token.
Configure and copy your token
  • Enter second-brain-crawler in the Token name field.
  • Set Expiration to 90 days.
  • Leave Repository access set to Public Repositories (read-only). This is the default and all your crawler needs.
  • Scroll to the bottom of the page and click Generate token.
  • Copy the token value that appears on the screen.

GitHub only shows your token once

Once you leave this page, you cannot view the token again. Paste it somewhere safe (like a notes app) until you store it in your project's .env file in the next substep.

Why fine-grained tokens?

GitHub offers two token types: classic and fine-grained. Fine-grained tokens let you scope access to specific repositories and permissions. For this project, the default read-only access to public repositories is all you need.

Create your project folder and store the token

Your project needs a clean folder structure. The github-ai-crawler/ folder holds the entire project, and backend/ inside it contains your Python code.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder and move into the backend/ subfolder by running these commands:
mkdir -p github-ai-crawler/backend
cd github-ai-crawler/backend
  • Confirm you are in the right folder by running:
pwd

You should see a path ending in github-ai-crawler/backend.

Seeing a different path?

Make sure you ran both the mkdir -p and cd commands. If you only ran mkdir, you are still in the Desktop folder. Run cd github-ai-crawler/backend to move into it.

Help me fix my project folder path.

  • Create a Python virtual environment by running:
python3 -m venv venv
  • Confirm the virtual environment was created by running:
ls

You should see a venv folder in the output.

What is a virtual environment?

A virtual environment is an isolated Python setup that keeps your project's packages separate from your system Python. This prevents version conflicts between different projects on your computer.

Now activate the virtual environment so that any packages you install stay inside this project.

macOS

  • Activate the virtual environment by running:
source venv/bin/activate

Windows

  • Activate the virtual environment by running:
venv\Scripts\activate

Your terminal prompt now starts with (venv), confirming the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command from inside the github-ai-crawler/backend/ folder where the venv folder lives.

On Windows, if you see an execution policy error, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell and try the activation command again.

Why isn't my Python virtual environment activating?

  • Create a .env file to store your GitHub token by running this command, replacing your-token-here with the token you copied from GitHub:
echo "GITHUB_TOKEN=your-token-here" > .env

What does this command do?

The echo command writes the text inside the quotes into a new file called .env. The > symbol redirects the output to the file instead of printing it on screen. Your crawler will read this file later to authenticate with the GitHub API.

  • Verify the file was created correctly by running:
cat .env

You should see GITHUB_TOKEN= followed by your actual token value.

See your-token-here instead of your actual token?

Run the echo command again, this time pasting your real token in place of your-token-here. Make sure the token is inside the double quotes.

If you lost the token, go back to GitHub and generate a new one. Remember, GitHub only displays a token once.

My .env file does not contain my real GitHub token. How do I fix it?

✔️ Awesome, I've got everything!

Great. Double check you have saved your .env file and that (venv) still appears in your terminal prompt.

ⓧ I'd like to double check the full code

Your .env file should contain exactly one line. Replace your-token-here with your actual GitHub token.

GITHUB_TOKEN=your-token-here

Your project folder is set up and your GitHub token is securely stored. Next up, you will build the topic config file and the crawler itself.

Build the Topic Config and Crawler Foundation

Your GitHub token is saved and your project folder is ready to go. Now it's time to write actual code.

Every good crawler starts with a clear definition of what it's looking for. In this step, you'll create a config file that defines your AI research categories, then build the Python functions that search GitHub and pull detailed information about each repository.

In this step, get ready to:
  • Create a topics.yaml config defining your AI research categories.
  • Build the search_repos() function that queries the GitHub Search API.
  • Add the enrich_repo() function that fetches full metadata and README excerpts.
Define your AI research categories

Instead of hardcoding search queries into your Python script, you'll store them in a separate YAML config file. This way, you can add, remove, or tweak your research categories without touching the crawler code.

  • Create a new file called topics.yaml inside your backend/ folder and paste the following config:
categories:
  - name: AI Orchestration Frameworks
    queries:
      - "LLM orchestration framework"
      - "AI agent framework"
      - "topic:langchain OR topic:crewai OR topic:autogen"
    min_stars: 100
    language: ""

  - name: Agent Architectures
    queries:
      - "AI agent swarm multi-agent"
      - "tool calling function calling LLM"
      - "ReAct agent reasoning"
    min_stars: 50
    language: ""

  - name: LLM Infrastructure
    queries:
      - "LLM serving inference engine"
      - "LLM fine-tuning training framework"
      - "LLM evaluation benchmark harness"
    min_stars: 100
    language: ""

  - name: AI Security and Governance
    queries:
      - "AI security LLM safety guardrails"
      - "prompt injection defense"
      - "AI governance compliance framework"
    min_stars: 30
    language: ""

What does this config do?

Each category under categories defines a research area you want to explore on GitHub.

  • name is the human-readable label for the category.
  • queries is a list of GitHub search strings. The crawler runs each one against the Search API.
  • min_stars filters out repos below that threshold, so you only get established projects.
  • language is left empty for now, meaning any programming language is included.
  • Save topics.yaml.
  • Confirm it has four categories listed in VS Code's editor. You should see AI Orchestration Frameworks, Agent Architectures, LLM Infrastructure, and AI Security and Governance.

YAML looking weird or showing errors?

  • YAML is indentation-sensitive. Each level uses exactly 2 spaces, not tabs. Check that queries lines are indented 4 spaces from the left edge.
  • Make sure each query string is wrapped in double quotes. Missing a closing quote will break the file.
  • Help me fix my topics.yaml file
Install dependencies and create the search function

The crawler needs a few Python packages to talk to the GitHub API, parse YAML configs, and load environment variables. You'll install those first, then create the crawler script.

  • Create a new file called requirements.txt inside your backend/ folder and paste the following:
requests==2.33.1
beautifulsoup4==4.14.3
PyYAML==6.0.3
python-dotenv==1.1.0

What are these packages?

  • requests makes HTTP calls to the GitHub API.
  • beautifulsoup4 is an HTML parser, useful for cleaning up README content later.
  • PyYAML reads your topics.yaml config file into Python dictionaries.
  • python-dotenv loads your .env file so your GitHub token stays out of your code.
  • Save requirements.txt.
  • Make sure your virtual environment is still active (you should see (venv) at the start of your terminal prompt). Install the packages by running:
pip install -r requirements.txt
  • You should see Successfully installed followed by the four package names in the terminal output.

Seeing install errors?

  • If you see No such file or directory: requirements.txt, make sure your terminal is inside the backend/ folder where you saved the file.
  • If you don't see (venv) in your prompt, reactivate your virtual environment first.
  • Help me fix my pip install

Now you're ready to create the crawler itself. The REST API for GitHub Search lets you query repositories by keyword, topic, star count, and last-push date. Your first function will build these queries and send them to the API.

  • Create a new file called crawler.py inside your backend/ folder and paste the following imports, constants, and load_topics() function:
import requests
import yaml
import json
import os
import time
import base64
import re
from datetime import datetime, timezone
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN", "")
SEARCH_URL = "https://api.github.com/search/repositories"
API_BASE = "https://api.github.com/repos"
OUTPUT_DIR = "output"

HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "Second-Brain-Crawler",
}
if GITHUB_TOKEN:
    HEADERS["Authorization"] = f"Bearer {GITHUB_TOKEN}"


def load_topics(filepath="topics.yaml"):
    """Load topic categories from YAML config."""
    with open(filepath, "r", encoding="utf-8") as f:
        return yaml.safe_load(f)

What does this code do?

  • load_dotenv() reads your .env file and loads GITHUB_TOKEN into the environment so the script can use it without hardcoding your token.
  • SEARCH_URL and API_BASE store the two GitHub API endpoints the crawler will call.
  • HEADERS includes your authentication token so GitHub knows who is making the requests. Authenticated requests get a limit of 30 search requests per minute instead of 10.
  • load_topics() reads your topics.yaml file and returns it as a Python dictionary.
  • Save crawler.py. You won't see a change yet. The next chunk adds the function that actually searches GitHub.

The GitHub Search API accepts a query string with keywords, star filters, and date qualifiers. Your search_repos() function will assemble that query from your config and send it to the API.

  • In crawler.py, add the search_repos() function below load_topics() by pasting the following code:
def search_repos(query, min_stars=50, language="", max_results=30):
    """Search GitHub repositories using the Search API."""
    q = query
    if min_stars > 0:
        q += f" stars:>={min_stars}"
    if language:
        q += f" language:{language}"
    q += " pushed:>=2025-01-01"

    params = {
        "q": q,
        "sort": "stars",
        "order": "desc",
        "per_page": min(max_results, 100),
    }

    print(f"  Searching: {q[:80]}...")
    response = requests.get(SEARCH_URL, headers=HEADERS, params=params)

    if response.status_code == 200:
        data = response.json()
        return data.get("items", [])
    else:
        print(f"    Search failed: {response.status_code}")
        return []

What does this function do?

  • It takes a search query from your config and appends star and date filters. The pushed:>=2025-01-01 qualifier ensures you only find repos that have been updated recently.
  • It sends a GET request to the GitHub Search API at /search/repositories with results sorted by stars in descending order.
  • If the request succeeds, it returns the list of repo objects from the items key. If it fails, it prints the status code so you can debug.
  • Save crawler.py. You won't be able to test this function on its own yet. The next substep adds enrichment, and Step 4 ties everything together with the main() function.

Why use the Search API instead of scraping?

GitHub's Trending page only shows about 25 repos at a time and can't be filtered by arbitrary topics. The Search API lets you query across all of GitHub with keyword, topic, star, language, and date qualifiers.

For systematic discovery across multiple AI interest areas, the API is the right tool.

✔️ Awesome, I've got everything!

Great. Double check you've saved crawler.py.

ⓧ I'd like to double check the full code

Here is everything in crawler.py so far:

import requests
import yaml
import json
import os
import time
import base64
import re
from datetime import datetime, timezone
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN", "")
SEARCH_URL = "https://api.github.com/search/repositories"
API_BASE = "https://api.github.com/repos"
OUTPUT_DIR = "output"

HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "Second-Brain-Crawler",
}
if GITHUB_TOKEN:
    HEADERS["Authorization"] = f"Bearer {GITHUB_TOKEN}"


def load_topics(filepath="topics.yaml"):
    """Load topic categories from YAML config."""
    with open(filepath, "r", encoding="utf-8") as f:
        return yaml.safe_load(f)


def search_repos(query, min_stars=50, language="", max_results=30):
    """Search GitHub repositories using the Search API."""
    q = query
    if min_stars > 0:
        q += f" stars:>={min_stars}"
    if language:
        q += f" language:{language}"
    q += " pushed:>=2025-01-01"

    params = {
        "q": q,
        "sort": "stars",
        "order": "desc",
        "per_page": min(max_results, 100),
    }

    print(f"  Searching: {q[:80]}...")
    response = requests.get(SEARCH_URL, headers=HEADERS, params=params)

    if response.status_code == 200:
        data = response.json()
        return data.get("items", [])
    else:
        print(f"    Search failed: {response.status_code}")
        return []
Fetch full metadata with the enrich function

The Search API returns basic info about each repo, but your second brain needs richer data. The enrich_repo() function calls a second API endpoint to fetch full metadata and the README content for each repository.

  • In crawler.py, add the enrich_repo() function below search_repos() by pasting the following code:
def enrich_repo(repo):
    """Enrich a repository with full metadata and README excerpt."""
    full_name = repo["full_name"]
    enriched = {
        "name": full_name,
        "short_name": repo["name"],
        "description": repo.get("description", "") or "",
        "url": repo["html_url"],
        "stars": repo.get("stargazers_count", 0),
        "forks": repo.get("forks_count", 0),
        "open_issues": repo.get("open_issues_count", 0),
        "language": repo.get("language", "") or "Unknown",
        "topics": repo.get("topics", []),
        "created_at": repo.get("created_at", ""),
        "updated_at": repo.get("updated_at", ""),
        "license": "",
        "readme_excerpt": "",
    }

    # Extract license
    license_info = repo.get("license")
    if license_info:
        enriched["license"] = license_info.get("spdx_id", "") or ""

    # Fetch README excerpt
    readme_url = f"{API_BASE}/{full_name}/readme"
    try:
        resp = requests.get(readme_url, headers=HEADERS)
        if resp.status_code == 200:
            content_b64 = resp.json().get("content", "")
            if content_b64:
                readme_text = base64.b64decode(content_b64).decode("utf-8", errors="replace")
                # Strip markdown headers and links for a clean excerpt
                clean = re.sub(r"[#*\[\]`>]", "", readme_text)
                clean = re.sub(r"\(http[^\)]+\)", "", clean)
                clean = " ".join(clean.split())
                enriched["readme_excerpt"] = clean[:500]
    except requests.RequestException:
        pass

    time.sleep(0.5)
    return enriched

What does this function do?

  • It builds an enriched dictionary from the search result, pulling out fields like stars, forks, language, topics, and license.
  • It calls GET /repos/{owner}/{repo}/readme to fetch the README file. GitHub returns README content as Base64-encoded text, so the function decodes it and strips out Markdown formatting characters to get a clean excerpt.
  • The excerpt is capped at 500 characters. This gives Claude Code enough context to summarize each repo without hitting token limits later.
  • time.sleep(0.5) pauses half a second between API calls to stay within GitHub's rate limits (5,000 requests per hour with a token).
  • Save crawler.py.
  • Scroll through the file and confirm you see three functions: load_topics(), search_repos(), and enrich_repo().

Seeing red underlines or import errors?

  • Make sure you installed the packages with pip install -r requirements.txt earlier in this step. VS Code might also need you to select the correct Python interpreter. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and pick the one that shows your venv path.
  • Check that your .env file is in the same backend/ folder as crawler.py. The load_dotenv() call looks for .env in the current working directory.
  • Help me debug import errors in my crawler

✔️ Awesome, I've got everything!

Great. Make sure crawler.py is saved and you're ready to move on.

ⓧ I'd like to double check the full code

Here is the complete crawler.py file at the end of this step:

import requests
import yaml
import json
import os
import time
import base64
import re
from datetime import datetime, timezone
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN", "")
SEARCH_URL = "https://api.github.com/search/repositories"
API_BASE = "https://api.github.com/repos"
OUTPUT_DIR = "output"

HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "Second-Brain-Crawler",
}
if GITHUB_TOKEN:
    HEADERS["Authorization"] = f"Bearer {GITHUB_TOKEN}"


def load_topics(filepath="topics.yaml"):
    """Load topic categories from YAML config."""
    with open(filepath, "r", encoding="utf-8") as f:
        return yaml.safe_load(f)


def search_repos(query, min_stars=50, language="", max_results=30):
    """Search GitHub repositories using the Search API."""
    q = query
    if min_stars > 0:
        q += f" stars:>={min_stars}"
    if language:
        q += f" language:{language}"
    q += " pushed:>=2025-01-01"

    params = {
        "q": q,
        "sort": "stars",
        "order": "desc",
        "per_page": min(max_results, 100),
    }

    print(f"  Searching: {q[:80]}...")
    response = requests.get(SEARCH_URL, headers=HEADERS, params=params)

    if response.status_code == 200:
        data = response.json()
        return data.get("items", [])
    else:
        print(f"    Search failed: {response.status_code}")
        return []


def enrich_repo(repo):
    """Enrich a repository with full metadata and README excerpt."""
    full_name = repo["full_name"]
    enriched = {
        "name": full_name,
        "short_name": repo["name"],
        "description": repo.get("description", "") or "",
        "url": repo["html_url"],
        "stars": repo.get("stargazers_count", 0),
        "forks": repo.get("forks_count", 0),
        "open_issues": repo.get("open_issues_count", 0),
        "language": repo.get("language", "") or "Unknown",
        "topics": repo.get("topics", []),
        "created_at": repo.get("created_at", ""),
        "updated_at": repo.get("updated_at", ""),
        "license": "",
        "readme_excerpt": "",
    }

    # Extract license
    license_info = repo.get("license")
    if license_info:
        enriched["license"] = license_info.get("spdx_id", "") or ""

    # Fetch README excerpt
    readme_url = f"{API_BASE}/{full_name}/readme"
    try:
        resp = requests.get(readme_url, headers=HEADERS)
        if resp.status_code == 200:
            content_b64 = resp.json().get("content", "")
            if content_b64:
                readme_text = base64.b64decode(content_b64).decode("utf-8", errors="replace")
                # Strip markdown headers and links for a clean excerpt
                clean = re.sub(r"[#*\[\]`>]", "", readme_text)
                clean = re.sub(r"\(http[^\)]+\)", "", clean)
                clean = " ".join(clean.split())
                enriched["readme_excerpt"] = clean[:500]
    except requests.RequestException:
        pass

    time.sleep(0.5)
    return enriched

Your crawler can now search GitHub and enrich each result with full metadata and README excerpts. Next up, you'll add the functions that turn these results into Markdown files and run the full pipeline.

Generate Markdown Files for Your Vault

Your crawler can now search GitHub and enrich each result with full metadata and a README excerpt. That enriched data is sitting in Python dictionaries, but your second brain needs Markdown files.

In this step, you will add the functions that convert each enriched repo into a structured Markdown file with YAML frontmatter. Then you will wire everything together into a main() function that orchestrates the full pipeline and run it for the first time.

In this step, get ready to:
  • Add a function that converts enriched repo data into Markdown files with YAML frontmatter.
  • Add the main orchestrator that ties search, enrichment, and file writing together.
  • Run the crawler and verify that it produces structured Markdown output.
Add the generate_markdown() function

Each enriched repo dictionary needs to become a Markdown file that Obsidian can read. The file format uses YAML frontmatter at the top (metadata between --- delimiters) followed by human-readable body sections.

  • In crawler.py, find the end of the enrich_repo() function, then copy the following code below it:

def generate_markdown(repo, category_name):
    # Build YAML frontmatter with all repo metadata
    frontmatter = {
        "title": repo["short_name"],
        "full_name": repo["name"],
        "category": category_name,
        "stars": repo["stars"],
        "forks": repo["forks"],
        "open_issues": repo["open_issues"],
        "language": repo["language"],
        "license": repo["license"],
        "topics": repo["topics"],
        "created_at": repo["created_at"],
        "updated_at": repo["updated_at"],
        "url": repo["url"],
        "crawled_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
    }

    # Format frontmatter as YAML between --- delimiters
    lines = ["---"]
    for key, value in frontmatter.items():
        if isinstance(value, list):
            lines.append(f"{key}:")
            for item in value:
                lines.append(f"  - {item}")
        else:
            lines.append(f"{key}: \"{value}\"" if isinstance(value, str) else f"{key}: {value}")
    lines.append("---")
    lines.append("")

    # Add title and summary stats
    lines.append(f"# {repo['short_name']}")
    lines.append("")
    lines.append(f"**Repository**: [{repo['name']}]({repo['url']})")
    lines.append(f"**Category**: {category_name}")
    lines.append(f"**Stars**: {repo['stars']:,} | **Forks**: {repo['forks']:,} | **Issues**: {repo['open_issues']:,}")
    lines.append(f"**Language**: {repo['language']} | **License**: {repo['license'] or 'Not specified'}")
    lines.append("")

    # Append description and README excerpt if available
    if repo["description"]:
        lines.append("## Description")
        lines.append("")
        lines.append(repo["description"])
        lines.append("")

    if repo["readme_excerpt"]:
        lines.append("## README Excerpt")
        lines.append("")
        lines.append(repo["readme_excerpt"])
        lines.append("")

    return "\n".join(lines)

What does this code do?

The frontmatter dictionary holds all the structured metadata (stars, language, license, topics) that Obsidian can query and filter on later.

The for loop serializes each key-value pair into YAML format, handling lists (like topics) differently from plain strings. The crawled_at timestamp records when you discovered each repo so you can track freshness over time.

You will not see any output yet. The next substep adds the main() function that calls generate_markdown() for every repo.

Add the main() orchestrator and run the crawler

The main() function ties every piece together. It loads your topic categories, runs searches across all queries, deduplicates results, enriches each unique repo, and writes Markdown files to an output directory.

  • In crawler.py, add the following code below the generate_markdown() function you just wrote:

def sanitize_filename(name):
    # Replace slashes with double dashes for safe filenames
    return name.replace("/", "--") + ".md"


def main():
    print("Loading topic categories...")
    config = load_topics()
    categories = config.get("categories", [])
    print(f"Found {len(categories)} categories.\n")

    # Create the output directory if it doesn't exist
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    # Search each category and deduplicate by full repo name
    all_repos = {}
    for category in categories:
        cat_name = category["name"]
        min_stars = category.get("min_stars", 50)
        language = category.get("language", "")
        print(f"Category: {cat_name}")

        for query in category.get("queries", []):
            results = search_repos(query, min_stars=min_stars, language=language)
            for repo in results:
                full_name = repo["full_name"]
                if full_name not in all_repos:
                    all_repos[full_name] = {
                        "repo": repo,
                        "category": cat_name,
                    }
            time.sleep(2)

        print(f"  Total unique repos so far: {len(all_repos)}\n")

    # Enrich each unique repo and write a Markdown file
    print(f"Enriching {len(all_repos)} unique repositories...")
    count = 0
    for full_name, entry in all_repos.items():
        count += 1
        print(f"  [{count}/{len(all_repos)}] {full_name}")
        enriched = enrich_repo(entry["repo"])
        md_content = generate_markdown(enriched, entry["category"])

        filename = sanitize_filename(full_name)
        filepath = os.path.join(OUTPUT_DIR, filename)
        with open(filepath, "w", encoding="utf-8") as f:
            f.write(md_content)

    print(f"\nDone! {count} Markdown files written to {OUTPUT_DIR}/")


if __name__ == "__main__":
    main()

What does this code do?

  • sanitize_filename() converts a repo's full name (like owner/repo) into a safe filename by replacing the slash with double dashes.
  • The all_repos dictionary deduplicates results. If the same repo appears across multiple search queries, it only gets stored once.
  • time.sleep(2) pauses 2 seconds between search queries to stay within the GitHub Search API's rate limit of 30 requests per minute.
  • if __name__ == "__main__" ensures main() only runs when you execute the file directly, not when importing it as a module.
  • Save crawler.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete crawler.py file. Compare it against yours and fix any differences.

import requests
import yaml
import json
import os
import time
import base64
import re
from datetime import datetime, timezone
from dotenv import load_dotenv

load_dotenv()

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN", "")
SEARCH_URL = "https://api.github.com/search/repositories"
API_BASE = "https://api.github.com/repos"
OUTPUT_DIR = "output"

HEADERS = {
    "Accept": "application/vnd.github+json",
    "User-Agent": "Second-Brain-Crawler",
}
if GITHUB_TOKEN:
    HEADERS["Authorization"] = f"Bearer {GITHUB_TOKEN}"


def load_topics(filepath="topics.yaml"):
    with open(filepath, "r", encoding="utf-8") as f:
        return yaml.safe_load(f)


def search_repos(query, min_stars=50, language="", max_results=30):
    q = query
    if min_stars > 0:
        q += f" stars:>={min_stars}"
    if language:
        q += f" language:{language}"
    q += " pushed:>=2025-01-01"
    params = {
        "q": q,
        "sort": "stars",
        "order": "desc",
        "per_page": min(max_results, 100),
    }
    print(f"  Searching: {q[:80]}...")
    response = requests.get(SEARCH_URL, headers=HEADERS, params=params)
    if response.status_code == 200:
        data = response.json()
        return data.get("items", [])
    else:
        print(f"    Search failed: {response.status_code}")
        return []


def enrich_repo(repo):
    full_name = repo["full_name"]
    enriched = {
        "name": full_name,
        "short_name": repo["name"],
        "description": repo.get("description", "") or "",
        "url": repo["html_url"],
        "stars": repo.get("stargazers_count", 0),
        "forks": repo.get("forks_count", 0),
        "open_issues": repo.get("open_issues_count", 0),
        "language": repo.get("language", "") or "Unknown",
        "topics": repo.get("topics", []),
        "created_at": repo.get("created_at", ""),
        "updated_at": repo.get("updated_at", ""),
        "license": "",
        "readme_excerpt": "",
    }
    license_info = repo.get("license")
    if license_info:
        enriched["license"] = license_info.get("spdx_id", "") or ""
    readme_url = f"{API_BASE}/{full_name}/readme"
    try:
        resp = requests.get(readme_url, headers=HEADERS)
        if resp.status_code == 200:
            content_b64 = resp.json().get("content", "")
            if content_b64:
                readme_text = base64.b64decode(content_b64).decode("utf-8", errors="replace")
                clean = re.sub(r"[#*\[\]`>]", "", readme_text)
                clean = re.sub(r"\(http[^\)]+\)", "", clean)
                clean = " ".join(clean.split())
                enriched["readme_excerpt"] = clean[:500]
    except requests.RequestException:
        pass
    time.sleep(0.5)
    return enriched


def generate_markdown(repo, category_name):
    frontmatter = {
        "title": repo["short_name"],
        "full_name": repo["name"],
        "category": category_name,
        "stars": repo["stars"],
        "forks": repo["forks"],
        "open_issues": repo["open_issues"],
        "language": repo["language"],
        "license": repo["license"],
        "topics": repo["topics"],
        "created_at": repo["created_at"],
        "updated_at": repo["updated_at"],
        "url": repo["url"],
        "crawled_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
    }
    lines = ["---"]
    for key, value in frontmatter.items():
        if isinstance(value, list):
            lines.append(f"{key}:")
            for item in value:
                lines.append(f"  - {item}")
        else:
            lines.append(f"{key}: \"{value}\"" if isinstance(value, str) else f"{key}: {value}")
    lines.append("---")
    lines.append("")
    lines.append(f"# {repo['short_name']}")
    lines.append("")
    lines.append(f"**Repository**: [{repo['name']}]({repo['url']})")
    lines.append(f"**Category**: {category_name}")
    lines.append(f"**Stars**: {repo['stars']:,} | **Forks**: {repo['forks']:,} | **Issues**: {repo['open_issues']:,}")
    lines.append(f"**Language**: {repo['language']} | **License**: {repo['license'] or 'Not specified'}")
    lines.append("")
    if repo["description"]:
        lines.append("## Description")
        lines.append("")
        lines.append(repo["description"])
        lines.append("")
    if repo["readme_excerpt"]:
        lines.append("## README Excerpt")
        lines.append("")
        lines.append(repo["readme_excerpt"])
        lines.append("")
    return "\n".join(lines)


def sanitize_filename(name):
    return name.replace("/", "--") + ".md"


def main():
    print("Loading topic categories...")
    config = load_topics()
    categories = config.get("categories", [])
    print(f"Found {len(categories)} categories.\n")
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    all_repos = {}
    for category in categories:
        cat_name = category["name"]
        min_stars = category.get("min_stars", 50)
        language = category.get("language", "")
        print(f"Category: {cat_name}")
        for query in category.get("queries", []):
            results = search_repos(query, min_stars=min_stars, language=language)
            for repo in results:
                full_name = repo["full_name"]
                if full_name not in all_repos:
                    all_repos[full_name] = {
                        "repo": repo,
                        "category": cat_name,
                    }
            time.sleep(2)
        print(f"  Total unique repos so far: {len(all_repos)}\n")
    print(f"Enriching {len(all_repos)} unique repositories...")
    count = 0
    for full_name, entry in all_repos.items():
        count += 1
        print(f"  [{count}/{len(all_repos)}] {full_name}")
        enriched = enrich_repo(entry["repo"])
        md_content = generate_markdown(enriched, entry["category"])
        filename = sanitize_filename(full_name)
        filepath = os.path.join(OUTPUT_DIR, filename)
        with open(filepath, "w", encoding="utf-8") as f:
            f.write(md_content)
    print(f"\nDone! {count} Markdown files written to {OUTPUT_DIR}/")


if __name__ == "__main__":
    main()
Run the crawler and inspect the output

The crawler is complete. Time to let it loose on the GitHub API and see what it finds across your four topic categories.

  • Make sure your terminal is inside the github-ai-crawler/backend/ folder with your virtual environment activated.
  • Run the crawler by executing this command:
python3 crawler.py

What should I see?

The crawler prints progress as it runs. First you will see it loading categories and searching each query. Then it enriches each unique repo one by one.

The full run takes 2 to 5 minutes depending on how many repos it finds. The time.sleep() calls keep you within the GitHub API rate limits. When finished, you will see a line like Done! 85 Markdown files written to output/ (your exact number will vary).

  • Once the crawler finishes, confirm the output files exist by running:
ls output/

You should see a list of Markdown files named with the pattern owner--repo.md (for example, langchain-ai--langchain.md).

  • Inspect one of the generated files to verify the structure by running:
cat output/$(ls output/ | head -1)

You should see YAML frontmatter between --- delimiters at the top, followed by a Markdown heading, metadata summary, description section, and README excerpt.

Seeing an error when you run the crawler?

  • ModuleNotFoundError: No module named 'requests' means your virtual environment is not activated. Run source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows), then try again.
  • Search failed: 401 means your GitHub token is invalid or missing. Open .env and make sure GITHUB_TOKEN contains the full token string with no extra spaces or quotes.
  • Search failed: 403 means you have hit the rate limit. Wait a few minutes and try again. Authenticated requests allow 30 search calls per minute.
  • FileNotFoundError for topics.yaml means your terminal is not in the backend/ folder. Run cd ~/Desktop/github-ai-crawler/backend and try again.

Why is my crawler failing or returning zero results?

Your crawler just turned GitHub's API into a pile of structured, searchable Markdown files. Next up, you will build the Obsidian vault where these files live and teach Claude Code how to read them.

Set Up Your Second Brain Vault

Your crawler has done its job. You now have a folder full of Markdown files, each one a structured snapshot of a GitHub repository your pipeline discovered and enriched.

Raw data is only useful if you can find and connect it later. In this step, you'll use Claude Code to scaffold your Obsidian vault with a structured folder system that separates raw crawled data from curated wiki content.

In this step, get ready to:
  • Scaffold the raw/wiki folder structure inside your Obsidian vault using Claude Code.
  • Create a CLAUDE.md file that teaches Claude your vault's conventions.
  • Copy your crawler output into the vault so Obsidian can display it.
Open Claude Code and scaffold the vault structure

Your Obsidian vault is just a regular folder on your computer. Claude Code can create files and folders inside it directly.

What is the raw/wiki pattern?

This architecture separates your vault into two zones. The raw/ folder holds unprocessed source material like crawled data and clippings. The wiki/ folder holds curated, cross-linked knowledge pages you actually read and reference.

New information always lands in raw/ first, then gets processed into wiki/ pages. This keeps your curated knowledge clean and traceable back to its source.

  • Navigate to your Obsidian vault by running this command:
cd ~/Documents/second-brain
  • Open Claude Code by running:
claude

Why launch Claude Code from this folder?

Claude Code treats the directory you launch it from as its working project. Any files it creates will land inside this folder. It also automatically reads a CLAUDE.md file from this directory if one exists, which you will create shortly.

Once Claude Code starts, you will see an interactive prompt where you can type or paste instructions.

  • Paste the following prompt into Claude Code to scaffold the folder structure:
Create the following folder structure in this directory:

- raw/github-discoveries/ (for storing raw crawled Markdown files)
- wiki/concepts/ (for concept and technology pages)
- wiki/projects/ (for individual project deep-dives)
- wiki/people/ (for notable contributors and researchers)
- wiki/index.md (a top-level index linking to all wiki sections)
- wiki/log.md (a changelog tracking what was added and when)

For index.md, add headings for Concepts, Projects, and People with placeholder text explaining what goes in each section. For log.md, add a heading and a placeholder entry with today's date.

What will Claude Code do with this prompt?

Claude Code reads your prompt, plans the folder and file creation, then asks for your permission before making changes. You will see it list each folder and file it wants to create.

  • When Claude Code asks for permission to create the folders and files, approve the changes.
  • Confirm the folders were created by checking that Claude Code's output lists all six paths: raw/github-discoveries/, wiki/concepts/, wiki/projects/, wiki/people/, wiki/index.md, and wiki/log.md.

Claude Code not responding or erroring out?

Make sure you launched Claude Code from the ~/Documents/second-brain directory. If you see an authentication error, confirm you are signed into a Pro, Max, Team, or Enterprise Claude account. Free accounts do not work with Claude Code.

help me troubleshoot Claude Code errors

Create CLAUDE.md to define vault conventions

Claude Code automatically reads a file called CLAUDE.md when it launches from a directory. This file is like a rulebook that teaches Claude how your vault is organized and how to write content for it.

  • Paste this prompt into Claude Code to create CLAUDE.md:
Create a CLAUDE.md file at the root of this directory with these four sections:

## Project Structure
Document the raw/wiki folder separation. Explain that raw/ contains unprocessed source material (crawled repos, notes, clippings) and wiki/ contains curated, cross-linked knowledge pages. New information always lands in raw/ first, then gets processed into wiki/ pages.

## Page Conventions
Define the YAML frontmatter schema for wiki pages:
- title (string, required)
- category (string: concept | project | person | comparison)
- tags (list of strings)
- sources (list of URLs)
- created (ISO date)
- updated (ISO date)

## Style Guide
- Write in clear, direct prose
- Cite sources with links when referencing specific claims
- Note contradictions between sources rather than hiding them
- Use [[wiki-links]] to connect related pages
- Keep pages focused on one topic each

## Domain Context
Ask me 3-5 questions about my AI work, interests, and goals so you can tailor wiki content to my specific context. Wait for my answers before writing this section.

What do the four sections do?

  • Project Structure tells Claude where raw data lives and where curated wiki pages go.
  • Page Conventions defines the YAML frontmatter schema so every wiki page has consistent metadata.
  • Style Guide sets writing rules so Claude produces clear, well-sourced pages with wiki-links between related topics.
  • Domain Context personalizes the vault. Claude will ask about your AI interests before filling in this section.
  • Approve the file creation when Claude Code asks for permission.
  • Claude Code will ask you 3-5 questions about your AI work and interests for the Domain Context section. Answer each question based on your real interests.

Not sure what to answer?

You can share things like which AI topics excite you most, what projects you are working on, or what role you are targeting in your career. There are no wrong answers. Claude uses your responses to write more relevant wiki content later.

  • After Claude finishes writing the Domain Context section, confirm that CLAUDE.md was created by checking Claude Code's output for a success message referencing the file.

Claude skipped the Domain Context questions?

Sometimes Claude creates all four sections in one go without pausing to ask questions. If that happens, paste a follow-up prompt: "Update the Domain Context section in CLAUDE.md. Ask me 3-5 questions about my AI work first, then rewrite the section based on my answers."

help me get Claude Code to ask Domain Context questions

Copy your crawler output into the vault

Your crawler output is sitting in the project folder on your Desktop. Your vault is in Documents. You need to copy the Markdown files into the vault's raw/github-discoveries/ folder so Obsidian and Claude Code can both access them.

  • Exit Claude Code by typing:
/exit
  • Copy all Markdown files from your crawler output into the vault by running this command:
cp ~/Desktop/github-ai-crawler/backend/output/*.md ~/Documents/second-brain/raw/github-discoveries/

What does this command do?

The cp command copies files. The *.md pattern matches every Markdown file in the output folder. The destination is the raw/github-discoveries/ folder you just created inside your vault.

  • Confirm the files were copied by running:
ls ~/Documents/second-brain/raw/github-discoveries/

You should see a list of .md files, one for each repository your crawler discovered. Each filename follows the owner--repo.md pattern.

  • Switch to Obsidian and open your second-brain vault.
  • In the left sidebar, expand the raw folder, then expand github-discoveries.
  • Click on any file to open it. You should see the YAML frontmatter at the top and the description and README excerpt sections below it.

Files not showing up in Obsidian?

Make sure your vault is pointed at ~/Documents/second-brain. You can check this in Obsidian by clicking the vault name in the bottom-left corner. If you created the vault in a different location in Step 1, update the cp command destination to match your actual vault path.

help me find my Obsidian vault location

Your vault now has structure, conventions, and raw data. Next up, you will create a custom slash command that turns Claude Code into your personal research librarian.

Create the /discover Slash Command and Ingest

Your vault is loaded with raw discovery files and a CLAUDE.md that teaches Claude Code how your vault is organized.

Now you'll wire up a custom slash command that turns those raw files into a cross-linked knowledge base. One command is all it takes to generate concept pages, project pages, and comparison tables from your crawled data.

In this step, get ready to:
  • Create a custom /discover slash command file.
  • Run the command to generate wiki pages from crawled repos.
  • Explore the cross-linked knowledge graph in Obsidian.
Create the /discover slash command

Claude Code slash commands are Markdown files stored in a .claude/commands/ directory at the root of your project. Each file becomes a command you can invoke with / inside a Claude Code session.

Claude Code only scans for new commands when a session starts. You'll create the command file first, then restart Claude Code to pick it up.

  • Open a new terminal window. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.

Why a new terminal?

Your existing terminal is running a Claude Code session. You need a separate terminal to create files on disk. Keep the Claude Code terminal open for the next substep.

  • Navigate to your vault and create the slash commands directory by running these commands:
cd ~/Documents/second-brain
mkdir -p .claude/commands
  • Confirm the directory was created by running:
ls .claude/commands

You should see an empty directory listed with no errors.

Seeing 'No such file or directory'?

Make sure you ran cd ~/Documents/second-brain first. The mkdir -p command creates all parent directories, so if .claude/ does not exist yet, it creates both folders in one go.

Still stuck? Help me troubleshoot why my .claude/commands directory was not created.

Now you need to create the actual slash command file. This file contains the instructions Claude Code will follow when you run /discover.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to Documents, select the second-brain folder, then click Open.
  • In the Explorer sidebar on the left, expand the .claude folder, then expand commands.
  • Right-click the commands folder and select New File.
  • Name the file discover.md and press Enter.
  • Paste the following content into discover.md:
Read every Markdown file in `raw/github-discoveries/`. Parse the YAML frontmatter and body content from each file.

Then create wiki pages following these instructions:

## Concept Pages

For each unique `category` value found across the crawled files, create a page in `wiki/concepts/` with a slugified filename (e.g., `ai-orchestration-frameworks.md`).

Each concept page must include:
- YAML frontmatter with `type: concept`, `category`, and `last_updated`
- A brief overview paragraph explaining what this category covers
- A list of all repositories in this category as `[[wiki-links]]` to their project pages, sorted by star count (highest first)
- A summary of common languages, licenses, and themes

## Project Pages

For each repository, create a page in `wiki/projects/` named after the repo (e.g., `langchain.md`).

Each project page must include:
- YAML frontmatter with `type: project`, `stars`, `language`, `license`, `url`, `category`, and `last_updated`
- The description and README excerpt from the raw file
- A `[[wiki-link]]` to the relevant concept page
- A "Related Projects" section with `[[wiki-links]]` to other projects in the same category

## Comparison Pages

When 3 or more repositories in a category serve a similar purpose, create a comparison page in `wiki/concepts/` (e.g., `orchestration-framework-comparison.md`).

Each comparison page must include:
- YAML frontmatter with `type: comparison`, `category`, and `last_updated`
- A Markdown table comparing repos by stars, language, license, and a one-line summary
- `[[wiki-links]]` to each project page

## Update Index

Replace the contents of `wiki/index.md` with:
- A section for each category listing its concept page and project pages as `[[wiki-links]]`
- A "Comparisons" section listing all comparison pages
- A "Recently Crawled" section with the 10 most recently crawled repos by `crawled_at` date

## Update Log

Append a new entry to `wiki/log.md` with:
- Current date and time
- Number of raw files processed
- Number of concept, project, and comparison pages created
- Categories covered

What does this command file do?

This file is the instruction set Claude Code follows when you run /discover. It tells Claude to read every raw file your crawler produced and create three types of wiki pages from them.

  • Concept pages group repos by category (like "AI Orchestration Frameworks") and link out to individual project pages.
  • Project pages give each repo its own wiki entry with metadata, a README excerpt, and cross-links back to the concept page.
  • Comparison pages create side-by-side tables for categories where multiple tools compete.
  • The command also updates wiki/index.md as a central hub and logs every ingestion run to wiki/log.md.
  • Save discover.md with Cmd+S (macOS) or Ctrl+S (Windows).
  • Back in your new terminal, confirm the file was created by running:
ls .claude/commands

You should see discover.md listed in the output.

Don't see discover.md?

Make sure you created the file inside .claude/commands/ and not in the vault root or another folder. The file must be named exactly discover.md (lowercase, with the .md extension).

If the .claude folder is hidden in VS Code's Explorer, click File, then Preferences, then Settings, and search for files.exclude to check if dot-folders are excluded.

Need more help? Help me figure out why discover.md is not showing up in .claude/commands/.

✔️ Awesome, I've got everything!

Great. Double check you've saved discover.md before moving on.

ⓧ I'd like to double check the full code

Here is the complete discover.md file. Make sure your version matches exactly.

Read every Markdown file in `raw/github-discoveries/`. Parse the YAML frontmatter and body content from each file.

Then create wiki pages following these instructions:

## Concept Pages

For each unique `category` value found across the crawled files, create a page in `wiki/concepts/` with a slugified filename (e.g., `ai-orchestration-frameworks.md`).

Each concept page must include:
- YAML frontmatter with `type: concept`, `category`, and `last_updated`
- A brief overview paragraph explaining what this category covers
- A list of all repositories in this category as `[[wiki-links]]` to their project pages, sorted by star count (highest first)
- A summary of common languages, licenses, and themes

## Project Pages

For each repository, create a page in `wiki/projects/` named after the repo (e.g., `langchain.md`).

Each project page must include:
- YAML frontmatter with `type: project`, `stars`, `language`, `license`, `url`, `category`, and `last_updated`
- The description and README excerpt from the raw file
- A `[[wiki-link]]` to the relevant concept page
- A "Related Projects" section with `[[wiki-links]]` to other projects in the same category

## Comparison Pages

When 3 or more repositories in a category serve a similar purpose, create a comparison page in `wiki/concepts/` (e.g., `orchestration-framework-comparison.md`).

Each comparison page must include:
- YAML frontmatter with `type: comparison`, `category`, and `last_updated`
- A Markdown table comparing repos by stars, language, license, and a one-line summary
- `[[wiki-links]]` to each project page

## Update Index

Replace the contents of `wiki/index.md` with:
- A section for each category listing its concept page and project pages as `[[wiki-links]]`
- A "Comparisons" section listing all comparison pages
- A "Recently Crawled" section with the 10 most recently crawled repos by `crawled_at` date

## Update Log

Append a new entry to `wiki/log.md` with:
- Current date and time
- Number of raw files processed
- Number of concept, project, and comparison pages created
- Categories covered
Restart Claude Code and run /discover

Claude Code only loads slash commands when a session starts. Since you just created discover.md, you need to restart your session for Claude Code to detect it.

  • Switch to the terminal window where Claude Code is running.
  • Exit the current session by typing:
/exit
  • Start a new Claude Code session by running:
claude

Make sure you are still in the ~/Documents/second-brain directory when you run this. Claude Code reads CLAUDE.md and detects slash commands from the directory it launches in.

  • Run the discover command by typing:
/discover

What happens when you run /discover?

Claude Code reads the instructions from discover.md and starts processing every file in raw/github-discoveries/. It creates concept pages, project pages, and comparison tables in the wiki/ folders.

This can take a few minutes depending on how many repos your crawler found. Claude Code will ask for permission to create and edit files. Allow it each time.

Once Claude Code finishes, you should see new files appearing in wiki/concepts/ and wiki/projects/.

  • Confirm the wiki pages were created by running this in your separate terminal:
ls wiki/concepts/ wiki/projects/

You should see multiple .md files in both directories. The concept pages will be named after your categories (like ai-orchestration-frameworks.md) and the project pages will be named after individual repos.

Claude Code does not recognize /discover?

Make sure you restarted Claude Code after creating the file. Type /exit, then run claude again from the ~/Documents/second-brain directory.

Check that discover.md is at the path .claude/commands/discover.md relative to the vault root. If you accidentally placed it in .claude/ without the commands/ subfolder, Claude Code will not find it.

Still not working? Help me troubleshoot why Claude Code does not recognize my /discover slash command.

Explore your knowledge graph in Obsidian

Your wiki is now populated with interconnected pages. Obsidian's Graph View lets you see all the connections between your wiki pages as a visual network.

  • Switch to the Obsidian app.
  • Open Graph View by pressing Cmd+G (macOS) or Ctrl+G (Windows).

You should see a network of nodes and lines. Each node is a wiki page. Each line is a cross-link between pages. Concept pages will be hubs with many connections radiating outward to project pages.

  • Click the wiki/index.md node in the graph (or open it from the file explorer on the left).
  • Verify it lists all your categories with links to concept pages and project pages.
  • Click through a few wiki-links to confirm the cross-linked structure works. Concept pages should link to project pages, and project pages should link back to their concept page.
  • Open wiki/log.md and confirm it has a timestamped entry from the ingestion run.

What is the raw/wiki pattern?

The raw folder holds unprocessed data straight from your crawler. The wiki folder holds curated, cross-linked pages that Claude Code synthesized from that raw data.

This separation means you can re-crawl and dump new files into raw/ without touching your curated wiki. Then run /discover again to integrate the new findings. Your knowledge base grows without losing previous work.

Graph View looks empty or disconnected?

Open a concept page and check that it uses [[double bracket]] wiki-links, not plain Markdown links. Obsidian only draws graph connections for [[wiki-links]].

If your wiki folders are empty, go back to your Claude Code session and check for error messages. Claude Code may have run out of context or encountered a permission issue. Run /discover again to retry.

Still having trouble? Help me figure out why my Obsidian Graph View is not showing connections between wiki pages.

Secret mission

Add a Freshness Radar

Your crawler finds repos once, but the AI landscape moves fast. New tools gain hundreds of stars overnight. Can you add a freshness radar that re-checks star counts and highlights which repos are gaining momentum since your last crawl?

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with free APIs, so there are no ongoing costs.

Resources you used:

  • github-ai-crawler/ project folder on your Desktop (contains crawler.py, topics.yaml, .env, and the Python virtual environment).
  • output/ directory with generated Markdown files from the crawler.
  • second-brain Obsidian vault in your Documents folder (contains CLAUDE.md, raw/, wiki/, and .claude/commands/).
  • GitHub Fine-grained Personal Access Token (second-brain-crawler), set to expire in 90 days.
  • Installed tools: Obsidian, Claude Code.

Keep everything running

No action needed. Choose this if you plan to keep crawling new repos and building out your second brain.

  • Your second-brain vault stays in your Documents folder and opens in Obsidian anytime.
  • Your GitHub token remains active until its 90-day expiration. You can regenerate it from GitHub Settings when it expires.
  • Run python3 crawler.py anytime to fetch fresh results and copy them into your vault.

Pause - I'll come back to this later

Keep your files but deactivate your virtual environment to free up terminal resources.

  • Deactivate the Python virtual environment by running deactivate in your terminal.
  • Your github-ai-crawler/ folder and second-brain vault stay on disk. You can reactivate the virtual environment and re-run the crawler when you are ready.
  • Your GitHub token will remain active until its 90-day expiration date. No action needed there.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Delete the github-ai-crawler/ folder from your Desktop. This removes crawler.py, topics.yaml, the .env file, and the virtual environment.
  • Delete the second-brain folder from your Documents folder. This removes the entire Obsidian vault, including all wiki pages and crawled data.
  • Revoke your GitHub token. Go to github.com, click your profile picture, then click Settings. Click Developer settings, then Fine-grained tokens. Find second-brain-crawler and click Delete.
  • Optionally uninstall Obsidian and Claude Code if you no longer need them.

Mission accomplished

Nice Work!

You just built a personalized AI research pipeline that turns GitHub into a curated knowledge base inside your second brain. Instead of manually browsing for new tools and frameworks, you now have a repeatable system that discovers, enriches, and organizes open source AI projects for you.

Here's what you accomplished:

  • Built a Python crawler that searches the GitHub REST API across multiple AI-focused topic categories and enriches each repo with full metadata and README excerpts.
  • Set up an Obsidian vault with the raw/wiki architecture and a CLAUDE.md schema that teaches Claude Code your vault's conventions and page structure.
  • Created a custom /discover slash command that compiles crawled repos into cross-linked wiki pages with concept pages, project pages, and comparison tables.
  • Secret Mission: Added a freshness radar that tracks star growth between crawl runs and surfaces fast-rising repos in your wiki.

Ready to quiz yourself?