Build a Local AI Second Brain

Build a self-maintaining knowledge base with Ollama and Obsidian.

Introduction

30 Second Summary

Every article you read, video you watch, and note you jot down adds to your knowledge. But without a system to connect those pieces, most of it fades within days.

In this project, you will build a local AI-powered second brain that reads your sources, writes structured wiki pages, creates cross-references, and maintains a searchable index. All powered by a free, private LLM running on your own machine.

What You'll Build

A self-maintaining personal knowledge base following Andrej Karpathy's LLM Wiki architecture, where a local AI agent in Obsidian reads raw sources and generates an interlinked wiki with semantic search and knowledge graphs.

By the end of this project, you'll have:

  • A structured Obsidian vault with the Karpathy 3-layer architecture powered entirely by a local LLM through Ollama.
  • A content ingestion pipeline that converts YouTube videos and web articles into structured, cross-linked wiki pages with zero cloud dependency.
  • Semantic search and a knowledge graph visualizing connections across your entire wiki.
  • Secret Mission: Set up automated health checks that detect contradictions, orphan pages, and missing links across your wiki.

Are there any prerequisites?

You need Obsidian and Ollama already installed on your Windows PC. You should also be comfortable running basic commands in PowerShell.

This project runs entirely on your local machine. No cloud accounts, API keys, or subscriptions required.

Before We Start

Before We Start...

Pull Models and Install Plugins

Your AI second brain needs two things to work: a language model that can read, reason, and write, and plugins that connect that model to your Obsidian vault.

In this step, you will pull two local models through Ollama and install the plugins that wire everything together inside Obsidian.

In this step, get ready to:
  • Pull two Ollama models for AI reasoning and semantic search.
  • Install four Obsidian plugins for agent-powered workflows.
  • Set up the Web Clipper browser extension for article ingestion.
Pull Ollama models

You need two models. The first handles reasoning and writing. The second creates embeddings for semantic search across your wiki.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Pull both models by running these commands:
ollama pull qwen2.5:7b
ollama pull nomic-embed-text

What are these models?

The qwen2.5:7b model is your reasoning engine. It reads sources, writes wiki pages, and maintains cross-references. It weighs about 4.7GB and fits comfortably in 6GB VRAM.

The nomic-embed-text model is a small embedding model (~274MB) that converts text into numeric vectors. This powers semantic search so you can find related notes by meaning, not just keywords.

  • Each pull may take a few minutes depending on your internet speed. Wait for both to finish before continuing.
  • Verify both models are available by running:
ollama list
  • You should see both qwen2.5:7b and nomic-embed-text listed in the output.

Don't see both models listed?

  • If the pull failed with a network error, check your internet connection and try the command again. Ollama resumes partial downloads.
  • If you see a disk space error, you need at least 5GB free for both models. Free up space and retry.
  • Help me troubleshoot why my Ollama model pull failed on Windows.

Now you need to configure Ollama so that Obsidian plugins can communicate with it.

Why is this extra configuration needed?

By default, Ollama only accepts requests from its own process. Obsidian plugins run inside the Obsidian app, which has a different origin. Setting the OLLAMA_ORIGINS environment variable tells Ollama to also accept requests from app://obsidian.md so the plugins can connect.

  • If Ollama is running in the background (you will see its icon in the system tray near the clock), right-click the icon and select Quit Ollama.
  • Back in your PowerShell window, set the environment variable and start Ollama by running:
$env:OLLAMA_ORIGINS="app://obsidian.md*"
ollama serve
  • You should see log output appear in the terminal, indicating that Ollama is running and listening for requests.
  • Leave this terminal window open. Ollama needs to keep running for the plugins to work.

Make this permanent

The environment variable you just set only lasts for this terminal session. To make it permanent, open System Properties (search for "environment variables" in the Start menu), click Environment Variables, and add a new user variable named OLLAMA_ORIGINS with the value app://obsidian.md*. Then Ollama will always accept Obsidian connections, even when started from the system tray.

Seeing a port-in-use error?

  • If you see a message about port 11434 already being in use, Ollama is still running in the background. Right-click the Ollama icon in your system tray and select Quit Ollama, then run the commands above again.
  • If there is no tray icon, open Task Manager and end any ollama processes before retrying.
  • Help me fix a port conflict when starting Ollama serve on Windows.
Install Obsidian plugins

You need four plugins. Two are available from the community directory. One requires a manual install from GitHub. The fourth is a browser extension covered in the next substep.

  • Open Obsidian.
  • Click the gear icon in the bottom-left corner to open Settings.
  • In the left sidebar, click Community plugins.
  • If Restricted mode is on, click Turn on community plugins and confirm.
  • Click Browse to open the plugin directory.

Install the first plugin:

  • Search for Smart Connections in the search bar.
  • Click Install, then click Enable.

What is Smart Connections?

Smart Connections provides semantic search across your vault using a built-in local embedding model. It requires zero API keys and zero configuration for its core features. When you open a note, it shows you semantically related notes ranked by meaning.

Install the second plugin:

  • Back in the plugin directory, search for TubeSage.
  • Click Install, then click Enable.

What is TubeSage?

TubeSage extracts transcripts from YouTube videos and uses an LLM to generate structured summaries with timestamped links back to the video. It connects directly to Ollama for local processing.

Now install Note Agent manually. This plugin is not yet in the community directory, so you will download it from GitHub.

  • Open your browser and go to https://github.com/n4s5ti/obsidian-note-agent/releases.
  • From the latest release, download three files: main.js, manifest.json, and styles.css.
  • In Obsidian, right-click any file in the left file explorer panel and select Reveal in File Explorer (Windows) or Reveal in Finder (macOS). This opens your vault folder.
  • Navigate into the .obsidian folder, then into plugins.
  • Create a new folder called obsidian-note-agent.
  • Copy the three downloaded files (main.js, manifest.json, styles.css) into the obsidian-note-agent folder.
  • Back in Obsidian, press Cmd+R (macOS) or Ctrl+R (Windows) to reload.
  • Go to Settings > Community plugins.
  • Find Smart Note Agent in the list and toggle it on to enable it.

What is Note Agent?

Note Agent gives a local LLM full read and write access to your vault. It can create files, update content, search across notes, and maintain cross-references. It supports three modes: Ask (read-only), Edit (write with your approval), and Scheduled (automated background tasks).

Don't see Smart Note Agent after reloading?

  • Double-check that all three files are inside .obsidian/plugins/obsidian-note-agent/ and not in a nested subfolder.
  • Make sure the folder is named exactly obsidian-note-agent with no extra spaces or characters.
  • Try closing Obsidian completely and reopening it instead of just reloading.
  • Help me troubleshoot installing a manual Obsidian plugin from GitHub.
Install Obsidian Web Clipper

The Web Clipper is a browser extension that saves web articles directly into your vault as clean Markdown. This is how you will feed articles into your knowledge base.

  • Open your browser (Chrome, Edge, Brave, or Firefox).
  • Go to your browser's extension store and search for Obsidian Web Clipper.
  • Click Add to browser (the exact button text varies by browser) and confirm the installation.
  • After installing, click the Web Clipper icon in your browser toolbar to open its settings.
  • Set the default vault name to match your Obsidian vault name exactly.
  • You should see the Web Clipper icon in your browser toolbar, confirming it is installed and ready.

Can't find the extension?

  • For Chrome, Edge, or Brave, search in the Chrome Web Store.
  • For Firefox, search in Firefox Add-ons.
  • Make sure you are searching for the official Obsidian Web Clipper by the Obsidian team, not a third-party alternative.

Your toolchain is ready. Next up, you will build the vault architecture that turns these tools into a self-maintaining knowledge base.

Build the Karpathy Vault Architecture

Your models are pulled and your plugins are ready. Now you need a place for all that intelligence to operate on.

Andrej Karpathy's LLM Wiki pattern splits a knowledge base into three layers: raw sources (immutable), AI-maintained wiki pages (generated and cross-linked), and a schema that governs how the AI behaves. In this step, you will build that architecture inside your Obsidian vault.

In this step, get ready to:
  • Create the 3-layer folder structure for raw sources and wiki pages.
  • Create the index and log files that track wiki activity.
  • Write the AGENTS.md schema that teaches the LLM how to maintain your wiki.
Create the 3-layer folder structure

The Karpathy architecture separates your vault into two top-level areas. The raw/ folder holds immutable source material that you (the human) curate. The wiki/ folder is where the AI writes and maintains all generated pages.

  • In the Obsidian left sidebar, look for the File Explorer panel (the folder icon at the top).
  • Right-click in the empty space of the File Explorer and select New folder.
  • Name the folder raw and press Enter.
  • Right-click the raw folder and select New folder to create four subfolders inside it: articles, videos, notes, and assets.
  • Right-click in the empty space of the File Explorer again and create another top-level folder called wiki.
  • Right-click the wiki folder and create four subfolders inside it: sources, concepts, entities, and synthesis.

Prefer PowerShell?

If you know the path to your vault on disk, you can create all the folders at once by running these commands in PowerShell from your vault's root directory:

mkdir raw, raw/articles, raw/videos, raw/notes, raw/assets
mkdir wiki, wiki/sources, wiki/concepts, wiki/entities, wiki/synthesis

Switch back to Obsidian afterwards and the folders will appear in the File Explorer.

What do these folders do?

The raw/ subfolders separate your source material by type: web articles, YouTube transcripts, manual notes, and images. The AI never edits anything here.

The wiki/ subfolders organize the AI's output: one page per source summary, one page per concept, one page per person or tool, and cross-cutting synthesis pages that connect multiple sources together.

Your File Explorer should now show raw/ with 4 subfolders and wiki/ with 4 subfolders.

Folders not showing up?

If you created folders via PowerShell but they are not visible in Obsidian, press Ctrl+R to reload the app. Obsidian picks up file system changes on reload.

If you accidentally created a folder in the wrong location, right-click it in the File Explorer and select Delete, then try again.

Why aren't my vault folders appearing in Obsidian?

Create the index and log files

Every wiki needs bookkeeping. The index tracks what pages exist. The log records every operation the AI performs so you can audit its work.

  • Right-click the wiki folder in the File Explorer and select New note.
  • Name the note index (Obsidian adds the .md extension automatically).
  • Paste the following content into wiki/index.md:
# Wiki Index

## Sources
(none yet)

## Concepts
(none yet)

## Entities
(none yet)

## Synthesis
(none yet)
  • Right-click the wiki folder again and create another new note called log.
  • Paste the following content into wiki/log.md:
# Activity Log

All ingest, query, and lint operations are logged here.

What do these files do?

The index.md file is the master table of contents. Every time the AI creates a new wiki page, it adds an entry here. This gives the AI a quick way to see what already exists without scanning every file.

The log.md file is an audit trail. Each ingest, query, or lint operation gets a timestamped record so you can trace what the AI did and when.

Verify that both files appear inside wiki/ in your File Explorer. Clicking each one should show the content you just pasted.

Write the AGENTS.md schema

This is the most important file in your vault. It is a single document that turns a generic LLM into a disciplined wiki maintainer by defining roles, directory rules, and three core workflows.

Why is this called AGENTS.md?

Karpathy's original pattern uses a CLAUDE.md file to instruct Claude Code. Since you are using a local Ollama model through the Note Agent plugin instead, this file is named AGENTS.md. The concept is identical: a schema file the AI reads to understand how to behave.

  • Right-click in the empty space of the File Explorer (at the vault root level, not inside raw/ or wiki/) and select New note.
  • Name the note AGENTS.
  • Delete any default content Obsidian inserts, then paste the full schema below:
# Second Brain Schema

## Role

You are the wiki maintainer for this knowledge base. You read raw sources, write structured wiki pages, maintain cross-references, and keep the index current. The human curates sources and asks questions. You do the grunt work: summarizing, cross-referencing, filing, and bookkeeping.

## Directory Structure

- raw/ - Immutable source material. Never modify files here.
  - raw/articles/ - Web articles clipped via Obsidian Web Clipper
  - raw/videos/ - YouTube notes generated by TubeSage
  - raw/notes/ - Manual notes, PDFs, transcripts
  - raw/assets/ - Images and attachments
- wiki/ - AI-maintained knowledge base. You write and maintain everything here.
  - wiki/sources/ - One summary page per raw source
  - wiki/concepts/ - One page per key concept
  - wiki/entities/ - One page per person, tool, or organization
  - wiki/synthesis/ - Cross-cutting analysis and overviews
- wiki/index.md - Master index of all wiki pages
- wiki/log.md - Activity log of all operations
- AGENTS.md - This file. The schema that governs your behavior.

## Frontmatter Format

Every wiki page must have YAML frontmatter:

---
title: Page Title
type: source | concept | entity | synthesis
created: YYYY-MM-DD
last_updated: YYYY-MM-DD
source_files: ["raw/articles/example.md"]
related: ["wiki/concepts/related-topic.md"]
tags: [relevant, tags]
---

## Ingest Workflow

When asked to ingest a source from raw/:

1. Read the source file completely before writing anything.
2. Create a summary page in wiki/sources/ named after the source slug.
3. Identify 3-8 key concepts. For each: find an existing wiki/concepts/ page or create a new one.
4. Identify people, tools, or organizations mentioned. For each: find an existing wiki/entities/ page or create one.
5. Add cross-references using [[wikilinks]]. Every concept page should link to related concepts and the source it came from.
6. Add backlinks: update the related: frontmatter field on every page you reference.
7. Update wiki/index.md with new entries under the appropriate section.
8. Append to wiki/log.md: ## [YYYY-MM-DD] ingest | source-filename | pages created/updated: [list]

## Query Workflow

When asked a question:

1. Read wiki/index.md to identify relevant pages.
2. Read those pages. Synthesize an answer with [[wikilink]] citations.
3. If the answer reveals a new insight worth preserving, suggest: "Save this as a wiki page?"
4. If approved, write to wiki/synthesis/ and update the index.

## Lint Workflow

When asked to lint or health-check:

1. Scan all files in wiki/.
2. Report:
   - Orphan pages (no incoming [[wikilinks]] from other wiki pages)
   - Dead links ([[wikilinks]] pointing to files that do not exist)
   - Contradictions (same claim with different values across pages)
   - Stale index entries (listed in index.md but file missing, or file exists but not in index)
   - Concepts mentioned in multiple sources but lacking their own page
3. Write the report to wiki/maintenance/lint-YYYY-MM-DD.md.

## Conventions

- Use [[wikilinks]] for all internal references.
- Cite source files as [[raw/articles/filename]] or [[raw/videos/filename]].
- Use kebab-case for filenames: my-concept-name.md
- Keep summaries concise: 200-400 words per source page.
- Concept pages should define the concept in the first paragraph, then list related context.
- Entity pages should state what/who it is, why it matters, and link to sources where it appears.
- Never modify files in raw/. They are immutable source truth.
- Always update the index and log after any write operation.

What does this schema do?

The schema defines three workflows that the AI follows when you give it instructions:

  • Ingest reads a raw source, creates wiki pages for concepts and entities, adds cross-links, and updates the index.
  • Query answers questions by reading existing wiki pages, and optionally saves valuable answers as new synthesis pages.
  • Lint scans the wiki for problems: orphan pages, dead links, contradictions, and stale index entries.

The Conventions section enforces consistency across the wiki: kebab-case filenames, wikilinks for all references, and immutable raw sources.

Confirm your AGENTS.md appears at the vault root (not inside raw/ or wiki/). Clicking it should show the full schema with all three workflows visible when you scroll through.

Schema not rendering correctly?

If you see raw Markdown syntax instead of formatted headings and lists, make sure you are in Reading view (click the book icon in the top-right corner of the note). Editing view shows the raw Markdown, which is expected.

If the file was accidentally created inside a subfolder, drag it to the vault root in the File Explorer.

Help me fix my AGENTS.md file placement or formatting.

✔️ Awesome, I've got everything!

Great. Double check you can see AGENTS.md at the vault root, wiki/index.md with four section headings, and wiki/log.md with the Activity Log header.

ⓧ I'd like to double check the full code

Here are all three files. Compare your vault contents against these:

# Second Brain Schema

## Role

You are the wiki maintainer for this knowledge base. You read raw sources, write structured wiki pages, maintain cross-references, and keep the index current. The human curates sources and asks questions. You do the grunt work: summarizing, cross-referencing, filing, and bookkeeping.

## Directory Structure

- raw/ - Immutable source material. Never modify files here.
  - raw/articles/ - Web articles clipped via Obsidian Web Clipper
  - raw/videos/ - YouTube notes generated by TubeSage
  - raw/notes/ - Manual notes, PDFs, transcripts
  - raw/assets/ - Images and attachments
- wiki/ - AI-maintained knowledge base. You write and maintain everything here.
  - wiki/sources/ - One summary page per raw source
  - wiki/concepts/ - One page per key concept
  - wiki/entities/ - One page per person, tool, or organization
  - wiki/synthesis/ - Cross-cutting analysis and overviews
- wiki/index.md - Master index of all wiki pages
- wiki/log.md - Activity log of all operations
- AGENTS.md - This file. The schema that governs your behavior.

## Frontmatter Format

Every wiki page must have YAML frontmatter:

---
title: Page Title
type: source | concept | entity | synthesis
created: YYYY-MM-DD
last_updated: YYYY-MM-DD
source_files: ["raw/articles/example.md"]
related: ["wiki/concepts/related-topic.md"]
tags: [relevant, tags]
---

## Ingest Workflow

When asked to ingest a source from raw/:

1. Read the source file completely before writing anything.
2. Create a summary page in wiki/sources/ named after the source slug.
3. Identify 3-8 key concepts. For each: find an existing wiki/concepts/ page or create a new one.
4. Identify people, tools, or organizations mentioned. For each: find an existing wiki/entities/ page or create one.
5. Add cross-references using [[wikilinks]]. Every concept page should link to related concepts and the source it came from.
6. Add backlinks: update the related: frontmatter field on every page you reference.
7. Update wiki/index.md with new entries under the appropriate section.
8. Append to wiki/log.md: ## [YYYY-MM-DD] ingest | source-filename | pages created/updated: [list]

## Query Workflow

When asked a question:

1. Read wiki/index.md to identify relevant pages.
2. Read those pages. Synthesize an answer with [[wikilink]] citations.
3. If the answer reveals a new insight worth preserving, suggest: "Save this as a wiki page?"
4. If approved, write to wiki/synthesis/ and update the index.

## Lint Workflow

When asked to lint or health-check:

1. Scan all files in wiki/.
2. Report:
   - Orphan pages (no incoming [[wikilinks]] from other wiki pages)
   - Dead links ([[wikilinks]] pointing to files that do not exist)
   - Contradictions (same claim with different values across pages)
   - Stale index entries (listed in index.md but file missing, or file exists but not in index)
   - Concepts mentioned in multiple sources but lacking their own page
3. Write the report to wiki/maintenance/lint-YYYY-MM-DD.md.

## Conventions

- Use [[wikilinks]] for all internal references.
- Cite source files as [[raw/articles/filename]] or [[raw/videos/filename]].
- Use kebab-case for filenames: my-concept-name.md
- Keep summaries concise: 200-400 words per source page.
- Concept pages should define the concept in the first paragraph, then list related context.
- Entity pages should state what/who it is, why it matters, and link to sources where it appears.
- Never modify files in raw/. They are immutable source truth.
- Always update the index and log after any write operation.
# Wiki Index

## Sources
(none yet)

## Concepts
(none yet)

## Entities
(none yet)

## Synthesis
(none yet)
# Activity Log

All ingest, query, and lint operations are logged here.

Your vault architecture is complete. Next up, you will wire the Note Agent plugin to Ollama and test that the AI can actually read your schema and follow its instructions.

Configure and Test the AI Agent

Your vault architecture is in place. You have raw source folders, wiki output folders, and a schema file that tells an AI exactly how to maintain everything.

Now you need to connect the AI to your vault and prove it can follow instructions. In this step, you will wire Note Agent to your local Ollama instance, verify it can read your schema, and run a full ingest to generate your first wiki pages.

In this step, get ready to:
  • Wire Note Agent to Ollama with the correct provider, URL, model, and mode settings.
  • Test the connection by asking the agent to read and summarize your schema.
  • Perform your first ingest to generate wiki pages with cross-references.
Wire Note Agent to Ollama

Note Agent needs to know which LLM provider to use and how to reach it. Since Ollama is already serving on your machine, you just need to point the plugin at localhost.

  • In Obsidian, open Settings (click the gear icon in the bottom-left corner).
  • In the left sidebar, scroll down to Smart Note Agent under Community plugins.
  • Set Provider to Ollama.
  • Confirm Base URL is set to http://localhost:11434 (this is the default).
  • Set Model to qwen2.5:7b.
  • Set Mode to Edit.

Why Edit mode?

Edit mode gives the agent write access to your vault, but only with your explicit approval. When the agent wants to create or change a file, it shows you a unified diff first. You review and click Approve before anything is written to disk.

This matches the Karpathy pattern where the LLM proposes changes and the human curates. You stay in control while the AI does the heavy lifting.

Close the Settings window. Your Note Agent is now configured.

Test the connection and first interaction

Before trusting the agent with real work, you need to confirm it can connect to Ollama and read files from your vault.

  • Click the bot icon in the left ribbon to open the Note Agent chat panel.
  • If you do not see the bot icon, press Ctrl+P to open the Command Palette and type Open Note Agent.

You should see a chat panel open on the right side of your Obsidian window.

  • Type the following message into the Note Agent chat and press Enter:
Read the file AGENTS.md and confirm you understand the wiki structure and your responsibilities. Summarize the three core workflows.

Wait for the agent to respond. This may take 10-30 seconds depending on your hardware.

What should I see?

The agent should read your AGENTS.md file and produce a summary that mentions the three workflows: Ingest (processing raw sources into wiki pages), Query (answering questions from wiki content), and Lint (health-checking for orphan pages and dead links).

If the agent summarizes all three workflows correctly, your connection is working and the model understands your schema.

Agent not responding or showing an error?

  • Make sure Ollama is still running. Check your system tray for the Ollama icon, or switch to your PowerShell window and confirm ollama serve is active.
  • Verify the OLLAMA_ORIGINS environment variable was set before starting Ollama. Without it, Obsidian plugins cannot connect due to CORS restrictions.
  • Double-check that the Base URL in Smart Note Agent settings matches http://localhost:11434 exactly.

Still stuck? Help me troubleshoot my Note Agent connection to Ollama.

Perform your first ingest

The agent can read your schema. Now put it to work. You need a raw source file for the agent to process through the Ingest workflow.

  • In Obsidian's file explorer on the left, right-click the raw/notes folder.
  • Click New note.
  • Name the file test-source.
  • Write a paragraph or two about any topic you are genuinely interested in. This could be a concept you recently learned, a summary of a podcast episode, or notes from a meeting.

Pick something real

The more substantive your test source, the better the wiki output. A few sentences about a single concept gives the agent enough material to generate meaningful wiki pages with actual cross-references.

  • Save the file (Ctrl+S).
  • Switch to the Note Agent chat panel and type the following message:
Following the Ingest workflow in AGENTS.md, process the file raw/notes/test-source.md. Create appropriate wiki pages in wiki/sources/ and wiki/concepts/, add cross-links, update wiki/index.md, and append to wiki/log.md.
  • Press Enter to send the message.

The agent will read your test source, then propose file creations as unified diffs. Each proposed change appears as a diff you can review.

  • Review each proposed change and click Approve to allow the agent to write it to your vault.

What does the agent create?

  • A source summary page in wiki/sources/ named after your file (e.g. wiki/sources/test-source.md).
  • One or more concept pages in wiki/concepts/ for key ideas in your source.
  • Updated wiki/index.md with new entries under the Sources and Concepts sections.
  • A timestamped log entry in wiki/log.md recording what was created.

Every generated page should contain [[wikilinks]] that cross-reference other pages. This is the linking behavior that makes the knowledge base grow smarter with each source you add.

  • Open wiki/index.md in the file explorer. Confirm it now lists entries under the Sources and Concepts sections.
  • Open wiki/log.md. Confirm it has a new timestamped entry recording the ingest operation.
  • Open one of the concept pages in wiki/concepts/. Confirm it has YAML frontmatter and [[wikilinks]] pointing back to the source page.

Agent created files but they are missing wikilinks or frontmatter?

The model may not have followed every convention on the first try. Send a follow-up message asking it to fix the specific issue:

"Check the pages you just created. Make sure every wiki page has YAML frontmatter matching the format in AGENTS.md, and that all internal references use [[wikilinks]]."

If the agent consistently ignores parts of the schema, help me get my AI agent to follow the AGENTS.md schema more closely.

Your AI agent is wired up and proven. It can read raw sources, generate structured wiki pages, maintain cross-references, and keep the index current. Next up, you will feed it real content from YouTube videos and web articles.

Set Up YouTube and Web Ingestion

Your AI agent can read files and write wiki pages. Now you need to feed it real-world knowledge.

In Karpathy's LLM Wiki pattern, the wiki only becomes powerful when you have diverse sources flowing in. In this step, you will configure two ingestion pipelines that turn YouTube videos and web articles into structured raw material for the agent to process.

In this step, get ready to:
  • Configure TubeSage to generate structured notes from YouTube videos using Ollama.
  • Clip a web article into your vault using Obsidian Web Clipper.
  • Process both sources through Note Agent to populate your wiki with cross-linked pages.
Configure TubeSage for YouTube ingestion

TubeSage extracts transcripts from YouTube videos and uses your local LLM to generate structured summaries. You need to point it at your Ollama server so all processing stays on your machine.

  • Open Obsidian Settings.
  • Scroll down in the left sidebar and click TubeSage.
  • Accept the MIT license terms if prompted.
  • Under the Language model heading, find the Provider dropdown.
  • Select Ollama from the dropdown.
  • Enter http://localhost:11434 in the Ollama server URL field.
  • Select qwen2.5:7b as the model (or type it in if a dropdown appears).
  • Set the output folder to raw/videos.
  • Enable Add YouTube Timestamp Links so generated notes include clickable links to video moments.
  • Set Default Summary Mode to Extensive for detailed notes.

What do these settings do?

The Provider and server URL tell TubeSage to route all AI calls to your local Ollama instance. No data leaves your machine.

The Extensive summary mode generates longer, more detailed notes compared to Fast mode. Timestamp links let you click directly to the relevant moment in the video from inside your notes.

Now test TubeSage with a real video.

  • Click the TubeSage icon in the left ribbon (or open the Command Palette with Ctrl+P and search for TubeSage).
  • Paste any YouTube video URL into the prompt.
  • Confirm the output folder is raw/videos and run processing.

What should I see?

TubeSage will take a minute or two to extract the transcript and generate the summary. When it finishes, a new Markdown file appears in raw/videos/ containing video metadata (title, channel, date), an AI-generated summary, key points with timestamped links back to the video, and transcript sections.

  • Open the file explorer in Obsidian and navigate to raw/videos/.
  • Open the newly created note and confirm it has metadata, a summary, and timestamped key points.

TubeSage not generating a summary?

Make sure Ollama is still running in your PowerShell window. If you closed it, restart with $env:OLLAMA_ORIGINS="app://obsidian.md*" followed by ollama serve.

If the transcript extraction fails, try a different YouTube video. Some videos have disabled captions or region-restricted transcripts.

Help me troubleshoot TubeSage not generating summaries with Ollama.

Clip a web article with Web Clipper

Obsidian Web Clipper captures web articles as clean Markdown and saves them directly into your vault. You set the destination folder each time you clip.

  • Open your browser and navigate to any article you find interesting (a blog post, news article, or tutorial).
  • Click the Obsidian Web Clipper icon in your browser toolbar.
  • In the Footer area of the clipper popup, select your vault from the Vault dropdown.
  • In the Folder field, type raw/articles.
  • Click Add to Obsidian.

What does Web Clipper do?

Web Clipper extracts the main article content as Markdown, stripping out navigation, ads, and sidebars. Images are linked to their web URLs rather than downloaded. The result is a clean source file ready for your AI agent to process.

  • Switch back to Obsidian and open the file explorer.
  • Navigate to raw/articles/ and confirm your clipped article appears as a Markdown file.

Article not appearing in your vault?

Check that the vault name in Web Clipper settings exactly matches your Obsidian vault name (not the file path). Also verify the folder name is raw/articles with no leading slash.

If Obsidian opens but the note is empty, try selecting the article text first (Ctrl+A) before clipping.

Help me fix Web Clipper not saving articles to my vault.

Process both sources through the wiki agent

You now have raw source material from two different formats sitting in your vault. The final step is to run each through the Ingest workflow so your wiki grows with structured, cross-linked pages.

Start with the web article. In the Note Agent chat panel, send this prompt:

  • Tell the agent to ingest the web article by pasting this prompt into the Note Agent chat:
Ingest the new article at raw/articles/[[ARTICLE_FILENAME="your-article-filename"]].md following the Ingest workflow in AGENTS.md. Extract key concepts, create wiki pages, and update the index.

What does this prompt do?

The agent reads your raw article, then follows the Ingest workflow from AGENTS.md. It creates a source summary in wiki/sources/, identifies key concepts and creates pages in wiki/concepts/, adds cross-references using [[wikilinks]], and updates wiki/index.md and wiki/log.md.

  • Review each proposed change the agent shows as a diff.
  • Click Approve for each file creation or modification.

Now process the YouTube notes. Send this prompt to the agent:

  • Tell the agent to ingest the YouTube source by pasting this prompt:
Ingest the YouTube notes at raw/videos/[[VIDEO_FILENAME="your-video-filename"]].md following the Ingest workflow in AGENTS.md. Extract concepts, identify people/tools mentioned, create entity pages, and cross-link with existing wiki pages.

Why mention entities specifically?

YouTube videos often mention specific people, tools, or organizations. By explicitly asking the agent to create entity pages in wiki/entities/, you get richer cross-references. These entity pages link back to every source where the person or tool appears.

  • Review and approve each proposed change.
  • Once all changes are approved, open wiki/index.md and confirm it now lists entries under Sources, Concepts, and Entities.
  • Open wiki/log.md and verify new timestamped ingest records appear for both sources.
  • Browse the concept pages in wiki/concepts/ and check for [[wikilinks]] that connect shared concepts across both sources.

Agent creating files in the wrong location?

If the agent writes pages outside the wiki/ folder, remind it to re-read AGENTS.md before processing. The schema defines that all generated content belongs in wiki/ subfolders.

If the agent is not finding existing concept pages to cross-reference, ask it to run list_files on wiki/concepts/ first to see what already exists.

Help me get the agent to follow the AGENTS.md schema correctly.

Your wiki is growing. You now have multiple sources feeding into an interconnected knowledge base, with the AI agent handling all the grunt work of summarizing, filing, and cross-linking. Next up, you will explore this web of knowledge using semantic search and Obsidian's graph view.

Explore the Knowledge Graph and Semantic Search

Your wiki now has pages generated from three different sources, all cross-linked with wikilinks. But navigating a growing wiki by clicking through individual pages is slow. You need ways to discover connections you didn't explicitly create.

In this step, you'll activate two discovery tools. Semantic search finds related notes by meaning, not just keyword matches. The knowledge graph visualizes how your wiki pages connect as a network. You'll also use the AI agent to synthesize insights across sources and save the results as permanent wiki pages.

In this step, get ready to:
  • Configure Smart Connections for semantic search with local embeddings.
  • Explore the knowledge graph to visualize wiki connections.
  • Query the wiki and save a synthesis page from the results.
Configure Smart Connections for semantic search

Smart Connections ships with a built-in local embedding model that works immediately with zero configuration. No API key is needed for core features. Once enabled, it indexes your vault automatically using local embeddings that never leave your machine.

  • Open Settings > Smart Connections.
  • Confirm the plugin is enabled. You do not need to change any settings. The default local embedding model handles everything.
  • Close settings and wait for the indexing notice to appear. Smart Connections begins embedding your vault files in the background.

What is happening during indexing?

Smart Connections reads each note in your vault and converts it into a numerical representation (an embedding) that captures its meaning. When you browse a note later, the plugin compares its embedding to every other note and ranks them by similarity.

This all runs locally on your machine. No data is sent to any server.

  • Once indexing is complete, click the Smart Connections icon in the left ribbon to open the Connections view sidebar.
  • Open any wiki page (for example, one of your concept pages in wiki/concepts/).

The sidebar now shows semantically related notes ranked by meaning. Pages that discuss similar ideas appear at the top, even if they don't share exact keywords or explicit wikilinks.

Not seeing related notes in the sidebar?

  • Make sure indexing has finished. If you see a progress indicator, wait for it to complete before checking results.
  • Confirm you have a note open in the editor. The Connections sidebar only shows results for the currently active note.
  • Try closing and reopening the Connections sidebar from the left ribbon icon.

Help me troubleshoot why Smart Connections is not showing related notes in my Obsidian vault.

Explore the knowledge graph

Obsidian's built-in Graph View renders every note and wikilink as a visual network. This is where you can see the shape of your wiki: which pages are hubs, which are orphans, and how concepts cluster together.

  • Open Graph View by pressing Cmd+G (macOS) or Ctrl+G (Windows).

You should see your wiki taking shape as a network. Source summary pages link to concept pages, concept pages link to each other and to entity pages, and the index acts as a hub node connecting everything.

  • In the Graph View controls panel on the left, turn on Show orphans to spot any disconnected pages.
  • In the Search filter at the top of the controls, type path:wiki/ to show only the AI-generated knowledge pages.
  • Increase the Link distance slider for better readability between nodes.

What am I looking at?

Each dot is a note in your vault. Each line is a wikilink between two notes. The path:wiki/ filter removes the raw source files so you see only the wiki layer your AI agent maintains.

Pages with many connections appear as larger nodes. Your wiki/index.md will likely be the biggest hub since every page is listed there.

Graph View looks empty or has no connections?

  • Make sure your wiki pages use [[wikilinks]] syntax, not plain Markdown links. Obsidian's Graph View only recognizes wikilinks.
  • Remove the path:wiki/ filter temporarily to confirm the graph works with all vault files visible.
  • If you see nodes but no lines, check that your concept and source pages have [[wikilinks]] pointing to other wiki pages.

Help me figure out why my Obsidian Graph View is not showing connections between my wiki pages.

Query the wiki and create a synthesis page

So far the AI agent has been writing pages from individual sources. Now you'll ask it a cross-cutting question that synthesizes knowledge from multiple wiki pages. If the answer is valuable, you'll save it as a permanent synthesis page. This is the "filing loop" where valuable query answers become wiki pages, making the knowledge base grow through use.

  • Open Settings > Smart Note Agent.
  • Change Mode from Edit to Ask. This switches the agent to read-only mode for safe querying.
  • Close settings and return to the Note Agent chat panel.
  • Type the following prompt into the chat:
Based on the wiki pages, what are the key themes across all sources? Which concepts appear in multiple sources?

What does Ask mode do?

In Ask mode, the agent reads your wiki pages and synthesizes an answer with wikilink citations. It cannot create or modify files. This makes it safe to explore questions without worrying about accidental edits.

The agent reads your wiki pages and responds with a synthesis of key themes across your sources. It identifies which concepts appear in multiple places and how they connect.

  • Review the agent's response. If it surfaces a useful insight, you'll save it as a permanent wiki page.
  • Go back to Settings > Smart Note Agent and switch Mode back to Edit.
  • In the Note Agent chat, type the following prompt to save the analysis:
Save that analysis as a new synthesis page at wiki/synthesis/[topic]-overview.md and update wiki/index.md with the new entry under the Synthesis section. Append a query log entry to wiki/log.md.

What happens when I approve?

The agent proposes file creations as unified diffs, just like during ingestion. It creates a new synthesis page in wiki/synthesis/ with type: synthesis frontmatter, adds an entry to wiki/index.md under the Synthesis section, and logs the operation in wiki/log.md.

This is the Query Workflow defined in your AGENTS.md schema coming to life. Every valuable answer becomes a permanent part of your knowledge base.

  • Review the proposed changes and click Approve for each diff.
  • Open wiki/index.md and confirm the Synthesis section now has an entry.
  • Open wiki/log.md and confirm a new query log entry has been appended.

Agent not proposing file changes?

  • Make sure you switched back to Edit mode. In Ask mode, the agent cannot write files.
  • Check that Ollama is still running in your terminal. If the terminal was closed, the model connection drops.
  • Try rephrasing your prompt to explicitly name the file path and mention updating the index.

Help me get Note Agent to write a synthesis page to my wiki.

Your wiki now has three discovery layers: wikilinks for explicit connections, semantic search for meaning-based similarity, and the knowledge graph for visual exploration. Next up, you'll take on a secret mission to set up automated health checks that keep your wiki consistent as it grows.

Secret mission

Automated Wiki Health Checks

Your wiki is growing, but how do you know it's healthy? Pages can become orphaned, links can break, and the index can drift out of sync. In this secret mission, you'll set up automated lint checks that catch these problems before they snowball.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama models (qwen2.5:7b and nomic-embed-text, ~5GB total disk space).
  • Obsidian vault files (raw/, wiki/, and AGENTS.md).
  • Obsidian plugins (Smart Connections, TubeSage, Note Agent).
  • Web Clipper browser extension.

Keep everything running

No action needed. Choose this if you are still actively building your knowledge base and want Ollama ready to ingest new sources at any time.

  • Ollama continues serving in the background via the system tray icon.
  • Your vault, wiki pages, and all cross-references persist on disk.
  • The scheduled lint task runs daily as configured.
  • There are no cloud services, subscriptions, or API keys to worry about.

Pause - I'll come back to this later

Shut down Ollama to free up system memory. Your vault and all wiki files persist on disk until you are ready to return.

  • Right-click the Ollama icon in your system tray and select Quit.
  • Your Obsidian vault, plugins, and all generated wiki pages remain on disk.
  • When you want to resume, relaunch Ollama from the Start menu and your setup is exactly as you left it.

Delete - I don't want to use this again

Remove all project files and plugins for a completely clean slate. There is no cost reason to delete anything. This is purely for tidiness.

  • In Obsidian, right-click the raw/ folder in the file explorer and select Delete folder. Repeat for the wiki/ folder and the AGENTS.md file.
  • To uninstall the plugins, open Settings > Community plugins. Click the trash icon next to Smart Connections, TubeSage, and Smart Note Agent to remove each one.
  • To remove Web Clipper, open your browser's extension management page and uninstall Obsidian Web Clipper.
  • To reclaim the ~5GB of disk space used by Ollama models, consult the Ollama documentation for model management commands.

Nice Work!

Nice Work!

Nice work! You have built a fully local, self-maintaining knowledge base that turns raw sources into an interconnected wiki. No cloud services, no API costs, no data leaving your machine.

You've learned how to:

  • Build a 3-layer knowledge base following Karpathy's LLM Wiki architecture, powered entirely by a local LLM through Ollama.
  • Configure an agentic LLM with Note Agent that reads raw sources, writes structured wiki pages, creates cross-references, and maintains an index with your approval.
  • Set up ingestion pipelines using TubeSage and Obsidian Web Clipper that convert YouTube videos and web articles into cross-linked wiki pages with semantic search and a knowledge graph showing how your ideas connect.
  • Secret Mission: Configured automated health checks that detect contradictions, orphan pages, dead links, and index drift across your wiki.

Ready to quiz yourself?