AST-Aware Code Ingestion Pipeline
Parse any Git repo into structured function-level chunks using tree-sitter.
Introduction
30 Second Summary
When AI coding assistants suggest completions or answer questions about your codebase, they need to find the right piece of code to reference. Most tools split code into fixed-size token windows, like tearing pages out of a book at arbitrary intervals. Functions get cut in half, unrelated code gets merged together, and the AI gives confused responses.
In this project, you will build a command-line tool that parses Git repositories using Abstract Syntax Trees to extract every function and class at its exact boundaries. Instead of guessing where code starts and ends, your tool will know.
What You'll Build
A Python CLI tool that ingests any Git repository and produces structured JSON output with function-level and class-level chunks, each captured at precise AST node boundaries using tree-sitter.
By the end of this project, you'll have:
- An exploratory script that visualizes how tree-sitter sees your code as a syntax tree, revealing the exact node types and line spans for every function and class.
- A production ingestion module that walks entire repositories, parses files across multiple languages, and outputs structured JSON chunks with repo name, file path, line range, symbol name, and full body text.
- Real output from ingesting the httpx open-source repository, proving your tool handles hundreds of files with decorators, nested functions, and class hierarchies.
- Secret Mission: Add Go language support, handle decorated functions, and extract TypeScript arrow functions assigned to variables.
Are there any prerequisites?
You should be comfortable with Python fundamentals (functions, classes, file I/O, and command-line arguments) and basic Git operations like cloning a repository. This is Project 1 of a 7-part capstone series on building a RAG-over-Codebase system. The ingest.py module you build here becomes a direct dependency of every subsequent project.
Set Up Your Environment
Before you can parse any code, you need the tools that understand code structure.
In this step, you'll install the tree-sitter parsing library, create your project folder, and clone a real open-source repository to use as test data.
In this step, get ready to:
- Install tree-sitter and language grammar packages.
- Create the project folder with empty starter files.
- Clone the httpx repository as your test corpus.
Install tree-sitter packages with pip
Tree-sitter is a parser generator that builds concrete syntax trees for source code. Each language needs its own grammar package, so you'll install the core library plus the Python and TypeScript grammars.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
- Install the tree-sitter packages by running this command:
pip install tree-sitter==0.25.2 tree-sitter-python==0.25.0 tree-sitter-typescript==0.23.2What are these three packages?
tree-sitter is the core parsing engine. It knows how to walk syntax trees, but doesn't know any specific language on its own.
tree-sitter-python and tree-sitter-typescript are grammar packages that teach tree-sitter how to parse those specific languages. Tree-sitter supports 100+ language grammars, all following this same plugin pattern.
- Verify the installation succeeded by running:
python -c "import tree_sitter; import tree_sitter_python; import tree_sitter_typescript; print('All packages installed successfully')"You should see All packages installed successfully printed in the terminal.
Seeing 'pip: command not found'?
Try pip3 instead of pip. On some systems, pip points to Python 2 while pip3 points to your Python 3 installation.
If that still fails, try python -m pip install instead. This ensures you're using the pip tied to the correct Python version.
Help me troubleshoot my pip installation
Create the project folder structure
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir ast-ingest
cd ast-ingest- Create the empty starter files by running this command:
macOS
touch explore_ast.py ingest.py sample.tsWindows
New-Item explore_ast.py, ingest.py, sample.ts -ItemType File- Confirm the files exist by running:
lsYou should see explore_ast.py, ingest.py, and sample.ts in the output.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Visual Studio Code and press Enter to open it.
- In VS Code, click File in the top menu bar.
- Click Open Folder.
- Select the ast-ingest folder on your Desktop, then click Open.
You should see your three empty files listed in the Explorer sidebar on the left.
Clone the httpx repository as a test corpus
You need a real codebase to test your ingestion tool against. httpx is a well-maintained Python HTTP library with classes, decorators, and async functions. It's complex enough to be interesting but small enough to process in seconds.
- Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, usually above Tab).
- Clone the httpx repository into your project folder by running this command:
git clone https://github.com/encode/httpxWhy httpx?
httpx has around 200 Python files with a mix of classes, decorators, async functions, and nested definitions. That variety means your parser will face real-world complexity from day one.
- Confirm the clone succeeded by running:
ls httpxYou should see folders like httpx, tests, and files like pyproject.toml listed in the output.
Clone failing or timing out?
Make sure you have a working internet connection and that git is installed. Run git --version to check.
If you're behind a corporate firewall or VPN, try disconnecting temporarily. Some networks block Git's SSH and HTTPS protocols.
Help me troubleshoot cloning the httpx repo
Your environment is ready. Next up, you'll write a short exploration script that reveals exactly how tree-sitter sees your code as a tree of nodes.
Explore How Tree-Sitter Sees Code
Your environment is set up with tree-sitter installed and a test corpus ready to go. Now comes the fun part: seeing how a parser actually understands your code.
When you read a Python file, you see functions, classes, and indentation. Tree-sitter sees something different. It sees a tree where every syntactic element, from an entire class down to a single comma, is a node with a type, a position, and children. In this step, you'll make that invisible structure visible.
In this step, get ready to:
- Parse a small Python snippet and print the full AST tree.
- Identify the node types tree-sitter uses for functions and classes.
- Understand start_point, end_point, and child_by_field_name.
Parse a small Python snippet and print the full AST tree
Think of an AST like a table of contents that knows exactly where every chapter, section, and paragraph starts and ends. Token splitting is like tearing pages out of a book at fixed intervals. The AST knows the boundaries.
You'll start by giving tree-sitter a small Python snippet and asking it to show you the full tree.
- Open explore_ast.py in VS Code and paste the following code:
import tree_sitter_python as tspython
from tree_sitter import Language, Parser
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)
code = b"""
class Calculator:
def __init__(self, value=0):
self.value = value
def add(self, x):
self.value += x
return self
def standalone_function(a, b):
\"\"\"Add two numbers together.\"\"\"
return a + b
"""
tree = parser.parse(code)What does this code do?
The first two lines import the Python grammar and the core tree-sitter library.
PY_LANGUAGE = Language(tspython.language()) loads the compiled Python grammar into a Language object. Parser(PY_LANGUAGE) creates a parser that knows Python syntax.
The code variable is a bytes literal containing a small class and a standalone function. parser.parse(code) turns those bytes into an AST stored in tree.
You won't see any output yet. The next chunk adds the visualization that prints the tree.
- Add the following code below tree = parser.parse(code) in explore_ast.py:
def walk_tree(node, indent=0):
"""Recursively print every node in the AST with its type and position."""
line_info = f"[line {node.start_point[0]+1}:{node.start_point[1]} - line {node.end_point[0]+1}:{node.end_point[1]}]"
print(" " * indent + f"{node.type} {line_info}")
for child in node.children:
walk_tree(child, indent + 2)
print("=== Full AST Structure ===")
print()
walk_tree(tree.root_node)What does this code do?
walk_tree is a recursive function that visits every node in the tree. For each node, it prints the node's type (like function_definition or identifier) and its line position.
The indentation increases by 2 for each level of nesting, so you can visually see parent-child relationships. tree.root_node is the top-level node that contains everything.
- Save explore_ast.py.
- Run the script by entering this command in your terminal:
python explore_ast.pyYou should see a large tree printed to your terminal. Each line shows a node type and its line position in the snippet.
Seeing a ModuleNotFoundError?
Make sure you installed all three packages in Step 1: tree-sitter, tree-sitter-python, and tree-sitter-typescript.
Also confirm your terminal is in the ast-ingest folder, not inside the httpx subfolder.
Help me fix my ModuleNotFoundError when running explore_ast.py
Identify the node types for functions and classes
Here's where it gets interesting. Look at your terminal output. You'll see a deeply nested tree, but the key node types jump out when you know what to look for.
What should I see in the output?
Near the top of the tree, you'll see class_definition for the Calculator class. Nested inside it, you'll see function_definition nodes for __init__ and add.
Further down, there's another function_definition for standalone_function. These are the exact node types you'll target in your chunk extractor.
Each node also shows [line X:Y - line X:Y] positions. Tree-sitter knows the precise byte range of every construct. That's how you'll extract exact source spans instead of cutting code at arbitrary token windows.
The two node types that matter for code chunking are function_definition and class_definition. Every other node type (identifier, parameters, block) is structure inside those top-level constructs.
Understand start_point, end_point, and child_by_field_name
You've seen the full tree, but for building a chunk extractor you need three specific properties: where a node starts, where it ends, and what its name is.
The child_by_field_name method is the key. Tree-sitter grammars define named fields on nodes. A function_definition node has a field called name that points directly to the identifier node for the function's name.
- Add the following code at the bottom of explore_ast.py, below the walk_tree(tree.root_node) line:
print()
print("=== Top-Level Nodes ===")
print()
for child in tree.root_node.children:
if child.type in ("function_definition", "class_definition"):
name_node = child.child_by_field_name("name")
name = code[name_node.start_byte:name_node.end_byte].decode("utf-8")
print(f" {child.type}: {name} (lines {child.start_point[0]+1}-{child.end_point[0]+1})")What does this code do?
This loop iterates over the direct children of the root node and filters for function_definition and class_definition nodes only.
child_by_field_name("name") grabs the identifier sub-node that holds the function or class name. You then slice the original bytes using start_byte and end_byte to get the actual text.
start_point and end_point give you the line numbers (0-indexed, so we add 1 for display). These are the exact boundaries you'll use to extract chunk spans in the next step.
- Save explore_ast.py and run it again:
python explore_ast.pyAt the bottom of the output, below the full AST, you should now see a section that looks like this:
=== Top-Level Nodes ===
class_definition: Calculator (lines 2-9)
function_definition: standalone_function (lines 11-13)Why does this matter for chunking?
You now have the three ingredients for AST-aware chunking: the node type tells you what kind of symbol it is, child_by_field_name gives you its name, and start_point/end_point gives you the exact line range.
Compare this to naive token splitting: a 500-token window might cut Calculator in half or merge unrelated code together. AST boundaries guarantee each chunk is one complete, meaningful unit.
Only seeing the Full AST section?
Make sure you added the new code after the walk_tree(tree.root_node) call, not inside the walk_tree function. Check your indentation. The print() and for loop should be at the top level (no indentation).
Help me figure out why my Top-Level Nodes section isn't printing
✔️ Awesome, I've got everything!
Great. Double check you've saved explore_ast.py and that both the Full AST Structure and Top-Level Nodes sections print when you run it.
ⓧ I'd like to double check the full code
import tree_sitter_python as tspython
from tree_sitter import Language, Parser
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)
code = b"""
class Calculator:
def __init__(self, value=0):
self.value = value
def add(self, x):
self.value += x
return self
def standalone_function(a, b):
\"\"\"Add two numbers together.\"\"\"
return a + b
"""
tree = parser.parse(code)
def walk_tree(node, indent=0):
"""Recursively print every node in the AST with its type and position."""
line_info = f"[line {node.start_point[0]+1}:{node.start_point[1]} - line {node.end_point[0]+1}:{node.end_point[1]}]"
print(" " * indent + f"{node.type} {line_info}")
for child in node.children:
walk_tree(child, indent + 2)
print("=== Full AST Structure ===")
print()
walk_tree(tree.root_node)
print()
print("=== Top-Level Nodes ===")
print()
for child in tree.root_node.children:
if child.type in ("function_definition", "class_definition"):
name_node = child.child_by_field_name("name")
name = code[name_node.start_byte:name_node.end_byte].decode("utf-8")
print(f" {child.type}: {name} (lines {child.start_point[0]+1}-{child.end_point[0]+1})")You now have a clear mental model of how tree-sitter represents code. Next up, you'll turn this exploration into a real chunk extractor that captures every function and class in a file.
Build the Chunk Extractor
In the last step, you explored how tree-sitter turns source code into a structured tree. You saw that every function and class has a distinct node type with precise byte offsets.
Now you'll build the extraction function that turns those nodes into structured chunk records. Each chunk captures the symbol name, type, line range, and full body text. By the end of this step, your extractor will handle both Python and TypeScript files.
In this step, get ready to:
- Write the extract_chunks() function that recursively walks the AST and captures functions and classes.
- Add TypeScript support with a language config pattern that makes new languages trivial to add.
- Test extraction on a single file and see structured chunk output.
Write the extraction function for Python
The key insight here is that we already know how to find functions and classes in the AST. We identified the node types in the last step. Now we just need a function that walks the tree, checks each node, and collects the ones we care about.
The extractor will be driven by a config dictionary that tells it which node types to look for. This means adding a new language later is just adding a new config entry.
- Open ingest.py in VS Code (it should be empty).
- Add the imports and language setup by pasting this code:
import json
import sys
from pathlib import Path
import tree_sitter_python as tspython
from tree_sitter import Language, Parser
# --- Language Setup ---
PY_LANGUAGE = Language(tspython.language())
py_parser = Parser(PY_LANGUAGE)
LANGUAGE_CONFIG = {
".py": {
"parser": py_parser,
"language": "python",
"function_types": ["function_definition"],
"class_types": ["class_definition"],
"name_field": "name",
},
}What does this code do?
The imports bring in the tree-sitter core library and the Python grammar. We create a parser the same way you did in explore_ast.py.
LANGUAGE_CONFIG maps file extensions to everything the extractor needs: which parser to use, which node types count as functions or classes, and which field holds the name. This is the pattern that makes multi-language support painless.
You won't see any output yet. The next chunk defines the function that actually uses this config.
- Below the LANGUAGE_CONFIG dictionary, add the extract_chunks function:
# --- Chunk Extraction ---
def extract_chunks(source_code: bytes, config: dict) -> list[dict]:
"""Parse source code and extract function/class chunks at AST boundaries."""
tree = config["parser"].parse(source_code)
chunks = []
def visit_node(node):
if node.type in config["function_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "function",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
elif node.type in config["class_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "class",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
for child in node.children:
visit_node(child)
visit_node(tree.root_node)
return chunksWhat does this code do?
extract_chunks takes raw source bytes and a config dict, then parses the code into an AST. The inner visit_node function does the real work. It checks each node's type against the config's function and class lists.
When it finds a match, it slices the original source bytes using start_byte and end_byte to capture the exact body. It also grabs the name via child_by_field_name. The recursive call at the bottom means it descends into class bodies to find methods too.
- Save ingest.py.
You won't see any output yet. The extraction function needs a test file and a quick run to prove it works. That's coming in the next substep.
Seeing a red underline on the import?
If VS Code highlights import tree_sitter_python with a red underline, your Python interpreter in VS Code might not match the one where you installed packages. Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows), type Python: Select Interpreter, and choose the Python 3.11+ installation where you ran pip install.
Why isn't VS Code recognizing my tree-sitter import?
Add TypeScript support with a language config pattern
Here's where the config pattern pays off. Adding an entire new language is just three things: import the grammar, create a parser, and add a config entry. The extract_chunks function doesn't need to change at all.
- In ingest.py, find this line near the top:
import tree_sitter_python as tspython- Add the TypeScript import directly below it, so both lines look like this:
import tree_sitter_python as tspython
import tree_sitter_typescript as tstypescript- Now find this line in the Language Setup section:
PY_LANGUAGE = Language(tspython.language())
py_parser = Parser(PY_LANGUAGE)- Add the TypeScript language and parser directly below, so all four lines look like this:
PY_LANGUAGE = Language(tspython.language())
TS_LANGUAGE = Language(tstypescript.language_typescript())
py_parser = Parser(PY_LANGUAGE)
ts_parser = Parser(TS_LANGUAGE)Why language_typescript() instead of language()?
The tree-sitter-typescript package bundles two grammars: TypeScript and TSX. The method language_typescript() gives you the plain TypeScript grammar. If you needed JSX support, you'd call language_tsx() instead.
- Finally, find your LANGUAGE_CONFIG dictionary and replace it with this version that includes both Python and TypeScript:
LANGUAGE_CONFIG = {
".py": {
"parser": py_parser,
"language": "python",
"function_types": ["function_definition"],
"class_types": ["class_definition"],
"name_field": "name",
},
".ts": {
"parser": ts_parser,
"language": "typescript",
"function_types": ["function_declaration", "method_definition"],
"class_types": ["class_declaration"],
"name_field": "name",
},
}What does the TypeScript config entry do?
TypeScript uses different node type names than Python. Top-level functions are function_declaration (not function_definition), and methods inside classes are method_definition. Classes are class_declaration. The config encodes these differences so extract_chunks doesn't need any language-specific logic.
- Save ingest.py.
Now you need a TypeScript file to test against. Let's create one with a class and a couple of standalone functions.
- Create a new file called sample.ts in your ast-ingest folder.
- Paste this TypeScript code into it:
class HttpClient {
private baseUrl: string;
constructor(baseUrl: string) {
this.baseUrl = baseUrl;
}
async get(path: string): Promise<Response> {
return fetch(this.baseUrl + path);
}
async post(path: string, body: object): Promise<Response> {
return fetch(this.baseUrl + path, {
method: "POST",
body: JSON.stringify(body),
});
}
}
function createClient(url: string): HttpClient {
return new HttpClient(url);
}
function parseResponse(response: Response): Promise<unknown> {
return response.json();
}What does this file contain?
This file gives you a good spread of TypeScript constructs to test against: one class with a constructor and two methods, plus two standalone functions. Your extractor should find all of them.
- Save sample.ts.
- Confirm the file exists by running:
lsYou should see sample.ts alongside explore_ast.py and ingest.py in the output.
✔️ Awesome, I've got everything!
Great. Double check you've saved both files before moving on.
ⓧ I'd like to double check the full code
Here's what your full ingest.py should look like at this point:
import json
import sys
from pathlib import Path
import tree_sitter_python as tspython
import tree_sitter_typescript as tstypescript
from tree_sitter import Language, Parser
# --- Language Setup ---
PY_LANGUAGE = Language(tspython.language())
TS_LANGUAGE = Language(tstypescript.language_typescript())
py_parser = Parser(PY_LANGUAGE)
ts_parser = Parser(TS_LANGUAGE)
LANGUAGE_CONFIG = {
".py": {
"parser": py_parser,
"language": "python",
"function_types": ["function_definition"],
"class_types": ["class_definition"],
"name_field": "name",
},
".ts": {
"parser": ts_parser,
"language": "typescript",
"function_types": ["function_declaration", "method_definition"],
"class_types": ["class_declaration"],
"name_field": "name",
},
}
# --- Chunk Extraction ---
def extract_chunks(source_code: bytes, config: dict) -> list[dict]:
"""Parse source code and extract function/class chunks at AST boundaries."""
tree = config["parser"].parse(source_code)
chunks = []
def visit_node(node):
if node.type in config["function_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "function",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
elif node.type in config["class_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "class",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
for child in node.children:
visit_node(child)
visit_node(tree.root_node)
return chunksAnd here's your sample.ts:
class HttpClient {
private baseUrl: string;
constructor(baseUrl: string) {
this.baseUrl = baseUrl;
}
async get(path: string): Promise<Response> {
return fetch(this.baseUrl + path);
}
async post(path: string, body: object): Promise<Response> {
return fetch(this.baseUrl + path, {
method: "POST",
body: JSON.stringify(body),
});
}
}
function createClient(url: string): HttpClient {
return new HttpClient(url);
}
function parseResponse(response: Response): Promise<unknown> {
return response.json();
}Test extraction on a single file
Time to see your extractor in action. You'll load the module in Python's interactive shell and run it against your TypeScript sample file.
- Make sure your terminal is in the ast-ingest folder.
- Start a Python interactive session by running:
python3- Inside the Python shell, import your extractor and run it on sample.ts by entering these lines:
from ingest import extract_chunks, LANGUAGE_CONFIG
# Read the TypeScript sample file as bytes
source = open("sample.ts", "rb").read()
# Extract chunks using the TypeScript config
chunks = extract_chunks(source, LANGUAGE_CONFIG[".ts"])
# Print each chunk's summary
for c in chunks:
print(f"{c['symbol_type']}: {c['symbol_name']} (lines {c['start_line']}-{c['end_line']})") You should see output listing the class and all the functions your extractor found:
class: HttpClient (lines 1-19)
function: constructor (lines 4-6)
function: get (lines 8-10)
function: post (lines 12-18)
function: createClient (lines 21-23)
function: parseResponse (lines 25-27)Why does it find methods AND the class?
Because visit_node recurses into every child. It finds the class_declaration at the top level, records it, then descends into the class body where it finds method_definition nodes for each method. This gives you both the big picture (the full class) and the fine-grained details (individual methods).
- Exit the Python shell by typing:
exit()Seeing a ModuleNotFoundError?
If Python says it can't find ingest, make sure you started python3 from the ast-ingest folder (the one containing ingest.py). Run pwd to check your current directory.
If you see an error about tree_sitter_typescript not being found, make sure the python3 you're running is the same one where you installed the packages with pip in Step 1.
Help me debug why my extract_chunks test isn't working.
Your chunk extractor works on single files. Next up, you'll wrap it in a function that walks an entire repository, processes every supported file, and writes all chunks to a JSON file. That's how you'll ingest the full httpx codebase in one command.
Walk a Full Repository and Output JSON
Your extract_chunks() function can parse a single file and return structured chunks. Now it's time to scale this up to an entire repository with hundreds of files.
In this step, you'll write a repository walker that finds every supported file, extracts its chunks, and writes the full result to a JSON file. Then you'll run it against the real httpx codebase to prove it works at scale.
In this step, get ready to:
- Write a repository walker that recursively finds and parses all supported files.
- Add a CLI interface that accepts a repo path and writes JSON output.
- Run the tool against httpx and inspect the structured chunks it produces.
Write the repository walker function
A single file extractor is useful for testing, but a real ingestion pipeline needs to walk an entire directory tree. Python's Path.rglob() gives you a recursive file iterator that handles this perfectly.
- In ingest.py, find the end of your extract_chunks() function (after the return chunks line).
- Add the following ingest_repo() function below it:
# --- Repository Walker ---
def ingest_repo(repo_path: str, repo_name: str) -> list[dict]:
"""Walk a repository and extract chunks from all supported files."""
all_chunks = []
root = Path(repo_path)
for file_path in root.rglob("*"):
if not file_path.is_file():
continue
ext = file_path.suffix
if ext not in LANGUAGE_CONFIG:
continue
config = LANGUAGE_CONFIG[ext]
try:
source_code = file_path.read_bytes()
except (OSError, UnicodeDecodeError):
continue
file_chunks = extract_chunks(source_code, config)
# Add repo-level metadata to each chunk
relative_path = str(file_path.relative_to(root)).replace("\\\\", "/")
for chunk in file_chunks:
chunk["repo"] = repo_name
chunk["path"] = relative_path
chunk["language"] = config["language"]
all_chunks.extend(file_chunks)
return all_chunksWhat does this code do?
Path(repo_path).rglob("*") recursively yields every file and directory inside the repo. The function skips directories and files with unsupported extensions.
The try/except block gracefully handles binary files or permission errors that would otherwise crash the entire run. After extracting chunks from each file, the function enriches every chunk with the repo name, relative file path, and language.
The .replace("\\\\", "/") normalizes Windows backslash paths to forward slashes so the JSON output is consistent across operating systems.
- Save ingest.py.
- You won't see a visible change yet. The next chunk adds the CLI entry point that actually calls this function.
Add the CLI interface with JSON output
Your walker function returns a list of chunks in memory. You need a command-line entry point that accepts a repo path, runs the ingestion, writes the result to a JSON file, and prints summary statistics so you know what it found.
- In ingest.py, add the following main() function and entry point guard below ingest_repo():
# --- CLI ---
def main():
if len(sys.argv) < 2:
print("Usage: python ingest.py <repo_path> [repo_name]")
print("Example: python ingest.py ./httpx httpx")
sys.exit(1)
repo_path = sys.argv[1]
repo_name = sys.argv[2] if len(sys.argv) > 2 else Path(repo_path).name
print(f"Ingesting repository: {repo_name}")
print(f" Path: {repo_path}")
print()
# Run the ingestion pipeline
chunks = ingest_repo(repo_path, repo_name)
# Write all chunks to a JSON file
output_file = f"{repo_name}_chunks.json"
with open(output_file, "w", encoding="utf-8") as f:
json.dump(chunks, f, indent=2)
# Print summary statistics
func_count = sum(1 for c in chunks if c["symbol_type"] == "function")
class_count = sum(1 for c in chunks if c["symbol_type"] == "class")
print(f" Total chunks: {len(chunks)}")
print(f" Functions: {func_count}")
print(f" Classes: {class_count}")
print()
print(f"Output written to: {output_file}")
# Show a preview of the first chunk
if chunks:
print()
print("=== Sample Chunk ===")
sample = chunks[0]
print(f" [{sample['repo']}] {sample['path']}")
print(f" {sample['symbol_type']}: {sample['symbol_name']}")
print(f" Lines {sample['start_line']}-{sample['end_line']}")
preview = sample["body"][:120].replace("\n", "\n ")
print(f" {preview}...")
if __name__ == "__main__":
main()What does this code do?
The argument check at the top ensures the learner passes a repo path. If they forget, the script prints a usage example and exits cleanly instead of crashing with an index error.
After ingestion, json.dump(chunks, f, indent=2) writes the entire chunk list as human-readable JSON. The summary stats give you a quick count of functions vs classes, and the sample chunk preview lets you verify the output looks correct without opening the full file.
The if __name__ == "__main__": guard means main() only runs when you execute the script directly. This lets other projects import ingest_repo() as a library without triggering the CLI.
- Save ingest.py.
Seeing a syntax error on save?
Make sure the main() function starts at the same indentation level (column 0) as ingest_repo() above it. Python is whitespace-sensitive, so an extra indent will nest it inside the previous function.
help me fix indentation errors in my ingest.py file
Run the tool against httpx
Here's where it gets interesting. You have a real open-source repository with hundreds of Python files sitting in your project folder. Let's point the ingestion tool at it and see what comes out.
- In your terminal, make sure you're inside the ast-ingest folder.
- Run the ingestion tool against your cloned httpx repository by running this command:
python ingest.py ./httpx httpxYou should see output showing the total number of chunks extracted, a breakdown of functions vs classes, and a sample chunk preview. The exact numbers depend on the current state of the httpx repository, but expect hundreds of chunks.
What should I see?
Your terminal should print something like:
Ingesting repository: httpx
Path: ./httpx
Total chunks: 400+
Functions: 300+
Classes: 60+
Output written to: httpx_chunks.json
=== Sample Chunk ===
[httpx] httpx/_api.py
function: request
Lines 5-55
def request(...The exact numbers will vary based on the current httpx codebase. What matters is that you see a large number of chunks successfully extracted.
- Confirm that httpx_chunks.json was created by running:
lsYou should see httpx_chunks.json listed alongside ingest.py, explore_ast.py, sample.ts, and the httpx/ folder.
- Open httpx_chunks.json in VS Code and inspect the first few entries.
Each chunk is a JSON object with these fields: repo, path, language, symbol_name, symbol_type, start_line, end_line, and body. Every chunk maps precisely to one function or class in the source code, cut at exact AST boundaries.
Why does this matter for retrieval?
Compare this to naive token-window chunking, which would slice the source code every 500 tokens regardless of where functions start or end. You'd get chunks that contain the bottom half of one function and the top half of the next.
Your AST-aware chunks are self-contained. Each one is a complete, meaningful unit of code. When you embed these in Project 2, a search for "HTTP POST request" will retrieve the entire post() method body, not a fragment that starts mid-function.
Seeing 0 total chunks?
Make sure you're running the command from inside the ast-ingest folder where the httpx/ directory lives. The path ./httpx is relative to your current working directory.
If you see a "No such file or directory" error, run ls to check that httpx/ appears in the listing. If it doesn't, you may need to cd into the correct folder first.
help me debug why ingest.py is returning 0 chunks
✔️ Awesome, I've got everything!
Great. Double check you've saved ingest.py and confirmed that httpx_chunks.json was generated successfully.
ⓧ I'd like to double check the full code
Here's the complete ingest.py file for comparison:
import json
import sys
from pathlib import Path
import tree_sitter_python as tspython
import tree_sitter_typescript as tstypescript
from tree_sitter import Language, Parser
# --- Language Setup ---
PY_LANGUAGE = Language(tspython.language())
TS_LANGUAGE = Language(tstypescript.language_typescript())
py_parser = Parser(PY_LANGUAGE)
ts_parser = Parser(TS_LANGUAGE)
LANGUAGE_CONFIG = {
".py": {
"parser": py_parser,
"language": "python",
"function_types": ["function_definition"],
"class_types": ["class_definition"],
"name_field": "name",
},
".ts": {
"parser": ts_parser,
"language": "typescript",
"function_types": ["function_declaration", "method_definition"],
"class_types": ["class_declaration"],
"name_field": "name",
},
}
# --- Chunk Extraction ---
def extract_chunks(source_code: bytes, config: dict) -> list[dict]:
"""Parse source code and extract function/class chunks at AST boundaries."""
tree = config["parser"].parse(source_code)
chunks = []
def visit_node(node):
if node.type in config["function_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "function",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
elif node.type in config["class_types"]:
name_node = node.child_by_field_name(config["name_field"])
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte].decode("utf-8")
if name_node
else "<anonymous>"
)
chunks.append({
"symbol_name": symbol_name,
"symbol_type": "class",
"start_line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1,
"body": source_code[node.start_byte:node.end_byte].decode("utf-8"),
})
for child in node.children:
visit_node(child)
visit_node(tree.root_node)
return chunks
# --- Repository Walker ---
def ingest_repo(repo_path: str, repo_name: str) -> list[dict]:
"""Walk a repository and extract chunks from all supported files."""
all_chunks = []
root = Path(repo_path)
for file_path in root.rglob("*"):
if not file_path.is_file():
continue
ext = file_path.suffix
if ext not in LANGUAGE_CONFIG:
continue
config = LANGUAGE_CONFIG[ext]
try:
source_code = file_path.read_bytes()
except (OSError, UnicodeDecodeError):
continue
file_chunks = extract_chunks(source_code, config)
relative_path = str(file_path.relative_to(root)).replace("\\\\", "/")
for chunk in file_chunks:
chunk["repo"] = repo_name
chunk["path"] = relative_path
chunk["language"] = config["language"]
all_chunks.extend(file_chunks)
return all_chunks
# --- CLI ---
def main():
if len(sys.argv) < 2:
print("Usage: python ingest.py <repo_path> [repo_name]")
print("Example: python ingest.py ./httpx httpx")
sys.exit(1)
repo_path = sys.argv[1]
repo_name = sys.argv[2] if len(sys.argv) > 2 else Path(repo_path).name
print(f"Ingesting repository: {repo_name}")
print(f" Path: {repo_path}")
print()
chunks = ingest_repo(repo_path, repo_name)
output_file = f"{repo_name}_chunks.json"
with open(output_file, "w", encoding="utf-8") as f:
json.dump(chunks, f, indent=2)
func_count = sum(1 for c in chunks if c["symbol_type"] == "function")
class_count = sum(1 for c in chunks if c["symbol_type"] == "class")
print(f" Total chunks: {len(chunks)}")
print(f" Functions: {func_count}")
print(f" Classes: {class_count}")
print()
print(f"Output written to: {output_file}")
if chunks:
print()
print("=== Sample Chunk ===")
sample = chunks[0]
print(f" [{sample['repo']}] {sample['path']}")
print(f" {sample['symbol_type']}: {sample['symbol_name']}")
print(f" Lines {sample['start_line']}-{sample['end_line']}")
preview = sample["body"][:120].replace("\n", "\n ")
print(f" {preview}...")
if __name__ == "__main__":
main()You now have a fully working ingestion pipeline that turns any Git repository into structured, searchable chunks. This ingest.py module becomes a direct dependency in the next projects of this capstone series, where you'll embed these chunks into vectors and build hybrid search over them.
Secret mission
Add Go Support and Handle Edge Cases
Your ingestion tool handles the happy path beautifully. But real codebases are messy. Decorators sit above functions, arrow functions hide inside variable declarations, and nested classes create naming ambiguity. Time to make your tool production-grade.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for the next capstone project, or clean them up.
Resources you used:
- Project folder with explore_ast.py, ingest.py, and sample.ts.
- Cloned httpx repository (test corpus).
- Generated httpx_chunks.json output file.
- Installed pip packages: tree-sitter, tree-sitter-python, tree-sitter-typescript.
Keep everything running
No action needed. Choose this if you plan to continue with the next capstone project, which imports ingest.py directly.
- Keep your project folder and the cloned httpx repo in place.
- Your pip packages stay installed and ready for the next project.
Pause - I'll come back to this later
Free up disk space from the test corpus while keeping your project files intact.
- Delete the cloned httpx folder. You can re-clone it anytime with a single command.
- Keep explore_ast.py, ingest.py, sample.ts, and httpx_chunks.json. These are small files.
- Your pip packages remain installed. No need to uninstall them.
Delete - I don't want to use this again
Remove all project resources and start fresh.
- Delete your entire project folder (including the cloned httpx repo and all generated output).
- Uninstall the pip packages by running:
pip uninstall tree-sitter tree-sitter-python tree-sitter-typescript -yPlanning to continue the capstone series?
Project 2 (Dense Embedding + Vector Storage) imports ingest.py directly. If you delete now, you will need to rebuild this project before starting Project 2.
Nice Work!
Nice Work!
You just built the exact ingestion layer that powers tools like Sourcegraph, Cursor, and Aider under the hood. Your code reads entire repositories and produces structured, function-level chunks at AST boundaries.
You've learned how to:
- Build a CLI tool that parses entire repositories into structured chunks using tree-sitter AST boundaries instead of naive token splitting.
- Support multiple languages (Python and TypeScript) with an extensible config pattern that makes adding new grammars trivial.
- Produce structured JSON output with chunk records ready to feed embeddings, BM25 indexes, and symbol graphs in upcoming capstone projects.
- Extended the tool with Go language support, decorator handling, and arrow function extraction.
Ready to quiz yourself?