Build MTS Knowledge Retrieval

Introduction

30 Second Summary

Finding the right file is only half the job. You still need to know which exact section supports the answer.

In this project, you will build a command-line retrieval foundation for the organizer's real MTS knowledge archive. You will prove its evidence layer with heading-aware Markdown chunks, TF-IDF, and cosine similarity before any chatbot is introduced.

What You'll Build

Picture asking a demo question in the terminal and seeing the exact source path, section heading, score, and excerpt behind each match.

By the end of this project, you'll have:

• A verified ZIP inspection that lists all 19 Markdown documents under their original MTS categories and archive paths.
• A before-and-after retrieval comparison that exposes oversized whole-document matches before replacing them with precise heading-level results.
• A diagnostic retrieval report covering every demo question. A synthetic out-of-vocabulary probe also proves the retriever can return no evidence.
• Secret Mission: Export a readable JSON catalog of every chunk for auditing and reuse in Project 2.

Are there any prerequisites?

You need the original organizer knowledge-base.zip file plus the demo questions CSV.

The Windows setup assumes Visual Studio Code is already installed. The project verifies Python 3.11 plus the Python extension before any retrieval work begins.

Before We Start

Before any chatbot work begins, commit to proving that the original MTS archive can be parsed, preserved, chunked, and searched as reliable evidence. This keeps every later recommendation grounded in retrieved sections instead of unverified model output.

Set Up the Windows Workspace

Global Python packages can make a broken setup look healthy. That hidden mismatch can break the retrieval scripts on another machine.

Here, you prepare a dedicated Visual Studio Code workspace. An isolated virtual environment pins scikit-learn to version 1.8.0.

Docker, GitHub, Ollama, chatbots, LLMs, and model APIs remain outside this project. This step stays focused on a repeatable local foundation.

In this step, get ready to:

• Create the Windows workspace and its three working directories.
• Confirm Python 3.11 and create a selected .venv environment.
• Install scikit-learn 1.8.0 from a pinned requirements file and protect local data from version control.

Create and open the workspace

Workspace boundaries keep private organizer inputs away from generated outputs. They also make each future file's role clear.

• Press the Windows key to open Windows search.
• Type Visual Studio Code and press Enter.
• Click File in the top menu bar.
• Select Open Folder....
• Navigate to your Desktop in the folder dialog.
• Use the folder creation control to create mts-energytool-consultant on your Desktop.
• Select the mts-energytool-consultant folder.
• Confirm the folder selection.

VS Code now treats mts-energytool-consultant as the workspace. The file tree on the left is ready for the project structure.

• Click View in the top menu bar.
• Select Terminal to open the integrated terminal.
• Choose PowerShell from the terminal profile control if another shell is selected.
• Create the three working directories by running this command:

New-Item -ItemType Directory -Path "scripts","local_data","runtime_data" -Force

What do these directories do?

• The workspace root holds reusable retrieval modules in later steps.
• The scripts directory holds runnable validation and retrieval checks.
• The local_data directory holds private organizer inputs.
• The runtime_data directory holds generated reports.

• Check the file tree on the left for scripts, local_data, and runtime_data.

You should see all three directories directly inside mts-energytool-consultant.

Directories missing from the file tree?

Confirm that the workspace name is mts-energytool-consultant. A different workspace sends the new directories to the wrong location.

Make sure the terminal profile is PowerShell before retrying the directory command.

Help me create the workspace directories in the correct folder.

Select Python 3.11 and create .venv

The system interpreter supplies Python to VS Code. The project environment then isolates its packages from everything installed globally.

• Check the Python version available to the workspace by running this command:

python --version

What does this check prove?

The command prints the Python interpreter currently available through PowerShell. This project targets Python 3.11.

✔️ I see Python 3.11.x

Python 3.11 is available. You can use it to create the project environment below.

ⓧ I see a different version

Your current command points to a different Python version. Install Python 3.11 alongside it so VS Code can use the required interpreter.

• Press the Windows key to open Windows search.
• Type Microsoft Store and press Enter.
• Search for Python 3.11 in Microsoft Store.
• Select the package published by the Python Software Foundation.
• Click Install.
• Return to the mts-energytool-consultant workspace in VS Code.

Python 3.11 is now available for the interpreter selection below.

ⓧ Command not found

PowerShell cannot find a Python interpreter. Install Python 3.11 before creating the project environment.

• Press the Windows key to open Windows search.
• Type Microsoft Store and press Enter.
• Search for Python 3.11 in Microsoft Store.
• Select the package published by the Python Software Foundation.
• Click Install.
• Return to the mts-energytool-consultant workspace in VS Code.

Python 3.11 is now available for the interpreter selection below.

The extension search can feel fiddly because several results mention Python. The official result is named Python and lists Microsoft as its publisher.

• Press Ctrl+Shift+X to open the Extensions view.
• Type Python into the extension search box.
• Select Python by Microsoft.
• Click Install if that button is visible.

If the extension page shows a manage control, the official Python extension is already installed.

• Press Ctrl+Shift+P to open the Command Palette.
• Type Python: Create Environment.
• Select Python: Create Environment.
• Select Venv as the environment type.
• Select an interpreter whose version begins with Python 3.11.

VS Code creates a .venv directory inside the workspace. It also selects the new environment automatically.

• Press Ctrl+Shift+P to reopen the Command Palette.
• Select Python: Select Interpreter.
• Choose the interpreter inside .venv.
• Close the existing integrated terminal.
• Click View in the top menu bar.
• Select Terminal to open a fresh integrated terminal.

The new PowerShell prompt should begin with (.venv). The file tree should also contain a .venv directory.

Can't create or select .venv?

Confirm that the official Python extension is enabled. Then retry Python: Create Environment with a Python 3.11 interpreter.

If Python was installed while VS Code was open, restart VS Code so the interpreter list refreshes.

Help me get VS Code to create and select my Python 3.11 .venv.

Pin dependencies and protect local data

A requirements file makes the dependency version repeatable. A Git ignore file keeps the virtual environment and private organizer inputs outside version control.

• Select the new-file control at the top of the workspace file tree.
• Enter requirements.txt as the file name.
• Add the compatibility pin to requirements.txt by pasting this line:

scikit-learn==1.8.0

Why pin this version?

The exact pin gives every installation the same scikit-learn version. That removes dependency drift from later retrieval checks.

• Save requirements.txt with Ctrl+S.
• Return to the fresh integrated PowerShell terminal.
• Install the pinned dependency into .venv by running this command:

python -m pip install -r requirements.txt

What does this command do?

Python runs the package installer from the selected environment. The installer reads requirements.txt and installs its exact dependency pin.

The installation should finish without an installation error. The final verification checks the installed version directly.

Dependency installation failed?

Confirm that requirements.txt is saved directly inside mts-energytool-consultant.

Confirm that the PowerShell prompt begins with (.venv) before retrying the installation command.

Help me install scikit-learn 1.8.0 inside the selected .venv.

✔️ Awesome, I've got everything!

Your requirements.txt file contains the Project 1 compatibility pin.

ⓧ I'd like to double check the full code

Compare your complete requirements.txt file with this reference.

scikit-learn==1.8.0

What should match?

The file contains one exact package pin. Extra packages would change the intended Project 1 environment.

• Select the new-file control at the top of the workspace file tree.
• Enter .gitignore as the file name.
• Add the workspace exclusions to .gitignore by pasting this content:

.venv/
__pycache__/
*.pyc
local_data/*
!local_data/.gitkeep
runtime_data/*
!runtime_data/.gitkeep

What does this file protect?

• The .venv rule excludes installed environment files.
• The cache rules exclude generated Python cache files.
• The local_data rule excludes private organizer inputs.
• The runtime_data rule excludes generated reports.

• Save .gitignore with Ctrl+S.
• Check the workspace file tree for .gitignore and requirements.txt.

Both files should appear directly inside mts-energytool-consultant.

✔️ Awesome, I've got everything!

Your .gitignore protects the environment, private inputs, and generated reports.

ⓧ I'd like to double check the full code

Compare your complete .gitignore file with this reference.

.venv/
__pycache__/
*.pyc
local_data/*
!local_data/.gitkeep
runtime_data/*
!runtime_data/.gitkeep

What should match?

Each exclusion and exception should appear on its own line. The exceptions keep space for future .gitkeep files without tracking private inputs or generated reports.

Before you run the final check, do you expect the fresh terminal to use the global interpreter or the selected project environment?

• Verify the selected Python version and installed packages by running these commands:

python --version
python -m pip list

What do these checks prove?

The first command identifies the interpreter version selected in the fresh terminal. The second command lists packages installed inside that same environment.

The first command should print Python 3.11.x. The package list should include scikit-learn 1.8.0.

That is the setup hurdle cleared. Your workspace now uses the expected interpreter and retrieval dependency.

Seeing the wrong version or a missing package?

Use Python: Select Interpreter to choose the interpreter inside .venv. Then open another fresh integrated terminal.

If scikit-learn is missing, rerun the requirements installation command from the fresh terminal.

Help me verify Python 3.11 and scikit-learn 1.8.0 in my .venv.

Your isolated Windows workspace is ready. Next, you will place the unchanged archive in local_data and prove that all 19 Markdown documents can be read directly from the ZIP.

Inspect the Original Knowledge Archive

Your isolated Python workspace in Visual Studio Code is ready. The evidence layer still depends on an unopened organizer archive.

The original ZIP archive is the chain of custody for all 19 Markdown documents.

You will verify UTF-8 decoding. You will also preserve each source path and MTS category.

In this step, get ready to:

• Copy the unchanged organizer archive into the private input directory.
• Build a loader that reads Markdown members directly from the archive.
• Validate the document count, categories, and original archive paths.

Preserve the organizer's original archive

Reading the archive directly keeps the organizer's member names intact. It also avoids path changes caused by extracting or renaming files.

• In Windows File Explorer, locate the organizer-supplied knowledge-base.zip.
• Drag knowledge-base.zip into the local_data directory in the VS Code file tree.
• Confirm local_data/knowledge-base.zip appears in the workspace file tree.
• Keep the archive compressed with its original filename.

The private input is now in place. Its internal Markdown paths remain untouched for the loader to inspect.

Build the archive loader

The loader turns each Markdown member into a frozen data record. Frozen records protect the source metadata from accidental changes after loading.

• Select the new-file icon in the VS Code workspace file tree.
• Name the file kb_core.py.
• Add the imports, known categories, and document record by pasting this code into kb_core.py:

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from zipfile import BadZipFile, ZipFile

KNOWN_CATEGORIES = (
    "Справочная информация",
    "Знания по энергетике",
    "Продукты",
)


@dataclass(frozen=True)
class KnowledgeDocument:
    source_path: str
    category: str
    title: str
    text: str

What does this foundation define?

• The imports provide immutable records, path handling, and read-only ZIP access.
• KNOWN_CATEGORIES lists the recognized MTS directory names.
• KnowledgeDocument keeps the exact member path beside its category, title, and text.

• Save kb_core.py.
• Confirm the editor shows the four KnowledgeDocument fields beneath the category list.

Seeing import or syntax warnings?

Check that kb_core.py sits beside requirements.txt. Confirm the selected interpreter still points to .venv.

Ask for help with the module foundation.

• Add the chunk record two blank lines below KnowledgeDocument by pasting this code:

@dataclass(frozen=True)
class KnowledgeChunk:
    chunk_id: str
    source_path: str
    category: str
    title: str
    heading_path: str
    content: str

    @property
    def search_text(self) -> str:
        return "\n".join(
            (
                self.title,
                self.category,
                self.source_path,
                self.heading_path,
                self.content,
            )
        )

Why keep chunk metadata together?

KnowledgeChunk keeps every future section connected to its original document. The search_text property combines searchable content without discarding evidence metadata.

• Save kb_core.py.
• Confirm KnowledgeChunk contains six fields plus the search_text property.

Is the property indented incorrectly?

Keep @property aligned with the chunk fields. Indent the property body one additional level.

Ask for help checking the dataclass indentation.

• Add the category helper two blank lines below KnowledgeChunk by pasting this code:

def _category_from_path(source_path: str) -> str:
    parts = PurePosixPath(source_path).parts
    for part in parts:
        if part in KNOWN_CATEGORIES:
            return part
    if len(parts) > 1:
        return parts[-2]
    return "Без категории"

How is the category preserved?

PurePosixPath interprets archive member names with ZIP-style separators. The helper returns a recognized MTS category when one appears in the original path.

• Save kb_core.py.
• Confirm the fallback value at the end of _category_from_path() remains in Russian.

Seeing damaged Russian text?

Undo the changed characters. Paste the category helper again without translating its string values.

Ask for help preserving the Russian source text.

• Add the heading parser two blank lines below _category_from_path() by pasting this code:

def _heading_from_line(line: str) -> tuple[int, str] | None:
    stripped = line.strip()
    marker, separator, heading = stripped.partition(" ")
    if not separator:
        return None
    if not 1 <= len(marker) <= 6:
        return None
    if set(marker) != {"#"}:
        return None
    clean_heading = heading.strip().strip("#").strip()
    if not clean_heading:
        return None
    return len(marker), clean_heading

How are headings recognized?

• The helper accepts heading markers from one through six hash characters.
• It returns the heading depth beside cleaned heading text.
• It returns no heading when the line does not match the expected structure.

• Save kb_core.py.
• Confirm _heading_from_line() ends by returning the marker length with the cleaned heading.

Seeing an error around the hash marker?

Confirm the set contains one quoted hash character. Check that each conditional remains inside _heading_from_line().

Ask for help checking the heading parser.

• Add the title helper two blank lines below _heading_from_line() by pasting this code:

def _title_from_text(text: str, source_path: str) -> str:
    in_fence = False
    for line in text.splitlines():
        stripped = line.strip()
        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fence = not in_fence
            continue
        if in_fence:
            continue
        heading = _heading_from_line(line)
        if heading and heading[0] == 1:
            return heading[1]
    return PurePosixPath(source_path).stem.replace("_", " ")

How is each title derived?

The helper uses the first level-one heading outside fenced code. A document without that heading receives a title based on its original member filename.

• Save kb_core.py.
• Confirm the final line replaces filename underscores with spaces.

Does a fence check look incomplete?

Confirm the condition checks both backtick fences and tilde fences. Keep the two fence values inside quoted strings.

Ask for help checking title extraction.

The main loader performs the archive checks in one function. Its first half locates Markdown members before reading their raw bytes.

• Start load_markdown_documents() two blank lines below _title_from_text() by pasting this first half:

def load_markdown_documents(archive_path: Path) -> list[KnowledgeDocument]:
    if not archive_path.is_file():
        raise FileNotFoundError(f"Archive not found: {archive_path}")

    try:
        with ZipFile(archive_path, mode="r") as archive:
            members = sorted(
                (
                    info
                    for info in archive.infolist()
                    if not info.is_dir()
                    and PurePosixPath(info.filename).suffix.lower() == ".md"
                ),
                key=lambda info: info.filename.casefold(),
            )

            documents: list[KnowledgeDocument] = []
            for info in members:
                with archive.open(info, mode="r") as member:
                    raw_text = member.read()

What does the first half check?

• The file check produces a clear failure when the archive is missing.
• Read mode keeps the original ZIP unchanged.
• The member filter keeps files whose original path ends in .md.

• Complete load_markdown_documents() by adding this continuation directly beneath the first half:

                try:
                    text = raw_text.decode("utf-8-sig")
                except UnicodeDecodeError as exc:
                    raise ValueError(
                        f"Markdown member is not valid UTF-8: {info.filename}"
                    ) from exc

                documents.append(
                    KnowledgeDocument(
                        source_path=info.filename,
                        category=_category_from_path(info.filename),
                        title=_title_from_text(text, info.filename),
                        text=text,
                    )
                )
    except BadZipFile as exc:
        raise ValueError(f"Invalid ZIP archive: {archive_path}") from exc

    return documents

What does the completed loader preserve?

• The utf-8-sig decoder accepts UTF-8 text with an optional byte order mark.
• The exact info.filename value becomes the document source path.
• Invalid ZIP data and invalid text encoding produce distinct failures.

• Save kb_core.py.
• Confirm the completed function ends with return documents.

Seeing indentation errors in the loader?

Keep the decoding block inside the member loop. Align the bad-ZIP handler with the outer try block.

Ask for help checking the loader structure.

The module also prepares reusable section records for later retrieval steps. These helpers preserve front matter, heading depth, and fenced code boundaries consistently.

• Add the front-matter helper two blank lines below load_markdown_documents() by pasting this code:

def _without_front_matter(text: str) -> list[str]:
    lines = text.splitlines()
    if not lines or lines[0].strip() != "---":
        return lines

    for index in range(1, len(lines)):
        if lines[index].strip() == "---":
            return lines[index + 1 :]
    return lines

Why remove front matter?

Front matter stores document-level metadata outside the readable section content. The helper removes a complete opening block while preserving every remaining line.

• Save kb_core.py.
• Confirm the helper returns every line after the closing front-matter marker.

Is ordinary content being removed?

Confirm the helper checks the first line before searching for a closing marker. A document without an opening marker must return all lines.

Ask for help checking front-matter removal.

The section parser is one continuous function. Add both halves directly beside each other before saving the completed function.

• Start chunk_markdown() two blank lines below _without_front_matter() by pasting this first half:

def chunk_markdown(
    document: KnowledgeDocument,
    document_number: int,
) -> list[KnowledgeChunk]:
    chunks: list[KnowledgeChunk] = []
    heading_stack: list[str] = []
    section_lines: list[str] = []
    section_number = 0
    in_fence = False

    def flush_section() -> None:
        nonlocal section_number
        content = "\n".join(section_lines).strip()
        section_lines.clear()
        if not content:
            return

        section_number += 1
        heading_path = (
            " > ".join(heading_stack) if heading_stack else document.title
        )
        chunks.append(
            KnowledgeChunk(
                chunk_id=f"{document_number:02d}-{section_number:03d}",
                source_path=document.source_path,
                category=document.category,
                title=document.title,

What does each section record keep?

The nested helper collects one non-empty section at a time. Its stable ID combines the document number with the section number.

• Complete chunk_markdown() by adding this continuation directly beneath the first half:

                heading_path=heading_path,
                content=content,
            )
        )

    for line in _without_front_matter(document.text):
        stripped = line.strip()
        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fence = not in_fence
            section_lines.append(line)
            continue

        heading = None if in_fence else _heading_from_line(line)
        if heading is None:
            section_lines.append(line)
            continue

        flush_section()
        level, heading_text = heading
        del heading_stack[level - 1 :]
        heading_stack.append(heading_text)

    flush_section()
    return chunks

How are section boundaries protected?

• Fence markers toggle code-fence tracking while remaining inside the section content.
• Heading-like lines inside a fence stay as content.
• A real heading flushes the previous section before updating the heading breadcrumb.

• Save kb_core.py.
• Confirm the function finishes by flushing the final section before returning its chunks.

Seeing an incomplete function warning?

Confirm the continuation begins immediately after the title=document.title, line. Keep the final flush outside the document-line loop.

Ask for help checking the parser structure.

• Add the collection helper two blank lines below chunk_markdown() by pasting this code:

def chunk_documents(
    documents: list[KnowledgeDocument],
) -> list[KnowledgeChunk]:
    chunks: list[KnowledgeChunk] = []
    for document_number, document in enumerate(documents, start=1):
        chunks.extend(chunk_markdown(document, document_number))
    return chunks

What does the collection helper do?

The helper assigns each document a stable one-based number. It combines every document's sections into one chunk list.

• Save kb_core.py.
• Confirm chunk_documents() is the final function in the file.

Does the final helper show a name warning?

Check that it calls chunk_markdown with the singular document variable. Confirm both type names match the dataclasses at the top of the file.

Ask for help checking the collection helper.

✔️ Awesome, I've got everything!

Great. Double check that you saved kb_core.py before building the inspection script.

ⓧ I'd like to double check the full code

Compare your complete kb_core.py with this reference.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from zipfile import BadZipFile, ZipFile

KNOWN_CATEGORIES = (
    "Справочная информация",
    "Знания по энергетике",
    "Продукты",
)


@dataclass(frozen=True)
class KnowledgeDocument:
    source_path: str
    category: str
    title: str
    text: str


@dataclass(frozen=True)
class KnowledgeChunk:
    chunk_id: str
    source_path: str
    category: str
    title: str
    heading_path: str
    content: str

    @property
    def search_text(self) -> str:
        return "\n".join(
            (
                self.title,
                self.category,
                self.source_path,
                self.heading_path,
                self.content,
            )
        )


def _category_from_path(source_path: str) -> str:
    parts = PurePosixPath(source_path).parts
    for part in parts:
        if part in KNOWN_CATEGORIES:
            return part
    if len(parts) > 1:
        return parts[-2]
    return "Без категории"


def _heading_from_line(line: str) -> tuple[int, str] | None:
    stripped = line.strip()
    marker, separator, heading = stripped.partition(" ")
    if not separator:
        return None
    if not 1 <= len(marker) <= 6:
        return None
    if set(marker) != {"#"}:
        return None
    clean_heading = heading.strip().strip("#").strip()
    if not clean_heading:
        return None
    return len(marker), clean_heading


def _title_from_text(text: str, source_path: str) -> str:
    in_fence = False
    for line in text.splitlines():
        stripped = line.strip()
        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fence = not in_fence
            continue
        if in_fence:
            continue
        heading = _heading_from_line(line)
        if heading and heading[0] == 1:
            return heading[1]
    return PurePosixPath(source_path).stem.replace("_", " ")


def load_markdown_documents(archive_path: Path) -> list[KnowledgeDocument]:
    if not archive_path.is_file():
        raise FileNotFoundError(f"Archive not found: {archive_path}")

    try:
        with ZipFile(archive_path, mode="r") as archive:
            members = sorted(
                (
                    info
                    for info in archive.infolist()
                    if not info.is_dir()
                    and PurePosixPath(info.filename).suffix.lower() == ".md"
                ),
                key=lambda info: info.filename.casefold(),
            )

            documents: list[KnowledgeDocument] = []
            for info in members:
                with archive.open(info, mode="r") as member:
                    raw_text = member.read()
                try:
                    text = raw_text.decode("utf-8-sig")
                except UnicodeDecodeError as exc:
                    raise ValueError(
                        f"Markdown member is not valid UTF-8: {info.filename}"
                    ) from exc

                documents.append(
                    KnowledgeDocument(
                        source_path=info.filename,
                        category=_category_from_path(info.filename),
                        title=_title_from_text(text, info.filename),
                        text=text,
                    )
                )
    except BadZipFile as exc:
        raise ValueError(f"Invalid ZIP archive: {archive_path}") from exc

    return documents


def _without_front_matter(text: str) -> list[str]:
    lines = text.splitlines()
    if not lines or lines[0].strip() != "---":
        return lines

    for index in range(1, len(lines)):
        if lines[index].strip() == "---":
            return lines[index + 1 :]
    return lines


def chunk_markdown(
    document: KnowledgeDocument,
    document_number: int,
) -> list[KnowledgeChunk]:
    chunks: list[KnowledgeChunk] = []
    heading_stack: list[str] = []
    section_lines: list[str] = []
    section_number = 0
    in_fence = False

    def flush_section() -> None:
        nonlocal section_number
        content = "\n".join(section_lines).strip()
        section_lines.clear()
        if not content:
            return

        section_number += 1
        heading_path = (
            " > ".join(heading_stack) if heading_stack else document.title
        )
        chunks.append(
            KnowledgeChunk(
                chunk_id=f"{document_number:02d}-{section_number:03d}",
                source_path=document.source_path,
                category=document.category,
                title=document.title,
                heading_path=heading_path,
                content=content,
            )
        )

    for line in _without_front_matter(document.text):
        stripped = line.strip()
        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fence = not in_fence
            section_lines.append(line)
            continue

        heading = None if in_fence else _heading_from_line(line)
        if heading is None:
            section_lines.append(line)
            continue

        flush_section()
        level, heading_text = heading
        del heading_stack[level - 1 :]
        heading_stack.append(heading_text)

    flush_section()
    return chunks


def chunk_documents(
    documents: list[KnowledgeDocument],
) -> list[KnowledgeChunk]:
    chunks: list[KnowledgeChunk] = []
    for document_number, document in enumerate(documents, start=1):
        chunks.extend(chunk_markdown(document, document_number))
    return chunks

Validate every original archive member

A dedicated package script makes the archive check repeatable. It prints the evidence needed to catch a missing file, a changed count, or a lost source path.

• Select the scripts directory in the VS Code workspace file tree.
• Create an empty file named __init__.py inside scripts.
• Confirm scripts/__init__.py appears in the workspace file tree.
• Create scripts/inspect_knowledge_base.py with the new-file icon.

The empty package marker allows Python to run the inspector through the scripts package. The inspector defaults to the private archive path and the required count of 19.

• Add the parser imports and command-line options by pasting this code into scripts/inspect_knowledge_base.py:

from __future__ import annotations

import argparse
from collections import Counter
from pathlib import Path

from kb_core import load_markdown_documents


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Inspect and validate the original MTS knowledge archive."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
        help="Path to the original knowledge-base.zip file.",
    )
    parser.add_argument(
        "--expected-count",
        type=int,
        default=19,
        help="Expected number of Markdown documents.",
    )
    return parser

What does the parser configure?

• The archive option defaults to local_data/knowledge-base.zip.
• The count option defaults to 19 Markdown documents.
• The loader import keeps all archive-reading logic inside kb_core.py.

• Save scripts/inspect_knowledge_base.py.
• Confirm build_parser() returns the configured parser.

Is the loader import unresolved?

Confirm kb_core.py sits beside requirements.txt. Confirm the inspection file sits inside scripts.

Ask for help checking the module layout.

• Add the inspection routine two blank lines below build_parser() by pasting this code:

def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)

    print(f"Archive: {args.zip}")
    print(f"Markdown documents: {len(documents)}")
    print("\nCategories:")
    category_counts = Counter(document.category for document in documents)
    for category, count in sorted(category_counts.items()):
        print(f"  {category}: {count}")

    print("\nMarkdown members:")
    for document in documents:
        print(f"  [{document.category}] {document.source_path}")

    if len(documents) != args.expected_count:
        raise SystemExit(
            "VALIDATION FAILED: "
            f"expected {args.expected_count} Markdown files, "
            f"found {len(documents)}"
        )

    print("\nVALIDATION PASSED")


if __name__ == "__main__":
    main()

What does the inspection prove?

• Counter groups loaded documents by their preserved category.
• The member loop prints every exact source path from the archive.
• The count check stops with VALIDATION FAILED unless the loader finds exactly 19 Markdown files.

• Save scripts/inspect_knowledge_base.py.
• Confirm the final guard calls main() only when the file runs as a script.

Seeing syntax warnings in the inspector?

Check the parentheses around SystemExit. Keep the final module guard outside main().

Ask for help checking the inspection script.

✔️ Awesome, I've got everything!

Great. Double check that scripts/__init__.py exists and that you saved the inspection script.

ⓧ I'd like to double check the full code

Compare your complete scripts/inspect_knowledge_base.py with this reference.

from __future__ import annotations

import argparse
from collections import Counter
from pathlib import Path

from kb_core import load_markdown_documents


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Inspect and validate the original MTS knowledge archive."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
        help="Path to the original knowledge-base.zip file.",
    )
    parser.add_argument(
        "--expected-count",
        type=int,
        default=19,
        help="Expected number of Markdown documents.",
    )
    return parser


def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)

    print(f"Archive: {args.zip}")
    print(f"Markdown documents: {len(documents)}")
    print("\nCategories:")
    category_counts = Counter(document.category for document in documents)
    for category, count in sorted(category_counts.items()):
        print(f"  {category}: {count}")

    print("\nMarkdown members:")
    for document in documents:
        print(f"  [{document.category}] {document.source_path}")

    if len(documents) != args.expected_count:
        raise SystemExit(
            "VALIDATION FAILED: "
            f"expected {args.expected_count} Markdown files, "
            f"found {len(documents)}"
        )

    print("\nVALIDATION PASSED")


if __name__ == "__main__":
    main()

Before you run the inspection, do you expect the unchanged archive to satisfy the expected document count?

• Return to the integrated PowerShell terminal from the previous step.
• Validate the original archive by running this command:

python -m scripts.inspect_knowledge_base

You should see Markdown documents: 19 near the top of the output.

You should also see category counts followed by every original .md member path.

The final line should read VALIDATION PASSED.

Did the archive validation fail?

• If the output mentions Archive not found, confirm the file path is exactly local_data/knowledge-base.zip.
• If the output mentions Invalid ZIP archive, replace the copied file with the unchanged organizer archive.
• If the output mentions invalid UTF-8, use the original organizer file without editing its members.
• If you see VALIDATION FAILED, confirm you copied the archive that contains exactly 19 Markdown documents.

Ask for help diagnosing the inspection output.

That is the evidence chain secured. Every later match can trace back to an unchanged archive member.

Your archive is verified at the document level. Next, you will search whole documents and see why broad matches cannot identify the precise supporting section.

Search Whole Documents

Your Python loader has proved that the original archive contains 19 readable Markdown documents. The next question is whether complete documents provide precise evidence for a demo question.

You will build a lexical baseline with TF-IDF. You will rank each document with cosine similarity. The result tests how much precision a whole-document search unit can provide.

In this step, get ready to:

• Copy the organizer's demo questions into the private input directory.
• Build reusable question loading and document retrieval modules.
• Run the first demo question against all 19 complete documents.

Load and validate the demo questions

The organizer's CSV supplies the questions that drive this retrieval check. A strict loader catches missing columns or empty values before they reach the ranking code.

• Drag the unchanged organizer file named demo_questions.csv into the local_data directory in the workspace file tree.
• Select local_data/demo_questions.csv in the workspace file tree.

You should see id,text as the required header row. You should also see at least one non-empty question row.

• Create questions_core.py at the top level of mts-energytool-consultant by using the file creation control above the workspace file tree.
• Define the question record by adding this first code chunk:

from __future__ import annotations

import csv
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class DemoQuestion:
    question_id: str
    text: str

What Does This Code Do?

• The imports provide CSV parsing plus filesystem path handling.
• The frozen DemoQuestion record keeps each organizer ID paired with its question text.
• A frozen record prevents later code from changing a loaded question accidentally.

• Save questions_core.py.
• Confirm that questions_core.py remains visible at the top level of the workspace file tree.

File Appearing in the Wrong Directory?

Check that questions_core.py sits beside kb_core.py. Move the file out of scripts if it was created there.

Ask for help if the workspace structure does not match. Help me place questions_core.py at the top level of my Python workspace.

• Add the CSV validation function below DemoQuestion by pasting this code:

def load_demo_questions(csv_path: Path) -> list[DemoQuestion]:
    if not csv_path.is_file():
        raise FileNotFoundError(f"Questions file not found: {csv_path}")

    with csv_path.open("r", encoding="utf-8-sig", newline="") as handle:
        reader = csv.DictReader(handle)
        fieldnames = set(reader.fieldnames or [])
        required = {"id", "text"}
        missing = required - fieldnames
        if missing:
            missing_list = ", ".join(sorted(missing))
            raise ValueError(f"Questions CSV is missing columns: {missing_list}")

        questions: list[DemoQuestion] = []
        for row_number, row in enumerate(reader, start=2):
            question_id = (row.get("id") or "").strip()
            text = (row.get("text") or "").strip()
            if not question_id or not text:
                raise ValueError(
                    f"Questions CSV row {row_number} has an empty id or text"
                )
            questions.append(DemoQuestion(question_id=question_id, text=text))

    if not questions:
        raise ValueError("Questions CSV contains no data rows")
    return questions

How Is the CSV Protected?

• The file check stops immediately when the organizer input is missing.
• The utf-8-sig encoding accepts UTF-8 content with an optional byte order mark.
• The header check requires both id and text.
• The row checks reject empty identifiers or empty question text.

• Save questions_core.py.
• Confirm that the editor shows the complete load_demo_questions() function without syntax diagnostics.

Seeing CSV Validation Problems?

Confirm that the file is named demo_questions.csv. Confirm that its first row uses the exact id,text columns.

Check every data row for a value in both columns. Help me diagnose my demo question CSV structure.

✔️ Awesome, I've got everything!

Your organizer CSV is private under local_data. Your question loader now validates every required field.

ⓧ I'd like to double check the full code

Compare your complete questions_core.py file with this reference.

from __future__ import annotations

import csv
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class DemoQuestion:
    question_id: str
    text: str


def load_demo_questions(csv_path: Path) -> list[DemoQuestion]:
    if not csv_path.is_file():
        raise FileNotFoundError(f"Questions file not found: {csv_path}")

    with csv_path.open("r", encoding="utf-8-sig", newline="") as handle:
        reader = csv.DictReader(handle)
        fieldnames = set(reader.fieldnames or [])
        required = {"id", "text"}
        missing = required - fieldnames
        if missing:
            missing_list = ", ".join(sorted(missing))
            raise ValueError(f"Questions CSV is missing columns: {missing_list}")

        questions: list[DemoQuestion] = []
        for row_number, row in enumerate(reader, start=2):
            question_id = (row.get("id") or "").strip()
            text = (row.get("text") or "").strip()
            if not question_id or not text:
                raise ValueError(
                    f"Questions CSV row {row_number} has an empty id or text"
                )
            questions.append(DemoQuestion(question_id=question_id, text=text))

    if not questions:
        raise ValueError("Questions CSV contains no data rows")
    return questions

Build the whole-document retriever

A retriever needs a consistent record for every searchable item. The same ranking class can then score complete documents or other evidence units without changing its core algorithm.

Why Use TF-IDF Here?

TF-IDF creates an inspectable lexical baseline for exact product terms and numeric language. scikit-learn stores the 19 documents in a sparse matrix that fits comfortably in memory.

This baseline keeps archive parsing separate from model behavior. Every score comes from vocabulary overlap that you can inspect.

• Create retrieval_core.py at the top level of mts-energytool-consultant by using the file creation control above the workspace file tree.
• Add the imports plus the shared search records by pasting this code:

from __future__ import annotations

from collections.abc import Sequence
from dataclasses import dataclass

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

from kb_core import KnowledgeDocument


@dataclass(frozen=True)
class SearchItem:
    item_id: str
    source_path: str
    category: str
    title: str
    heading_path: str
    content: str
    search_text: str


@dataclass(frozen=True)
class SearchResult:
    item: SearchItem
    score: float

What Are the Search Records?

• SearchItem keeps searchable text beside the metadata needed to inspect a match.
• SearchResult pairs one item with its cosine similarity score.
• The imports provide sparse TF-IDF vectors plus pairwise scoring.

• Save retrieval_core.py.
• Confirm that SearchItem and SearchResult appear as separate classes in the editor.

Seeing an Import Diagnostic?

Confirm that VS Code still uses the selected .venv. Confirm that retrieval_core.py sits beside kb_core.py.

Ask for help if the scikit-learn imports remain unresolved. Help me diagnose the imports in retrieval_core.py.

• Add the retriever constructor below SearchResult by pasting this code:

class TfidfRetriever:
    def __init__(self, items: Sequence[SearchItem]) -> None:
        if not items:
            raise ValueError("Cannot build a retriever without search items")

        self._items = tuple(items)
        self._vectorizer = TfidfVectorizer(
            lowercase=True,
            ngram_range=(1, 2),
            sublinear_tf=True,
        )
        self._matrix = self._vectorizer.fit_transform(
            item.search_text for item in self._items
        )

How Is the Index Built?

• The constructor rejects an empty collection because there would be nothing to index.
• The vectorizer normalizes text to lowercase.
• The ngram_range setting captures single words plus two-word phrases.
• The fit_transform() call learns the vocabulary and builds the document matrix once.

• Save retrieval_core.py.
• Confirm that TfidfRetriever contains an indented __init__() method.

Constructor Indentation Looking Wrong?

Keep __init__() indented inside TfidfRetriever. Keep the vectorizer settings indented inside the constructor.

Ask for help if Python indentation markers look inconsistent. Help me fix the TfidfRetriever constructor indentation.

• Add the search method inside TfidfRetriever below __init__() by pasting this indented code:

    def search(
        self,
        query: str,
        top_k: int = 3,
        min_score: float = 0.02,
    ) -> list[SearchResult]:
        clean_query = query.strip()
        if not clean_query:
            raise ValueError("Search query cannot be empty")
        if top_k <= 0:
            raise ValueError("top_k must be greater than zero")

        query_vector = self._vectorizer.transform([clean_query])
        if query_vector.nnz == 0:
            return []

        scores = cosine_similarity(query_vector, self._matrix)[0]
        ranked_indices = scores.argsort()[::-1]

        results: list[SearchResult] = []
        for index in ranked_indices:
            score = float(scores[index])
            if score < min_score:
                continue
            results.append(SearchResult(item=self._items[index], score=score))
            if len(results) == top_k:
                break
        return results

How Are Results Ranked?

• The method validates the query plus the requested result count.
• The learned vectorizer transforms the query into the same feature space as the documents.
• A query with no known features returns an empty result list.
• Cosine similarity ranks known matches before the score threshold and result limit are applied.

• Save retrieval_core.py.
• Confirm that search() aligns with __init__() inside the class.

Search Method Outside the Class?

Indent search() to the same level as __init__(). The method must remain inside TfidfRetriever.

Ask for help if the class boundary is unclear. Help me place search() inside TfidfRetriever.

• Add the whole-document adapter below the TfidfRetriever class by pasting this code:

def make_document_items(
    documents: Sequence[KnowledgeDocument],
) -> list[SearchItem]:
    items: list[SearchItem] = []
    for number, document in enumerate(documents, start=1):
        search_text = "\n".join(
            (
                document.title,
                document.category,
                document.source_path,
                document.text,
            )
        )
        items.append(
            SearchItem(
                item_id=f"document-{number:02d}",
                source_path=document.source_path,
                category=document.category,
                title=document.title,
                heading_path="(whole document)",
                content=document.text,
                search_text=search_text,
            )
        )
    return items

What Gets Indexed?

• The adapter creates one search item for each complete Markdown document.
• The searchable text combines the title plus the category.
• It also includes the original archive path plus the complete document text.
• The heading label records that each result represents a whole document.

• Save retrieval_core.py.
• Confirm that make_document_items() begins at the left edge below the class.

Adapter Nested Inside the Class?

Remove leading indentation from make_document_items(). It is a module-level function outside TfidfRetriever.

Ask for help if the function still appears nested. Help me fix the make_document_items() indentation.

• Complete the reusable adapters below make_document_items() by pasting this code:

def make_chunk_items(chunks: Sequence[object]) -> list[SearchItem]:
    items: list[SearchItem] = []
    for chunk in chunks:
        items.append(
            SearchItem(
                item_id=chunk.chunk_id,
                source_path=chunk.source_path,
                category=chunk.category,
                title=chunk.title,
                heading_path=chunk.heading_path,
                content=chunk.content,
                search_text=chunk.search_text,
            )
        )
    return items


def compact_excerpt(text: str, limit: int = 240) -> str:
    compact = " ".join(text.split())
    if len(compact) <= limit:
        return compact
    return compact[: limit - 1].rstrip() + "…"

Why Add These Helpers?

• The chunk adapter converts existing knowledge chunks into the shared search record.
• The excerpt helper collapses whitespace for readable terminal output.
• Long excerpts end with an ellipsis while their complete content remains inside the search item.

• Save retrieval_core.py.
• Confirm that make_chunk_items() and compact_excerpt() appear as module-level functions.

Excerpt Helper Showing a Syntax Diagnostic?

Check the quotes around the newline-free text expression. Check that the final ellipsis remains inside quotes.

Ask for help if the final helper still has a syntax issue. Help me compare compact_excerpt() with the supplied code.

✔️ Awesome, I've got everything!

Your shared retriever can now index complete documents. It can also return scored records with inspectable metadata.

ⓧ I'd like to double check the full code

Compare your complete retrieval_core.py file with this reference.

from __future__ import annotations

from collections.abc import Sequence
from dataclasses import dataclass

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

from kb_core import KnowledgeDocument


@dataclass(frozen=True)
class SearchItem:
    item_id: str
    source_path: str
    category: str
    title: str
    heading_path: str
    content: str
    search_text: str


@dataclass(frozen=True)
class SearchResult:
    item: SearchItem
    score: float


class TfidfRetriever:
    def __init__(self, items: Sequence[SearchItem]) -> None:
        if not items:
            raise ValueError("Cannot build a retriever without search items")

        self._items = tuple(items)
        self._vectorizer = TfidfVectorizer(
            lowercase=True,
            ngram_range=(1, 2),
            sublinear_tf=True,
        )
        self._matrix = self._vectorizer.fit_transform(
            item.search_text for item in self._items
        )

    def search(
        self,
        query: str,
        top_k: int = 3,
        min_score: float = 0.02,
    ) -> list[SearchResult]:
        clean_query = query.strip()
        if not clean_query:
            raise ValueError("Search query cannot be empty")
        if top_k <= 0:
            raise ValueError("top_k must be greater than zero")

        query_vector = self._vectorizer.transform([clean_query])
        if query_vector.nnz == 0:
            return []

        scores = cosine_similarity(query_vector, self._matrix)[0]
        ranked_indices = scores.argsort()[::-1]

        results: list[SearchResult] = []
        for index in ranked_indices:
            score = float(scores[index])
            if score < min_score:
                continue
            results.append(SearchResult(item=self._items[index], score=score))
            if len(results) == top_k:
                break
        return results


def make_document_items(
    documents: Sequence[KnowledgeDocument],
) -> list[SearchItem]:
    items: list[SearchItem] = []
    for number, document in enumerate(documents, start=1):
        search_text = "\n".join(
            (
                document.title,
                document.category,
                document.source_path,
                document.text,
            )
        )
        items.append(
            SearchItem(
                item_id=f"document-{number:02d}",
                source_path=document.source_path,
                category=document.category,
                title=document.title,
                heading_path="(whole document)",
                content=document.text,
                search_text=search_text,
            )
        )
    return items


def make_chunk_items(chunks: Sequence[object]) -> list[SearchItem]:
    items: list[SearchItem] = []
    for chunk in chunks:
        items.append(
            SearchItem(
                item_id=chunk.chunk_id,
                source_path=chunk.source_path,
                category=chunk.category,
                title=chunk.title,
                heading_path=chunk.heading_path,
                content=chunk.content,
                search_text=chunk.search_text,
            )
        )
    return items


def compact_excerpt(text: str, limit: int = 240) -> str:
    compact = " ".join(text.split())
    if len(compact) <= limit:
        return compact
    return compact[: limit - 1].rstrip() + "…"

Run the whole-document search

The runnable script connects your validated organizer inputs to the retriever. It selects the first demo question by default so every later comparison can start from the same query.

• Create scripts/search_documents.py inside the existing scripts directory by using the file creation control above the workspace file tree.
• Add the imports plus command-line options by pasting this code:

from __future__ import annotations

import argparse
from pathlib import Path

from kb_core import load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, compact_excerpt, make_document_items


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Search complete Markdown documents with TF-IDF."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument("--question-number", type=int, default=1)
    parser.add_argument("--top-k", type=int, default=3)
    return parser

What Do These Options Control?

• The default archive path points to the unchanged organizer ZIP.
• The default question path points to the private organizer CSV.
• The default question number selects the first data row.
• The default result limit requests three matches.

• Save scripts/search_documents.py.
• Confirm that search_documents.py appears beside inspect_knowledge_base.py inside scripts.

Script Created Outside the Package?

Move search_documents.py into the existing scripts directory. Keep scripts/__init__.py in the same directory.

Ask for help if the package layout differs. Help me place search_documents.py inside the scripts package.

• Add the main search setup below build_parser() by pasting this code:

def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    questions = load_demo_questions(args.questions)

    if not 1 <= args.question_number <= len(questions):
        raise SystemExit(
            f"question-number must be between 1 and {len(questions)}"
        )

    question = questions[args.question_number - 1]
    retriever = TfidfRetriever(make_document_items(documents))
    results = retriever.search(question.text, top_k=args.top_k)

    print(f"Question {question.question_id}: {question.text}")
    print(f"Search unit: whole documents ({len(documents)} indexed)\n")

    if not results:
        print("No evidence found.")
        return


if __name__ == "__main__":
    main()

How Is the Search Prepared?

• The script loads the original archive plus the organizer questions.
• The range check prevents an unavailable question number from being selected.
• The document adapter creates one item for every complete document.
• The retriever scores the selected question before the output is formatted.

• Save scripts/search_documents.py.
• Switch back to the integrated terminal that uses the selected .venv.
• Confirm the input and indexing path by running this command:

python -m scripts.search_documents

You should see the first question plus a line reporting 19 indexed whole documents. This proves the loader and retriever are connected to the organizer inputs.

Search Setup Failing?

A missing-file message points to either local_data/knowledge-base.zip or local_data/demo_questions.csv. Confirm that both unchanged organizer files are present.

An import failure usually means the terminal is outside mts-energytool-consultant or is using a different interpreter. Help me diagnose the whole-document search setup.

• Find the final if __name__ == "__main__": guard in scripts/search_documents.py.
• Insert the result formatting loop directly above that guard by pasting this code:

    for rank, result in enumerate(results, start=1):
        item = result.item
        print(f"{rank}. score={result.score:.4f}")
        print(f"   source={item.source_path}")
        print(f"   heading={item.heading_path}")
        print(f"   characters={len(item.content)}")
        print(f"   excerpt={compact_excerpt(item.content)}\n")

What Will Each Match Show?

• The rank and score show the retrieval order.
• The source preserves the original path inside the organizer archive.
• The heading identifies the search unit used for the match.
• The character count plus excerpt expose the size of the retrieved evidence.

• Save scripts/search_documents.py.

Before you run the completed search, do you think each result will identify one precise supporting section?

• Run the completed whole-document search with this command:

python -m scripts.search_documents

You should see up to three ranked matches for the first demo question. Every match shows (whole document) as its heading.

The large character counts reveal the intended limitation. A relevant document can rank well while leaving the exact supporting section unidentified.

What Did This Baseline Prove?

The lexical search can connect a demo question to relevant source files. Its evidence unit is still an entire document.

That oversized unit makes the supporting passage difficult to inspect. You have now seen the retrieval gap directly.

No Ranked Matches or Missing Details?

Confirm that demo_questions.csv contains the organizer's real questions. Confirm that the formatting loop sits inside main() above the final guard.

Compare all three new Python files with their full-code references if output fields are missing. Help me diagnose missing whole-document search results.

✔️ Awesome, I've got everything!

That limitation is now visible in your terminal. Your lexical baseline retrieves source files while exposing why complete documents are too broad as evidence.

ⓧ I'd like to double check the full code

Compare your complete scripts/search_documents.py file with this reference.

from __future__ import annotations

import argparse
from pathlib import Path

from kb_core import load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, compact_excerpt, make_document_items


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Search complete Markdown documents with TF-IDF."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument("--question-number", type=int, default=1)
    parser.add_argument("--top-k", type=int, default=3)
    return parser


def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    questions = load_demo_questions(args.questions)

    if not 1 <= args.question_number <= len(questions):
        raise SystemExit(
            f"question-number must be between 1 and {len(questions)}"
        )

    question = questions[args.question_number - 1]
    retriever = TfidfRetriever(make_document_items(documents))
    results = retriever.search(question.text, top_k=args.top_k)

    print(f"Question {question.question_id}: {question.text}")
    print(f"Search unit: whole documents ({len(documents)} indexed)\n")

    if not results:
        print("No evidence found.")
        return

    for rank, result in enumerate(results, start=1):
        item = result.item
        print(f"{rank}. score={result.score:.4f}")
        print(f"   source={item.source_path}")
        print(f"   heading={item.heading_path}")
        print(f"   characters={len(item.content)}")
        print(f"   excerpt={compact_excerpt(item.content)}\n")


if __name__ == "__main__":
    main()

You have a working TF-IDF baseline plus direct evidence of its document-level limitation. Next, you will narrow those oversized matches into heading-aware sections while preserving their source metadata.

Retrieve Heading-Aware Chunks

Your whole-document TF-IDF search in Visual Studio Code found relevant files. Its large matches still hid the exact supporting heading.

This step uses the heading-aware chunks already prepared in kb_core.py. You will connect those chunks to the retriever. You will then print evidence at section level.

In this step, get ready to:

• Trace how the Markdown parser preserves headings and complete sections.
• Create a runnable search script for heading-aware chunks.
• Compare section-level evidence with the earlier whole-document results.

Trace the chunk boundaries

A chunk represents one complete Markdown section. Its metadata keeps the evidence connected to the original document.

• Switch back to kb_core.py from earlier.
• Find def _without_front_matter(text: str) -> list[str]:.
• Confirm that front matter is removed only when opening and closing --- lines exist.
• Find def chunk_markdown(.
• Follow the in_fence flag to confirm that heading-like lines inside fenced code stay in the section content.
• Follow flush_section() to confirm that every section keeps all of its lines.

How does the chunk contract work?

Each KnowledgeChunk stores a stable chunk_id. It also carries the original source path and MTS category.

The heading stack becomes a breadcrumb in heading_path. Section lines remain together until the next real heading.

The search_text property combines metadata with content. This gives the retriever both document context and section evidence.

Create the chunk search script

The chunker and retriever already expose the data needed for section-level search. The new script coordinates those reusable pieces.

• Hover over the scripts folder in the Explorer sidebar and click the New File icon.
• Type search_chunks.py and press Enter.

You will see the blank search_chunks.py file listed inside scripts.

• Add the imports and command-line options by pasting this code into scripts/search_chunks.py:

from __future__ import annotations

import argparse
from pathlib import Path

from kb_core import chunk_documents, load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, compact_excerpt, make_chunk_items


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Search heading-aware Markdown chunks with TF-IDF."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument("--question-number", type=int, default=1)
    parser.add_argument("--top-k", type=int, default=3)
    return parser

What does this code prepare?

• The imports reuse the existing archive loader and chunker.
• The question loader keeps the same organizer-supplied demo set.
• The retriever imports convert chunks into searchable records.
• The parser defaults point to the unchanged ZIP and demo question file.

• Save scripts/search_chunks.py.

The top of the file now shows the reusable imports followed by four command-line options.

Seeing an underlined import?

Confirm that search_chunks.py sits inside the existing scripts folder. Confirm that your selected interpreter still points to the workspace .venv.

Check that kb_core.py and retrieval_core.py remain at the top level of mts-energytool-consultant.

Help me resolve the underlined imports in my chunk search script.

• Add the initial search flow below build_parser() by pasting this code:

def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    chunks = chunk_documents(documents)
    questions = load_demo_questions(args.questions)

    if not 1 <= args.question_number <= len(questions):
        raise SystemExit(
            f"question-number must be between 1 and {len(questions)}"
        )

    question = questions[args.question_number - 1]
    retriever = TfidfRetriever(make_chunk_items(chunks))
    results = retriever.search(question.text, top_k=args.top_k)

    print(f"Question {question.question_id}: {question.text}")
    print(f"Search unit: heading-aware chunks ({len(chunks)} indexed)\n")


if __name__ == "__main__":
    main()

What does the search flow do?

• The script loads documents directly from the original ZIP.
• The chunk_documents() call converts each document into heading-aware sections.
• The make_chunk_items() call maps chunk metadata into SearchItem records.
• The retriever searches the first demo question by default.

• Save scripts/search_chunks.py.

Before you run this checkpoint, how many heading-aware chunks do you think the 19 documents contain?

• Test the connection between the loader and retriever by running this command:

python -m scripts.search_chunks

What does this checkpoint prove?

The terminal prints the selected demo question. It also prints the number of heading-aware chunks indexed.

This successful run proves that the archive loader and chunker feed valid records into the retriever.

Did the checkpoint stop?

Confirm that the integrated terminal is still inside mts-energytool-consultant. The script resolves its input paths from that folder.

Confirm that the unchanged ZIP and organizer CSV remain inside local_data.

Help me debug the chunk search checkpoint.

Print the ranked sections

The retriever now has ranked results. The final display logic exposes the metadata that makes each result auditable.

• Return to scripts/search_chunks.py.
• Place your cursor on the blank line immediately before if __name__ == "__main__":.
• Add the result display logic by pasting this code:

    if not results:
        print("No evidence found.")
        return

    for rank, result in enumerate(results, start=1):
        item = result.item
        print(f"{rank}. score={result.score:.4f}")
        print(f"   source={item.source_path}")
        print(f"   category={item.category}")
        print(f"   heading={item.heading_path}")
        print(f"   excerpt={compact_excerpt(item.content)}\n")

What does the display logic reveal?

• The empty-result branch reports when the retriever finds no evidence.
• The ranking loop prints each similarity score.
• The source and category fields preserve the original archive context.
• The heading breadcrumb identifies the supporting section.
• The compact excerpt provides a readable evidence preview.

• Save scripts/search_chunks.py.

Before you run the completed script, which fields do you expect to replace the earlier character counts?

• Verify the heading-aware search by running this command:

python -m scripts.search_chunks

What should you see?

You will see the same first demo question used by the whole-document search. Up to three ranked matches follow.

Each match shows a score and original source path. It also shows the MTS category and heading breadcrumb.

The compact excerpt comes from one section. This gives you precise evidence instead of an oversized document.

You have closed the evidence gap from the earlier search. Every result now points to a specific section that you can inspect.

Missing headings or ranked results?

Confirm that the result loop remains inside main(). Every line in the pasted block starts with four spaces.

Confirm that make_chunk_items(chunks) is passed to TfidfRetriever. Passing document items would restore the broad search behavior.

Help me debug missing chunk headings or ranked results.

✔️ Awesome, I've got everything!

Great. Double check that you saved scripts/search_chunks.py after seeing section-level results.

ⓧ I'd like to double check the full code

Compare your complete scripts/search_chunks.py file with this reference.

from __future__ import annotations

import argparse
from pathlib import Path

from kb_core import chunk_documents, load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, compact_excerpt, make_chunk_items


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Search heading-aware Markdown chunks with TF-IDF."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument("--question-number", type=int, default=1)
    parser.add_argument("--top-k", type=int, default=3)
    return parser


def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    chunks = chunk_documents(documents)
    questions = load_demo_questions(args.questions)

    if not 1 <= args.question_number <= len(questions):
        raise SystemExit(
            f"question-number must be between 1 and {len(questions)}"
        )

    question = questions[args.question_number - 1]
    retriever = TfidfRetriever(make_chunk_items(chunks))
    results = retriever.search(question.text, top_k=args.top_k)

    print(f"Question {question.question_id}: {question.text}")
    print(f"Search unit: heading-aware chunks ({len(chunks)} indexed)\n")

    if not results:
        print("No evidence found.")
        return

    for rank, result in enumerate(results, start=1):
        item = result.item
        print(f"{rank}. score={result.score:.4f}")
        print(f"   source={item.source_path}")
        print(f"   category={item.category}")
        print(f"   heading={item.heading_path}")
        print(f"   excerpt={compact_excerpt(item.content)}\n")


if __name__ == "__main__":
    main()

How should this file compare?

This reference contains the parser and complete search flow. It also contains the ranked section display logic.

Your retriever can now identify precise supporting sections. Next up, you will run this evidence check across every demo question and record the results in a report.

Batch-Test the Demo Questions

Your heading-aware search now returns focused source sections for the first demo question. The next goal is to measure that retrieval behavior across the complete question set.

One successful query can hide weak coverage. A batch CSV report makes every result inspectable. An out-of-vocabulary probe also proves that TF-IDF can return no evidence when a query has no known features.

In this step, get ready to:

• Build a script that evaluates every demo question against one chunk index.
• Generate a diagnostic retrieval report containing each question's best candidate.
• Prove that an out-of-vocabulary query returns no evidence.

Build the evaluation script

The evaluator needs the loaders from earlier steps. It also needs a fixed probe whose text does not occur in the fitted vocabulary.

• In the VS Code file sidebar, select the scripts folder.
• Create a file named evaluate_demo_questions.py inside scripts.
• Paste the imports and probe definition into scripts/evaluate_demo_questions.py using this code:

from __future__ import annotations

import argparse
import csv
from pathlib import Path

from kb_core import chunk_documents, load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, make_chunk_items


PROBE_QUERY = "несуществующийтерминxyz"

What does this code prepare?

• The archive helpers load the original documents and convert them into heading-aware chunks.
• The question helper validates every row in the organizer's demo file.
• The retrieval helpers build one searchable chunk index.
• The PROBE_QUERY value provides a repeatable test for a query with no known vocabulary.

• Save scripts/evaluate_demo_questions.py.
• Confirm that evaluate_demo_questions.py is listed inside the scripts folder.

File missing from the scripts folder?

Check that the file is inside scripts under the mts-energytool-consultant workspace. Confirm that its name ends with .py.

Ask for help if the file appears outside the package. Help me place my evaluation script in the correct workspace folder.

Command-line options keep the evaluator repeatable. The defaults point to the existing organizer inputs and the generated report location.

• Place the cursor below the PROBE_QUERY definition.
• Add the argument parser using this code:

def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Create a retrieval report for every demo question."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument(
        "--output",
        type=Path,
        default=Path("runtime_data/retrieval_report.csv"),
    )
    parser.add_argument("--top-k", type=int, default=3)
    return parser

What does the parser control?

• The archive option identifies the unchanged knowledge ZIP.
• The questions option identifies the organizer's demo CSV.
• The output option identifies the report file inside runtime_data.
• The final option limits retrieval to three candidates per question.

• Save scripts/evaluate_demo_questions.py.
• Confirm that the editor now recognizes build_parser as a function in the file.

Parser function not recognized?

Check that def build_parser() begins at the left edge of the file. Make sure each line inside the function uses one consistent indentation level.

Ask for help if the function remains highlighted as invalid. Help me fix the parser function in my evaluation script.

Process every demo question

Building the chunk index once keeps every question on the same retrieval baseline. Each report row captures the best candidate while preserving the total number of returned candidates.

• Place the cursor below the build_parser function.
• Add the evaluator setup using this code:

def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    chunks = chunk_documents(documents)
    questions = load_demo_questions(args.questions)
    retriever = TfidfRetriever(make_chunk_items(chunks))

    args.output.parent.mkdir(parents=True, exist_ok=True)
    fieldnames = (
        "id",
        "text",
        "top_source",
        "top_heading",
        "top_score",
        "candidate_count",
    )

    questions_with_candidates = 0

What does this setup do?

• The script loads the archive and creates the same heading-aware chunks used by the previous search.
• One TfidfRetriever indexes those chunks for the complete batch.
• The output folder is created if it is missing.
• The field names define the exact columns written to the report.

• Save scripts/evaluate_demo_questions.py.
• Confirm that the editor now recognizes main as a second function.

Main function nested under the parser?

Align def main() with def build_parser() at the left edge. Keep the setup lines indented inside main.

Ask for help if the function boundaries still look unclear. Help me correct the function indentation in my evaluation script.

The report uses UTF-8 with a byte-order marker so its Russian text remains available to CSV readers. The header is written before any question rows.

• Place the cursor below questions_with_candidates = 0 inside main.
• Add the report writer using this code:

    with args.output.open(
        "w", encoding="utf-8-sig", newline=""
    ) as handle:
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        writer.writeheader()

How is the report started?

The context manager opens the configured report path for writing. DictWriter uses the earlier field order to create a consistent header.

• Save scripts/evaluate_demo_questions.py.
• Confirm that writer.writeheader() is nested inside the with block.

Writer lines aligned incorrectly?

Keep the with line inside main. Indent both writer lines one level farther than the with line.

Ask for help if the nesting remains invalid. Help me fix the CSV writer indentation.

Every validated question passes through the same retriever. An empty result produces blank best-match fields while keeping its candidate count visible.

• Place the cursor below writer.writeheader().
• Add the retrieval loop using this code:

        for question in questions:
            results = retriever.search(question.text, top_k=args.top_k)
            top_result = results[0] if results else None
            if results:
                questions_with_candidates += 1

            writer.writerow(
                {
                    "id": question.question_id,
                    "text": question.text,
                    "top_source": (
                        top_result.item.source_path if top_result else ""
                    ),
                    "top_heading": (
                        top_result.item.heading_path if top_result else ""
                    ),
                    "top_score": (
                        f"{top_result.score:.4f}" if top_result else ""
                    ),
                    "candidate_count": len(results),
                }
            )

How is each row built?

• The retriever returns up to the configured number of candidates for each question.
• The first result supplies the best source path and heading breadcrumb.
• The best score is formatted to four decimal places.
• An empty result keeps the source and score fields blank.

• Save scripts/evaluate_demo_questions.py.
• Confirm that the row dictionary ends with the candidate_count field.

Question loop outside the writer block?

Align the for question in questions: line with the two writer lines. Keep the row-building code inside that loop.

Ask for help if the nested blocks are still unclear. Help me align the batch retrieval loop correctly.

A progress line makes the batch visible while it runs. It shows the candidate count for every question plus the best score when one exists.

• Place the cursor below the completed writer.writerow call inside the question loop.
• Add the progress output using this code:

            print(
                f"{question.question_id}: candidates={len(results)} "
                f"top_score={top_result.score:.4f}"
                if top_result
                else f"{question.question_id}: candidates=0"
            )

What appears during the batch?

Questions with evidence show their candidate count plus the leading score. Questions without evidence show a candidate count of zero.

• Save scripts/evaluate_demo_questions.py.
• Confirm that the print call aligns with writer.writerow inside the question loop.

Progress output placed outside the loop?

Keep the print call at the same indentation level as writer.writerow. This placement gives every question its own terminal line.

Ask for help if only one progress line is produced. Help me place the progress output inside the question loop.

The final check searches for the fixed probe after every real question is processed. A zero-feature query returns an empty list before cosine ranking begins.

• Place the cursor below the completed with block.
• Add the probe validation and script entry point using this code:

    probe_results = retriever.search(PROBE_QUERY, top_k=args.top_k)
    print(
        f"\nQuestions with candidates: "
        f"{questions_with_candidates}/{len(questions)}"
    )
    print(f"Report: {args.output}")
    print(f"Out-of-vocabulary probe results: {len(probe_results)}")

    if probe_results:
        raise SystemExit(
            "VALIDATION FAILED: the out-of-vocabulary probe returned evidence"
        )
    print("VALIDATION PASSED")


if __name__ == "__main__":
    main()

How does the validation work?

• The probe is searched against the same fitted chunk index as the real questions.
• Any returned evidence stops the script with a validation failure.
• An empty probe result allows the script to print VALIDATION PASSED.
• The entry-point guard runs main() when the file is launched as a module.

• Save scripts/evaluate_demo_questions.py.
• Confirm that the entry-point guard starts at the left edge of the file.

Probe validation nested inside the CSV writer?

Align probe_results with the earlier setup lines inside main. Align the entry-point guard with both function definitions.

Ask for help if the file still shows an indentation problem. Help me fix the final indentation in my evaluation script.

✔️ Awesome, I've got everything!

Your batch evaluator is complete. Double check that scripts/evaluate_demo_questions.py is saved before running it.

ⓧ I'd like to double check the full code

Compare your saved file with this complete version.

from __future__ import annotations

import argparse
import csv
from pathlib import Path

from kb_core import chunk_documents, load_markdown_documents
from questions_core import load_demo_questions
from retrieval_core import TfidfRetriever, make_chunk_items


PROBE_QUERY = "несуществующийтерминxyz"


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Create a retrieval report for every demo question."
    )
    parser.add_argument(
        "--zip",
        type=Path,
        default=Path("local_data/knowledge-base.zip"),
    )
    parser.add_argument(
        "--questions",
        type=Path,
        default=Path("local_data/demo_questions.csv"),
    )
    parser.add_argument(
        "--output",
        type=Path,
        default=Path("runtime_data/retrieval_report.csv"),
    )
    parser.add_argument("--top-k", type=int, default=3)
    return parser


def main() -> None:
    args = build_parser().parse_args()
    documents = load_markdown_documents(args.zip)
    chunks = chunk_documents(documents)
    questions = load_demo_questions(args.questions)
    retriever = TfidfRetriever(make_chunk_items(chunks))

    args.output.parent.mkdir(parents=True, exist_ok=True)
    fieldnames = (
        "id",
        "text",
        "top_source",
        "top_heading",
        "top_score",
        "candidate_count",
    )

    questions_with_candidates = 0
    with args.output.open(
        "w", encoding="utf-8-sig", newline=""
    ) as handle:
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        writer.writeheader()

        for question in questions:
            results = retriever.search(question.text, top_k=args.top_k)
            top_result = results[0] if results else None
            if results:
                questions_with_candidates += 1

            writer.writerow(
                {
                    "id": question.question_id,
                    "text": question.text,
                    "top_source": (
                        top_result.item.source_path if top_result else ""
                    ),
                    "top_heading": (
                        top_result.item.heading_path if top_result else ""
                    ),
                    "top_score": (
                        f"{top_result.score:.4f}" if top_result else ""
                    ),
                    "candidate_count": len(results),
                }
            )
            print(
                f"{question.question_id}: candidates={len(results)} "
                f"top_score={top_result.score:.4f}"
                if top_result
                else f"{question.question_id}: candidates=0"
            )

    probe_results = retriever.search(PROBE_QUERY, top_k=args.top_k)
    print(
        f"\nQuestions with candidates: "
        f"{questions_with_candidates}/{len(questions)}"
    )
    print(f"Report: {args.output}")
    print(f"Out-of-vocabulary probe results: {len(probe_results)}")

    if probe_results:
        raise SystemExit(
            "VALIDATION FAILED: the out-of-vocabulary probe returned evidence"
        )
    print("VALIDATION PASSED")


if __name__ == "__main__":
    main()

What should match?

This reference combines the parser and batch evaluator in their final order. The indentation determines which statements run for each question and which statements run once after the report closes.

Run and inspect the report

The final run checks every organizer question and writes the diagnostic baseline. Before you run it, do you expect the synthetic probe to produce a candidate?

• Run the batch evaluator from the existing integrated terminal using this command:

python -m scripts.evaluate_demo_questions

What does this command prove?

The script builds one chunk index before processing the complete demo set. Each question receives up to three candidates from that shared index.

The final probe has no known TF-IDF features. Its empty result demonstrates that the retriever can decline to return evidence.

You should see one result line for every demo question. The final lines include the report location followed by Out-of-vocabulary probe results: 0 and VALIDATION PASSED.

Report missing or validation stopped?

• Confirm that the terminal is using the mts-energytool-consultant workspace.
• Check that knowledge-base.zip and demo_questions.csv remain inside local_data.
• Compare the indentation in scripts/evaluate_demo_questions.py with the complete reference.

Ask for help with the terminal output if the cause remains unclear. Help me debug my batch retrieval evaluation.

The generated report is a diagnostic baseline. Candidate presence shows lexical coverage. It does not prove that the candidate supports a correct answer.

• Expand runtime_data in the VS Code file sidebar.
• Select retrieval_report.csv to open it.
• Confirm that the report contains the id, text, top_source, top_heading, top_score, and candidate_count columns.
• Inspect any row with a low top_score value.
• Compare that row's question with its top_source and top_heading values.

How should you read the report?

The top_score field measures lexical similarity within this fitted index. A weaker score flags a row for closer inspection.

The candidate_count field records how many matches passed retrieval. Correctness still requires checking the source section itself.

You now have a repeatable baseline for every demo question. The retriever also knows when the fitted vocabulary provides no evidence.

Secret mission

Export the Chunk Catalog

Your retriever already creates a dependable catalog of heading-aware chunks in memory. In this mission, you will export that catalog as readable JSON so you can audit every boundary and reuse the same metadata in Project 2.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely inside your local mts-energytool-consultant workspace. No API call or paid service creates an ongoing cost.

Resources you used:

• A Python 3.11 virtual environment in .venv with scikit-learn 1.8.0 installed.
• Generated output in runtime_data.
• The diagnostic report at runtime_data/retrieval_report.csv.
• The exported chunk catalog at runtime_data/chunks.json.

Keep everything running

No action is needed. Choose this option if you want to continue directly into Project 2.

• Leave .venv in place so your selected environment remains ready.
• Leave runtime_data in place so your generated artifacts remain available.
• Keep the source files inside mts-energytool-consultant for future reference.
• Keep local_data protected by .gitignore so the organizer inputs remain outside version control.

Pause - I'll come back to this later

Pausing preserves every local artifact for your next session. No process or service needs to remain running.

• Close Visual Studio Code.
• Leave .venv in the workspace.
• Leave runtime_data in the workspace.
• Return to the mts-energytool-consultant workspace when you are ready to continue.

Delete - I don't want to use this again

Deleting these folders only removes reproducible local artifacts. Your source code stays intact. Your organizer inputs stay intact.

• Select .venv in the workspace file tree.
• Hold Ctrl while selecting runtime_data.
• Press the Delete key.
• Approve the deletion confirmation if one appears.
• Confirm that .venv is no longer listed.
• Confirm that runtime_data is no longer listed.

The source files remain available for reference. The organizer files remain protected inside local_data.

Nice Work!

Nice Work!

You did it! You built a verified retrieval foundation that traces every match to a precise MTS section.

You've learned how to:

• Validate all 19 Markdown documents directly inside the original ZIP archive. Preserve every original source path with its MTS category.
• Compare whole-document retrieval with heading-aware chunks. Use TF-IDF with cosine similarity to trace each result to its supporting section.
• Batch-test every demo question to generate a diagnostic retrieval report. Prove that an out-of-vocabulary query returns no evidence.
• Secret Mission: Export the complete chunk catalog as readable JSON. Preserve Russian text plus reusable metadata for Project 2.

Ready to quiz yourself?