Build a RAG Knowledge Base Locally

Build an offline RAG knowledge base for INFOBASIC-to-Java code translation.

Introduction

30 Second Summary

Every large organization has legacy code that works but is becoming harder to maintain. Translating it into a modern language is risky without a tool that truly understands the libraries and patterns unique to your codebase.

In this project, you will build a fully local RAG knowledge base that understands three domains: your Temenos documentation, your custom Java library APIs, and INFOBASIC programming references from the web. You will query it with natural language to get grounded answers about translating INFOBASIC code to Java.

What You'll Build

A fully offline, queryable knowledge base on your laptop that indexes Temenos documentation, Java API signatures, and INFOBASIC programming references into a single searchable vector store.

By the end of this project, you'll have:

  • A local Ollama installation running Qwen 2.5 Coder 7B for code-focused LLM inference and nomic-embed-text for document embedding, all running locally with no cloud dependencies.
  • A Python pipeline that extracts Java API signatures from .jar files using javap, ingests Temenos HTML and PDF guides, and scrapes INFOBASIC references from the web.
  • A persistent ChromaDB vector store with an interactive query interface and a subroutine mapping index pairing INFOBASIC T24 routines with their Java equivalents.
  • Secret Mission: run a pilot translation of a real INFOBASIC subroutine through the knowledge base and evaluate whether the LLM produces valid Java code grounded in your actual library APIs.

Are there any prerequisites?

This project assumes you have Python 3.14, OpenJDK 17, and VS Code already installed on your machine. You should also have basic Python experience.

Set Up Ollama and Pull Models

Your knowledge base needs two AI models running locally on your machine. One model generates code-aware responses when you ask questions. The other converts your documents into numerical vector embeddings that make them searchable by meaning.

Ollama is a tool that downloads and runs LLMs on your laptop. In this step, you'll install Ollama and pull down the two models your project needs.

In this step, get ready to:
  • Install Ollama on your Windows machine.
  • Pull the Qwen 2.5 Coder 7B and nomic-embed-text models.
  • Test the LLM with a sample query about INFOBASIC.
Install Ollama

Ollama runs LLMs locally and serves them through an API on your machine. Once installed, it runs as a background service that your Python code will connect to in later steps.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Install Ollama by running this command:
irm https://ollama.com/install.ps1 | iex

What does this command do?

irm downloads the official Ollama install script from the web. iex executes the downloaded script in your PowerShell session.

The script silently installs Ollama into your user directory (no admin rights required) and updates your session PATH so the ollama command works immediately.

Prefer a manual download?

You can also download the installer directly from ollama.com/download/windows and run the OllamaSetup.exe file.

  • Wait for the script to finish. You should see a message that says the install is complete.
  • Verify the installation by running:
ollama --version

✔️ I see a version number

You should see output like ollama version 0.16.3 (your version number may differ). Ollama is installed and running as a background service.

ⓧ Command not found

The install script updates PATH in the current session, but sometimes the update does not take effect.

  • Close the PowerShell window completely and open a fresh one.
  • Run ollama --version again.
  • If it still does not work, try restarting your computer, then opening a new PowerShell window.
  • As a last resort, download the installer manually from ollama.com/download/windows and run it.

What is Ollama?

Ollama runs LLMs locally on your machine and serves them through a REST API at http://localhost:11434. Your Python code will connect to this API later to send questions and receive AI-generated answers.

Everything runs offline on your laptop. No data leaves your machine, which is ideal for working with proprietary Temenos code.

Pull the code generation and embedding models

You need two models. Qwen 2.5 Coder 7B is a code-focused LLM that reads your questions and generates answers. nomic-embed-text is an embedding model that converts your documents into numerical vectors so they can be searched by meaning.

  • Pull the code generation model by running this command (this is a ~4.7GB download and may take several minutes):
ollama pull qwen2.5-coder:7b

What is Qwen 2.5 Coder 7B?

Qwen 2.5 Coder is a code-focused LLM from the Qwen family. The 7B (7 billion parameter) version fits comfortably in 32GB of RAM while still providing strong code reasoning. It has a 32K context window and supports 40+ programming languages.

  • Wait for the download to finish. You should see a progress bar followed by a confirmation.
  • Pull the embedding model by running:
ollama pull nomic-embed-text

Why two separate models?

Each model has a different job. Qwen 2.5 Coder 7B reads your questions and generates natural language answers. nomic-embed-text (~274MB) converts your documents into numerical vectors so ChromaDB can search them by meaning instead of exact keyword matches.

  • Wait for this download to finish (this one is much smaller at ~274MB).
  • Confirm both models are available by running:
ollama ls
  • You should see both qwen2.5-coder:7b and nomic-embed-text listed with their sizes.

A model is missing from the list?

  • Run the ollama pull command again for the missing model. Downloads can fail if your internet connection dropped during the transfer.
  • Make sure you have at least 5GB of free disk space. The two models together total roughly 5GB.

Why isn't my model showing up after running ollama pull?

Test the LLM with a sample query

Before wiring the models into your project, confirm the LLM can generate a useful response about INFOBASIC.

  • Test the LLM by running this command:
ollama run qwen2.5-coder:7b "What is a dynamic array in INFOBASIC?"

What should I see?

The model will take 30-60 seconds to respond on CPU. You should see a coherent explanation about INFOBASIC dynamic arrays. After the response finishes, Ollama drops you into an interactive chat session.

  • Once you see the response, type /bye and press Enter to exit the interactive session.

Response is very slow or times out?

  • CPU-only inference with the 7B model is expected to be slow (roughly 5-10 tokens per second). Give it at least 2 minutes before assuming something is wrong.
  • Close any memory-heavy applications (browsers with many tabs, Docker, IDEs) to free up RAM for the model.
  • If the command hangs with no output at all, open a new terminal and run ollama ls to confirm Ollama is still running.

Why is Ollama not generating a response to my query?

Your local LLM is up and running. Next up, you'll create the Python project and install the packages that wire everything together.

Set Up the Python Project and Install Dependencies

Ollama is running and both models are ready to go. Now you need a Python project that can talk to Ollama and store document embeddings locally.

In this step, you will create a project folder, install the RAG pipeline packages, and set up the folder structure that will hold your Temenos documentation, extracted .jar signatures, and scraped web content.

In this step, get ready to:
  • Create the project directory and a Python virtual environment.
  • Install the RAG pipeline packages.
  • Build the folder structure for your documents.
Create the project directory and virtual environment

Every Python project needs its own isolated environment so package versions don't collide with other projects on your machine. A virtual environment is a self-contained copy of Python and its packages inside your project folder.

  • In VS Code, open a new terminal by clicking Terminal in the top menu bar.
  • Click New Terminal.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder and move into it by running these commands:
mkdir infobasic-kb
cd infobasic-kb
  • Create a Python virtual environment by running:
python -m venv venv
  • Activate the virtual environment. The command differs by operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt now starts with (venv). This confirms the virtual environment is active.

Don't see (venv) in your prompt?

Make sure you ran the activation command for your operating system. On macOS, it uses source venv/bin/activate with forward slashes. On Windows, it uses venv\Scripts\activate with backslashes.

If you see a permissions error on Windows, try running this command in PowerShell first, then retry the activation command:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

help me activate my Python virtual environment

Install the Python packages

Your RAG pipeline relies on several packages that work together. LlamaIndex orchestrates the retrieval and generation loop. ChromaDB stores your document embeddings as a local vector database.

  • Install all required packages by running this command:
pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama llama-index-vector-stores-chroma chromadb beautifulsoup4 requests

What is each package for?

  • llama-index is the core RAG orchestration framework. It handles document loading, chunking, indexing, and querying.
  • llama-index-llms-ollama and llama-index-embeddings-ollama connect LlamaIndex to your local Ollama models for text generation and embedding.
  • llama-index-vector-stores-chroma integrates ChromaDB as the vector store backend.
  • chromadb is the local vector database that persists your embeddings to disk.
  • beautifulsoup4 parses HTML pages so you can extract clean text from web documentation.
  • requests handles HTTP requests for downloading web pages.
  • Verify the installation succeeded by running this import test:
python -c "from llama_index.llms.ollama import Ollama; print('OK')"

You should see OK printed in your terminal with no errors.

Seeing an import error?

Make sure your virtual environment is still active. Your prompt should start with (venv). If it does not, rerun the activation command from the previous substep.

If the error mentions a missing module, the pip install command may not have completed successfully. Scroll up in your terminal to check for error messages, then rerun the install command.

help me fix my LlamaIndex import error

Create the folder structure and add your documents

Your knowledge base will pull from three sources: Temenos user guides you already have, API signatures extracted from .jar files, and content scraped from the web. Each source needs its own folder.

  • Create the folder structure by running this command:

macOS

mkdir -p docs/temenos-guides docs/jar-signatures docs/scraped chroma_db

Windows

mkdir docs, docs\temenos-guides, docs\jar-signatures, docs\scraped, chroma_db

What is each folder for?

  • docs/temenos-guides/ holds your Temenos HTML and PDF user guide files.
  • docs/jar-signatures/ will store the API signatures you extract from your .jar files in the next step.
  • docs/scraped/ will hold INFOBASIC reference content scraped from the web.
  • chroma_db/ will store the persisted ChromaDB vector index.
  • Confirm the folders were created by running:
ls

You should see chroma_db, docs, and venv in the output.

  • Copy your Temenos HTML and PDF user guide files into the docs/temenos-guides/ folder. You can drag and drop them from Finder (macOS) or File Explorer (Windows) into the folder.
  • Place your .jar files in the infobasic-kb project root folder, or create a jars/ subfolder and place them there.

Which files do I need?

For the Temenos user guides, look for the HTML or PDF documentation that came with your T24 or Temenos Transact installation. These typically describe INFOBASIC syntax, subroutine usage, and configuration entries like PGM.FILE.

For the .jar files, use the library JARs from your Temenos Design Studio or T24 runtime. These contain the Java classes your translated code will call.

Don't have your Temenos files handy?

You can start with an empty docs/temenos-guides/ folder and add files later. The ingestion scripts you build in Step 4 will process whatever is in the folder at runtime.

help me find my Temenos documentation files

Your project environment is ready and loaded with packages. Next up, you will write a script that extracts Java API signatures from your .jar files using javap.

Extract API Signatures from .jar Files

Your Python environment is ready and all the packages are installed. Now it's time to feed the knowledge base its first batch of raw data: the API signatures hiding inside your Temenos Java libraries.

Your .jar files contain compiled Java bytecode that the LLM cannot read directly. You need to extract human-readable class and method signatures so the knowledge base knows exactly which classes exist in your libraries and how to call them.

In this step, get ready to:
  • Write a Python script that extracts class signatures from .jar files using javap.
  • Run the script against your Temenos .jar files.
  • Verify the extracted signatures match your Temenos classes.
Write the extraction script

The extraction script follows a two-stage approach. First, it lists every .class file inside a .jar using jar tf. Then it runs javap on each class to extract its full method signatures.

  • Create a new file called extract_jar_signatures.py in the infobasic-kb project folder.
  • Add the imports, configuration, and the class-listing function by copying this code into the file:
import subprocess
import os
import sys
import glob

# Directory where extracted signatures will be saved
OUTPUT_DIR = os.path.join("docs", "jar-signatures")

# Standard Java library prefixes to skip during extraction
SKIP_PREFIXES = [
    "java/", "javax/", "sun/", "com/sun/",
    "org/xml/", "org/w3c/", "jdk/"
]

def list_classes_in_jar(jar_path):
    """Run jar tf to list all .class files in a JAR, filtering out
    inner classes and standard Java library classes."""
    result = subprocess.run(
        ["jar", "tf", jar_path],
        capture_output=True, text=True
    )

    classes = []
    for line in result.stdout.splitlines():
        # Keep only .class files
        if not line.endswith(".class"):
            continue
        # Skip inner classes (filenames containing $)
        if "$" in os.path.basename(line):
            continue
        # Skip standard Java library classes
        if any(line.startswith(prefix) for prefix in SKIP_PREFIXES):
            continue
        classes.append(line)

    return classes

What does this code do?

jar tf lists every file inside a .jar archive. The function filters that list down to only .class files.

It skips two categories: inner classes (filenames containing $) which are implementation details, and standard Java library classes (packages like java/ and javax/). This keeps only your Temenos-specific classes.

SKIP_PREFIXES defines which package paths to ignore. If your .jar files include other third-party libraries you want to skip, add their package prefixes here.

  • Below the list_classes_in_jar function, add the signature extraction function:
def extract_signature(jar_path, class_entry):
    """Run javap on a single class to extract its full signature."""
    # Convert file path (com/temenos/MyClass.class) to fully qualified class name (com.temenos.MyClass)
    class_name = class_entry.replace("/", ".").replace(".class", "")

    # Use -p flag to show all members including private
    result = subprocess.run(
        ["javap", "-cp", jar_path, "-p", class_name],
        capture_output=True, text=True
    )

    return class_name, result.stdout

What does javap do here?

javap is the Java class file disassembler that ships with the JDK. The -cp flag tells it to look for classes inside the specified .jar file.

The -p flag shows all classes and members, including private ones. This gives you the complete picture of every field and method in each class, which is exactly what the LLM needs to suggest accurate API calls.

  • Below the extract_signature function, add the main function that ties everything together:
def main():
    """Find all JARs, extract Temenos-specific class signatures,
    and save each as a text file."""
    # Look for .jar files in the project root first, then jars/ subfolder
    jar_files = glob.glob("*.jar")
    if not jar_files:
        jar_files = glob.glob(os.path.join("jars", "*.jar"))

    if not jar_files:
        print("No .jar files found in project root or jars/ folder.")
        sys.exit(1)

    print(f"Found {len(jar_files)} JAR file(s)")
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    total_classes = 0

    for jar_path in jar_files:
        jar_name = os.path.basename(jar_path)
        print(f"\nProcessing: {jar_name}")

        # List all Temenos-specific classes in this JAR
        classes = list_classes_in_jar(jar_path)
        print(f"  Found {len(classes)} classes (after filtering)")

        for class_entry in classes:
            class_name, signature = extract_signature(jar_path, class_entry)

            if not signature.strip():
                continue

            # Use underscores in the filename to avoid path issues
            safe_filename = class_name.replace(".", "_") + ".txt"
            output_path = os.path.join(OUTPUT_DIR, safe_filename)

            with open(output_path, "w") as f:
                f.write(signature)

            total_classes += 1

    print(f"\nDone! Extracted {total_classes} class signatures to {OUTPUT_DIR}")


if __name__ == "__main__":
    main()

What does the main function do?

  • glob.glob("*.jar") searches for .jar files in the project root. If none are found, it checks the jars/ subfolder.
  • For each .jar, it lists classes, extracts each class's signature with javap, and saves the result as a .txt file.
  • Each output filename replaces dots with underscores (e.g. com_temenos_MyClass.txt) to avoid nested path issues.
  • Save extract_jar_signatures.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

import subprocess
import os
import sys
import glob

# Directory where extracted signatures will be saved
OUTPUT_DIR = os.path.join("docs", "jar-signatures")

# Standard Java library prefixes to skip during extraction
SKIP_PREFIXES = [
    "java/", "javax/", "sun/", "com/sun/",
    "org/xml/", "org/w3c/", "jdk/"
]

def list_classes_in_jar(jar_path):
    """Run jar tf to list all .class files in a JAR, filtering out
    inner classes and standard Java library classes."""
    result = subprocess.run(
        ["jar", "tf", jar_path],
        capture_output=True, text=True
    )

    classes = []
    for line in result.stdout.splitlines():
        # Keep only .class files
        if not line.endswith(".class"):
            continue
        # Skip inner classes (filenames containing $)
        if "$" in os.path.basename(line):
            continue
        # Skip standard Java library classes
        if any(line.startswith(prefix) for prefix in SKIP_PREFIXES):
            continue
        classes.append(line)

    return classes

def extract_signature(jar_path, class_entry):
    """Run javap on a single class to extract its full signature."""
    # Convert file path (com/temenos/MyClass.class) to fully qualified class name (com.temenos.MyClass)
    class_name = class_entry.replace("/", ".").replace(".class", "")

    # Use -p flag to show all members including private
    result = subprocess.run(
        ["javap", "-cp", jar_path, "-p", class_name],
        capture_output=True, text=True
    )

    return class_name, result.stdout

def main():
    """Find all JARs, extract Temenos-specific class signatures,
    and save each as a text file."""
    # Look for .jar files in the project root first, then jars/ subfolder
    jar_files = glob.glob("*.jar")
    if not jar_files:
        jar_files = glob.glob(os.path.join("jars", "*.jar"))

    if not jar_files:
        print("No .jar files found in project root or jars/ folder.")
        sys.exit(1)

    print(f"Found {len(jar_files)} JAR file(s)")
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    total_classes = 0

    for jar_path in jar_files:
        jar_name = os.path.basename(jar_path)
        print(f"\nProcessing: {jar_name}")

        # List all Temenos-specific classes in this JAR
        classes = list_classes_in_jar(jar_path)
        print(f"  Found {len(classes)} classes (after filtering)")

        for class_entry in classes:
            class_name, signature = extract_signature(jar_path, class_entry)

            if not signature.strip():
                continue

            # Use underscores in the filename to avoid path issues
            safe_filename = class_name.replace(".", "_") + ".txt"
            output_path = os.path.join(OUTPUT_DIR, safe_filename)

            with open(output_path, "w") as f:
                f.write(signature)

            total_classes += 1

    print(f"\nDone! Extracted {total_classes} class signatures to {OUTPUT_DIR}")


if __name__ == "__main__":
    main()
Run the extraction script against your .jar files

Make sure your Temenos .jar files are in the infobasic-kb project root or inside a jars/ subfolder. The script checks both locations automatically.

  • Run the extraction script by entering this command in your terminal:
python extract_jar_signatures.py

You should see output listing each JAR file being processed, the number of classes found after filtering, and a final count of total extracted signatures.

What should I see?

The script prints progress as it works. Each .jar file gets a line showing its name and how many Temenos-specific classes it contains. At the end, you see a summary like:

Done! Extracted 42 class signatures to docs\jar-signatures.

The exact number depends on how many classes your .jar files contain.

Seeing 'No .jar files found'?

  • Make sure your .jar files are in the infobasic-kb folder or in a jars/ subfolder inside it.
  • Check that your terminal's working directory is infobasic-kb by running cd to confirm.
  • Confirm the files have a .jar extension (not .zip or .war).

Still stuck? Help me figure out why extract_jar_signatures.py can't find my .jar files.

Verify the extracted signatures

Each output file contains the same information you would see in Temenos Design Studio: the class name, its inheritance hierarchy, all fields, and every method with full parameter types.

  • Confirm the signature files were created by running:
dir docs\jar-signatures

You should see .txt files listed, one per extracted class.

  • Open one of the generated files to spot-check its contents by running:
type docs\jar-signatures\[[SIGFILE="name of one .txt file from the listing above"]]

What should I see inside a signature file?

A typical javap output starts with the class declaration (including extends and implements clauses), followed by field declarations and method signatures. It looks something like this:

Compiled from "MyClass.java"
public class com.temenos.example.MyClass extends com.temenos.BaseClass {
  private java.lang.String fieldName;
  public com.temenos.example.MyClass();
  public void processRecord(java.lang.String, int);
  protected java.lang.Object getData();
}

Confirm you see recognizable Temenos class names and method signatures that match what you would expect from your libraries.

Signature files are empty or missing methods?

  • Verify that javap is available by running javap -version in your terminal. It should print the JDK version.
  • If signature files exist but contain no methods, the class may be an interface or abstract class with no concrete implementations. This is normal.
  • If the docs\jar-signatures\ folder is completely empty, check that your .jar files contain Temenos-specific packages (not just standard Java classes).

Need more help? Help me debug why javap is not extracting signatures from my .jar files.

Your knowledge base now has detailed API signatures from every Temenos class in your .jar files. Next up, you will ingest your Temenos user guides and scrape INFOBASIC reference content from the web to round out the knowledge base.

Ingest Documents and Scrape Web Resources

In the last step, you extracted Java API signatures from your .jar files and saved them as searchable text documents.

Your knowledge base needs more than just Java method signatures to produce accurate code translations. It also needs to understand INFOBASIC syntax, T24 programming patterns, and Temenos coding standards. In this step, you will build an ingestion script that pulls documentation from three sources: your local Temenos guides, scraped web references, and the .jar signature files you already generated.

In this step, get ready to:
  • Write a document ingestion script that loads local Temenos files and .jar signatures.
  • Scrape INFOBASIC reference content from jBASE docs, the T24 programming wiki, and the Acosani PDF.
  • Combine all sources into a single document collection chunked for retrieval.
Write the local document loader

Your Temenos user guides (HTML and PDF files) and the .jar signature text files are already sitting in local folders. LlamaIndex's SimpleDirectoryReader can read all of these formats out of the box. You just need to tell it which file extensions to include so it skips images, CSS, and other non-text files.

  • Create a new file called ingest_docs.py in your infobasic-kb folder.
  • Add the imports, directory constants, and the local document loading function by pasting this code:
import os
import requests
from bs4 import BeautifulSoup
from llama_index.core import SimpleDirectoryReader, Document
from llama_index.core.node_parser import SentenceSplitter

# Directories for document sources
TEMENOS_DIR = "docs/temenos-guides"
JAR_SIG_DIR = "docs/jar-signatures"
SCRAPED_DIR = "docs/scraped"

# Only load text-based file types
ALLOWED_EXTENSIONS = [".htm", ".html", ".pdf", ".txt"]

def load_local_docs():
    """Load Temenos guides and .jar signature files using SimpleDirectoryReader."""
    all_docs = []

    # Load Temenos HTML/PDF user guides
    if os.path.exists(TEMENOS_DIR):
        temenos_reader = SimpleDirectoryReader(
            input_dir=TEMENOS_DIR,
            required_exts=ALLOWED_EXTENSIONS,
            recursive=True
        )
        temenos_docs = temenos_reader.load_data()
        print(f"Loaded {len(temenos_docs)} documents from {TEMENOS_DIR}")
        all_docs.extend(temenos_docs)

    # Load extracted .jar signature text files
    if os.path.exists(JAR_SIG_DIR):
        jar_reader = SimpleDirectoryReader(
            input_dir=JAR_SIG_DIR,
            required_exts=[".txt"],
            recursive=True
        )
        jar_docs = jar_reader.load_data()
        print(f"Loaded {len(jar_docs)} documents from {JAR_SIG_DIR}")
        all_docs.extend(jar_docs)

    return all_docs

What does this code do?

The ALLOWED_EXTENSIONS list restricts which files SimpleDirectoryReader picks up. Without this filter, it would try to read every file in the folder, including images and stylesheets that would pollute your knowledge base.

The required_exts parameter tells the reader to only process files matching those extensions. The recursive=True flag makes it search subdirectories too.

  • Save ingest_docs.py.

You won't see a change yet. The next chunk adds the web scraping functions that populate docs/scraped/.

Add the web scraping functions

Your knowledge base needs INFOBASIC reference material from the web. Three sources contain the patterns and syntax the LLM needs to understand: the jBASE manual pages, the T24 programming wiki on GitHub, and the Acosani INFOBASIC programming PDF.

  • In ingest_docs.py, add the following web scraping functions below the load_local_docs() function:
def scrape_jbase_manual():
    """Scrape jBASE manual pages and save as clean text."""
    url = "https://static.zumasys.com/jbase/r99/knowledgebase/manuals/3.0/30manpages/man/manpage.htm"
    print(f"Scraping jBASE manual from {url}...")

    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")

        # Remove script and style elements before extracting text
        for element in soup(["script", "style"]):
            element.decompose()

        # Extract clean text content
        text = soup.get_text(separator=" ", strip=True)

        output_path = os.path.join(SCRAPED_DIR, "jbase_manual.txt")
        with open(output_path, "w", encoding="utf-8") as f:
            f.write(text)
        print(f"Saved jBASE manual to {output_path}")
    except Exception as e:
        print(f"Error scraping jBASE manual: {e}")

def scrape_t24_wiki():
    """Scrape T24 programming wiki pages from GitHub."""
    index_url = "https://api.github.com/repos/mathisi-io/t24dev/contents"
    print("Scraping T24 programming wiki from GitHub...")

    try:
        response = requests.get(index_url, timeout=30)
        response.raise_for_status()
        files = response.json()

        # Filter for markdown and text content files
        text_files = [f for f in files if f["name"].endswith((".md", ".txt"))]

        for file_info in text_files:
            raw_url = file_info.get("download_url", "")
            if not raw_url:
                continue

            file_response = requests.get(raw_url, timeout=30)
            file_response.raise_for_status()

            # Save each wiki page as a separate text file
            safe_name = file_info["name"].replace("/", "_")
            output_path = os.path.join(SCRAPED_DIR, f"t24_{safe_name}")
            with open(output_path, "w", encoding="utf-8") as f:
                f.write(file_response.text)

        print(f"Saved {len(text_files)} T24 wiki files to {SCRAPED_DIR}")
    except Exception as e:
        print(f"Error scraping T24 wiki: {e}")

def scrape_acosani_pdf():
    """Download the INFOBASIC programming PDF from Acosani."""
    url = "https://www.acosani.it/infobasic/libri/Programming%20in%20Infobasic.pdf"
    print(f"Downloading INFOBASIC PDF from {url}...")

    try:
        response = requests.get(url, timeout=60)
        response.raise_for_status()

        # Save the raw PDF so LlamaIndex can process it natively
        output_path = os.path.join(SCRAPED_DIR, "programming_in_infobasic.pdf")
        with open(output_path, "wb") as f:
            f.write(response.content)
        print(f"Saved INFOBASIC PDF to {output_path}")
    except Exception as e:
        print(f"Error downloading INFOBASIC PDF: {e}")

def scrape_all_sources():
    """Run all web scrapers and save content to docs/scraped/."""
    os.makedirs(SCRAPED_DIR, exist_ok=True)
    scrape_jbase_manual()
    scrape_t24_wiki()
    scrape_acosani_pdf()

What does this code do?

  • scrape_jbase_manual() fetches the jBASE manual page, strips out <script> and <style> elements, then saves the clean text. BeautifulSoup's get_text(separator=" ", strip=True) collapses the HTML into readable plain text.
  • scrape_t24_wiki() uses the GitHub API to list files in the mathisi-io/t24dev repository. It downloads each markdown and text file via its raw URL and saves them individually.
  • scrape_acosani_pdf() downloads the INFOBASIC programming PDF as a binary file. SimpleDirectoryReader can parse PDFs natively, so saving the raw file is enough.
  • scrape_all_sources() creates the docs/scraped/ directory if it does not exist, then runs all three scrapers in sequence.
  • Save ingest_docs.py.

You won't see a change yet. The next chunk adds the main block that runs everything.

Worried about the GitHub API rate limit?

The GitHub API allows 60 unauthenticated requests per hour. This script makes a small number of requests, so you should stay well within that limit. If you do hit a rate limit, wait a few minutes and run the script again.

help me troubleshoot GitHub API rate limiting in my scraping script

Combine all sources and run the ingestion

With the loading and scraping functions ready, you need a function that brings everything together. It should load all local documents, load the scraped content, and split the combined collection into retrieval-sized chunks using a SentenceSplitter.

  • In ingest_docs.py, add the following combine_all_documents() function and __main__ block below the scrape_all_sources() function:
def combine_all_documents():
    """Combine local docs and scraped content into a single collection."""
    # Load local Temenos guides and .jar signatures
    all_docs = load_local_docs()

    # Load scraped web content
    if os.path.exists(SCRAPED_DIR) and os.listdir(SCRAPED_DIR):
        scraped_reader = SimpleDirectoryReader(
            input_dir=SCRAPED_DIR,
            required_exts=ALLOWED_EXTENSIONS + [".pdf"],
            recursive=True
        )
        scraped_docs = scraped_reader.load_data()
        print(f"Loaded {len(scraped_docs)} documents from {SCRAPED_DIR}")
        all_docs.extend(scraped_docs)

    print(f"\nTotal documents loaded: {len(all_docs)}")

    # Split documents into retrieval-sized chunks
    splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
    nodes = splitter.get_nodes_from_documents(all_docs)
    print(f"Split into {len(nodes)} chunks (chunk_size=512, overlap=50)")

    return all_docs, nodes

if __name__ == "__main__":
    # Step 1: Scrape web resources
    print("=" * 50)
    print("SCRAPING WEB RESOURCES")
    print("=" * 50)
    scrape_all_sources()

    # Step 2: Load and combine all documents
    print("\n" + "=" * 50)
    print("LOADING AND COMBINING DOCUMENTS")
    print("=" * 50)
    documents, nodes = combine_all_documents()

    print("\nIngestion complete!")

What does this code do?

  • combine_all_documents() first calls load_local_docs() to gather Temenos guides and .jar signatures. Then it reads the docs/scraped/ folder (which was just populated by the scrapers).
  • SentenceSplitter(chunk_size=512, chunk_overlap=50) breaks the combined documents into 512-token chunks with 50 tokens of overlap between consecutive chunks. The overlap ensures that sentences split across chunk boundaries still appear in at least one complete chunk.
  • The __main__ block runs the scrapers first, then loads and combines everything. This ordering matters because the scraped files need to exist on disk before SimpleDirectoryReader can read them.
  • Save ingest_docs.py.
  • Run the ingestion script by running this command in your terminal:
python ingest_docs.py

You should see output listing the number of documents loaded from each source, the total document count, and the number of chunks created. The scraping step may take a minute depending on your internet connection.

What should I see?

The output starts with scraping progress messages for each web source. After scraping completes, you see document counts from each local directory and the scraped folder. The final line reports the total number of chunks created by the SentenceSplitter.

Your exact numbers will differ based on how many Temenos guide files and .jar signature files you have.

Seeing a connection error or timeout?

If any of the three web sources are temporarily unavailable, the script prints an error message but continues with the remaining sources. Your local documents will still load successfully.

If docs/temenos-guides/ is empty, the script skips that directory. Make sure you copied your Temenos HTML or PDF files into that folder in Step 2.

help me debug connection errors in my ingest_docs.py web scraping script

  • Verify the scraped files by running:
ls docs\scraped

You should see text files from the jBASE manual, T24 wiki pages, and the Acosani INFOBASIC PDF listed in the output.

✔️ Awesome, I've got everything!

Great. Double check you have saved ingest_docs.py.

ⓧ I'd like to double check the full code

Here is the complete ingest_docs.py file:

import os
import requests
from bs4 import BeautifulSoup
from llama_index.core import SimpleDirectoryReader, Document
from llama_index.core.node_parser import SentenceSplitter

# Directories for document sources
TEMENOS_DIR = "docs/temenos-guides"
JAR_SIG_DIR = "docs/jar-signatures"
SCRAPED_DIR = "docs/scraped"

# Only load text-based file types
ALLOWED_EXTENSIONS = [".htm", ".html", ".pdf", ".txt"]

def load_local_docs():
    """Load Temenos guides and .jar signature files using SimpleDirectoryReader."""
    all_docs = []

    # Load Temenos HTML/PDF user guides
    if os.path.exists(TEMENOS_DIR):
        temenos_reader = SimpleDirectoryReader(
            input_dir=TEMENOS_DIR,
            required_exts=ALLOWED_EXTENSIONS,
            recursive=True
        )
        temenos_docs = temenos_reader.load_data()
        print(f"Loaded {len(temenos_docs)} documents from {TEMENOS_DIR}")
        all_docs.extend(temenos_docs)

    # Load extracted .jar signature text files
    if os.path.exists(JAR_SIG_DIR):
        jar_reader = SimpleDirectoryReader(
            input_dir=JAR_SIG_DIR,
            required_exts=[".txt"],
            recursive=True
        )
        jar_docs = jar_reader.load_data()
        print(f"Loaded {len(jar_docs)} documents from {JAR_SIG_DIR}")
        all_docs.extend(jar_docs)

    return all_docs

def scrape_jbase_manual():
    """Scrape jBASE manual pages and save as clean text."""
    url = "https://static.zumasys.com/jbase/r99/knowledgebase/manuals/3.0/30manpages/man/manpage.htm"
    print(f"Scraping jBASE manual from {url}...")

    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")

        # Remove script and style elements before extracting text
        for element in soup(["script", "style"]):
            element.decompose()

        # Extract clean text content
        text = soup.get_text(separator=" ", strip=True)

        output_path = os.path.join(SCRAPED_DIR, "jbase_manual.txt")
        with open(output_path, "w", encoding="utf-8") as f:
            f.write(text)
        print(f"Saved jBASE manual to {output_path}")
    except Exception as e:
        print(f"Error scraping jBASE manual: {e}")

def scrape_t24_wiki():
    """Scrape T24 programming wiki pages from GitHub."""
    index_url = "https://api.github.com/repos/mathisi-io/t24dev/contents"
    print("Scraping T24 programming wiki from GitHub...")

    try:
        response = requests.get(index_url, timeout=30)
        response.raise_for_status()
        files = response.json()

        # Filter for markdown and text content files
        text_files = [f for f in files if f["name"].endswith((".md", ".txt"))]

        for file_info in text_files:
            raw_url = file_info.get("download_url", "")
            if not raw_url:
                continue

            file_response = requests.get(raw_url, timeout=30)
            file_response.raise_for_status()

            # Save each wiki page as a separate text file
            safe_name = file_info["name"].replace("/", "_")
            output_path = os.path.join(SCRAPED_DIR, f"t24_{safe_name}")
            with open(output_path, "w", encoding="utf-8") as f:
                f.write(file_response.text)

        print(f"Saved {len(text_files)} T24 wiki files to {SCRAPED_DIR}")
    except Exception as e:
        print(f"Error scraping T24 wiki: {e}")

def scrape_acosani_pdf():
    """Download the INFOBASIC programming PDF from Acosani."""
    url = "https://www.acosani.it/infobasic/libri/Programming%20in%20Infobasic.pdf"
    print(f"Downloading INFOBASIC PDF from {url}...")

    try:
        response = requests.get(url, timeout=60)
        response.raise_for_status()

        # Save the raw PDF so LlamaIndex can process it natively
        output_path = os.path.join(SCRAPED_DIR, "programming_in_infobasic.pdf")
        with open(output_path, "wb") as f:
            f.write(response.content)
        print(f"Saved INFOBASIC PDF to {output_path}")
    except Exception as e:
        print(f"Error downloading INFOBASIC PDF: {e}")

def scrape_all_sources():
    """Run all web scrapers and save content to docs/scraped/."""
    os.makedirs(SCRAPED_DIR, exist_ok=True)
    scrape_jbase_manual()
    scrape_t24_wiki()
    scrape_acosani_pdf()

def combine_all_documents():
    """Combine local docs and scraped content into a single collection."""
    # Load local Temenos guides and .jar signatures
    all_docs = load_local_docs()

    # Load scraped web content
    if os.path.exists(SCRAPED_DIR) and os.listdir(SCRAPED_DIR):
        scraped_reader = SimpleDirectoryReader(
            input_dir=SCRAPED_DIR,
            required_exts=ALLOWED_EXTENSIONS + [".pdf"],
            recursive=True
        )
        scraped_docs = scraped_reader.load_data()
        print(f"Loaded {len(scraped_docs)} documents from {SCRAPED_DIR}")
        all_docs.extend(scraped_docs)

    print(f"\nTotal documents loaded: {len(all_docs)}")

    # Split documents into retrieval-sized chunks
    splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
    nodes = splitter.get_nodes_from_documents(all_docs)
    print(f"Split into {len(nodes)} chunks (chunk_size=512, overlap=50)")

    return all_docs, nodes

if __name__ == "__main__":
    # Step 1: Scrape web resources
    print("=" * 50)
    print("SCRAPING WEB RESOURCES")
    print("=" * 50)
    scrape_all_sources()

    # Step 2: Load and combine all documents
    print("\n" + "=" * 50)
    print("LOADING AND COMBINING DOCUMENTS")
    print("=" * 50)
    documents, nodes = combine_all_documents()

    print("\nIngestion complete!")

Your ingestion pipeline is working. All three documentation sources are loaded, scraped, and chunked. Next up, you will embed these chunks into a ChromaDB vector store and build a query interface to search your knowledge base.

Build the Vector Store, Query Interface, and Subroutine Mapping

Your ingestion pipeline from the previous step loaded documents from three sources: Temenos guides, extracted .jar signatures, and scraped web content. Right now those documents are just sitting in memory with no way to search them.

In this step, you will embed every document chunk into a ChromaDB vector store, build an interactive query interface, and create a structured subroutine mapping that pairs INFOBASIC T24 routines with their Java library equivalents.

In this step, get ready to:
  • Embed all document chunks into a persistent ChromaDB vector store.
  • Build an interactive terminal query loop to search the knowledge base.
  • Create a subroutine mapping index that pairs 10 common T24 routines with their Java equivalents.
Configure LlamaIndex and embed documents into ChromaDB

The knowledge base script connects LlamaIndex to your local Ollama models, loads every document from all three source directories, and embeds them into a persistent vector store.

  • In VS Code, create a new file called build_kb.py in the infobasic-kb folder.
  • Add the imports and model configuration by pasting this code:
import chromadb
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure the local LLM for code-focused generation
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

# Configure the local embedding model for vectorizing documents
Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

What does this code do?

Settings.llm tells LlamaIndex to route all LLM calls to your local Ollama instance running qwen2.5-coder:7b. The temperature=0.1 keeps answers factual, and request_timeout=300.0 gives the CPU enough time to finish generating.

Settings.embed_model tells LlamaIndex to use nomic-embed-text for converting text chunks into vector embeddings. Both models run entirely on your machine.

  • Below the model configuration, add the document loading and vector store creation code:
# Load documents from all three source directories
print("Loading documents from Temenos guides...")
temenos_docs = SimpleDirectoryReader(
    input_dir="docs/temenos-guides",
    required_exts=[".htm", ".html", ".pdf", ".txt"]
).load_data()

print("Loading .jar signature files...")
jar_docs = SimpleDirectoryReader(
    input_dir="docs/jar-signatures",
    required_exts=[".txt"]
).load_data()

print("Loading scraped web content...")
scraped_docs = SimpleDirectoryReader(
    input_dir="docs/scraped",
    required_exts=[".txt"]
).load_data()

# Combine all document sources into one collection
all_docs = temenos_docs + jar_docs + scraped_docs
print(f"Total documents loaded: {len(all_docs)}")

# Set up ChromaDB for persistent local vector storage
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Split documents into chunks and embed them into the vector store
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
print("Embedding documents into ChromaDB (this may take several minutes on CPU)...")
index = VectorStoreIndex.from_documents(
    all_docs,
    storage_context=storage_context,
    transformations=[splitter]
)
print("Knowledge base built and persisted to ./chroma_db")

What does this code do?

  • SimpleDirectoryReader loads your Temenos guides, .jar signature files, and scraped web content, filtering to only process text-based file types.
  • chromadb.PersistentClient(path="./chroma_db") creates a file-based vector database that persists to disk. No server needed.
  • SentenceSplitter(chunk_size=512, chunk_overlap=50) breaks each document into 512-token chunks with 50 tokens of overlap, so the LLM gets focused context without losing sentence boundaries.
  • VectorStoreIndex.from_documents embeds every chunk using nomic-embed-text and stores the vectors in ChromaDB.
  • Save build_kb.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import chromadb
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure the local LLM for code-focused generation
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

# Configure the local embedding model for vectorizing documents
Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

# Load documents from all three source directories
print("Loading documents from Temenos guides...")
temenos_docs = SimpleDirectoryReader(
    input_dir="docs/temenos-guides",
    required_exts=[".htm", ".html", ".pdf", ".txt"]
).load_data()

print("Loading .jar signature files...")
jar_docs = SimpleDirectoryReader(
    input_dir="docs/jar-signatures",
    required_exts=[".txt"]
).load_data()

print("Loading scraped web content...")
scraped_docs = SimpleDirectoryReader(
    input_dir="docs/scraped",
    required_exts=[".txt"]
).load_data()

# Combine all document sources into one collection
all_docs = temenos_docs + jar_docs + scraped_docs
print(f"Total documents loaded: {len(all_docs)}")

# Set up ChromaDB for persistent local vector storage
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Split documents into chunks and embed them into the vector store
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
print("Embedding documents into ChromaDB (this may take several minutes on CPU)...")
index = VectorStoreIndex.from_documents(
    all_docs,
    storage_context=storage_context,
    transformations=[splitter]
)
print("Knowledge base built and persisted to ./chroma_db")
  • Run the build script in your terminal:
python build_kb.py

This will take several minutes on CPU as it embeds every document chunk. You should see progress messages for each source directory, then a final count of documents loaded and a confirmation that the knowledge base was persisted.

Script hangs or runs out of memory?

  • Make sure Ollama is running in the background. Open a separate terminal and run ollama ls to verify your models are available.
  • If you have a very large number of documents, the embedding step uses significant RAM. Close other applications to free up memory.
  • If the script times out, increase request_timeout in build_kb.py from 300.0 to 600.0.

Still stuck? Help me debug my build_kb.py script. Ollama is running locally with qwen2.5-coder:7b and nomic-embed-text. The script uses LlamaIndex with ChromaDB to embed documents.

Create the interactive query interface

Now that the knowledge base is built and persisted, you need a separate script to query it. This script loads the existing ChromaDB store from disk without re-embedding anything.

  • Create a new file called query_kb.py in your infobasic-kb folder and paste the following code:
import chromadb
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure the same LLM and embedding model as build_kb.py
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

# Load the existing ChromaDB vector store from disk
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

# Build the index from the persisted store (no re-embedding needed)
index = VectorStoreIndex.from_vector_store(vector_store)
print("Loaded knowledge base from ./chroma_db")

# Create a query engine that retrieves the top 5 most relevant chunks
query_engine = index.as_query_engine(similarity_top_k=5)

# Interactive query loop
print("Knowledge base ready. Type your question and press Enter.")
print("Type 'quit' to exit.\n")

while True:
    question = input("Question: ")
    if question.lower() in ("quit", "exit", "q"):
        print("Goodbye!")
        break
    if not question.strip():
        continue
    print("\nSearching knowledge base...\n")
    response = query_engine.query(question)
    print(f"Answer: {response}\n")

What does this code do?

  • VectorStoreIndex.from_vector_store(vector_store) loads the index directly from ChromaDB on disk. It skips the entire embedding step because the vectors already exist.
  • as_query_engine(similarity_top_k=5) creates a query engine that retrieves the 5 most relevant chunks for each question, then passes them to the LLM as context for its answer.
  • The while True loop lets you ask multiple questions in a single session without restarting the script each time.
  • Save query_kb.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import chromadb
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure the same LLM and embedding model as build_kb.py
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

# Load the existing ChromaDB vector store from disk
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

# Build the index from the persisted store (no re-embedding needed)
index = VectorStoreIndex.from_vector_store(vector_store)
print("Loaded knowledge base from ./chroma_db")

# Create a query engine that retrieves the top 5 most relevant chunks
query_engine = index.as_query_engine(similarity_top_k=5)

# Interactive query loop
print("Knowledge base ready. Type your question and press Enter.")
print("Type 'quit' to exit.\n")

while True:
    question = input("Question: ")
    if question.lower() in ("quit", "exit", "q"):
        print("Goodbye!")
        break
    if not question.strip():
        continue
    print("\nSearching knowledge base...\n")
    response = query_engine.query(question)
    print(f"Answer: {response}\n")
Build the subroutine mapping index

The query interface lets you ask free-form questions, but for code translation you need structured mappings. This script queries the knowledge base for each of the 10 most common T24 subroutines and records the best-matching Java class and method from your .jar signatures.

  • Create a new file called build_mapping.py in your infobasic-kb folder.
  • Add the configuration and subroutine mapping builder by pasting this code:
import json
import chromadb
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure LLM and embedding model
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

# Load the persisted ChromaDB index
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(vector_store)
query_engine = index.as_query_engine(similarity_top_k=5)

# The 10 common T24 subroutines to map to Java equivalents
T24_SUBROUTINES = [
    {"name": "F.READ", "query": "What Java class and method is equivalent to the INFOBASIC F.READ subroutine? Show the full method signature from the .jar files."},
    {"name": "F.READU", "query": "What Java class and method is equivalent to the INFOBASIC F.READU subroutine? Show the full method signature from the .jar files."},
    {"name": "F.WRITE", "query": "What Java class and method is equivalent to the INFOBASIC F.WRITE subroutine? Show the full method signature from the .jar files."},
    {"name": "F.DELETE", "query": "What Java class and method is equivalent to the INFOBASIC F.DELETE subroutine? Show the full method signature from the .jar files."},
    {"name": "OPF", "query": "What Java class and method is equivalent to the INFOBASIC OPF (open file) subroutine? Show the full method signature from the .jar files."},
    {"name": "CACHE.READ", "query": "What Java class and method is equivalent to the INFOBASIC CACHE.READ subroutine? Show the full method signature from the .jar files."},
    {"name": "EB.TRANS", "query": "What Java class and method is equivalent to the INFOBASIC EB.TRANS subroutine? Show the full method signature from the .jar files."},
    {"name": "TRANSACTION.ABORT", "query": "What Java class and method is equivalent to the INFOBASIC TRANSACTION.ABORT subroutine? Show the full method signature from the .jar files."},
    {"name": "FATAL.ERROR", "query": "What Java class and method is equivalent to the INFOBASIC FATAL.ERROR subroutine? Show the full method signature from the .jar files."},
    {"name": "EB.READLIST", "query": "What Java class and method is equivalent to the INFOBASIC EB.READLIST subroutine? Show the full method signature from the .jar files."},
]

def build_mappings():
    """Query the knowledge base for each T24 subroutine and build the mapping index."""
    mappings = []

    for subroutine in T24_SUBROUTINES:
        print(f"Looking up Java equivalent for {subroutine['name']}...")
        response = query_engine.query(subroutine["query"])

        # Store the raw response for manual review and refinement
        mapping = {
            "infobasic_subroutine": subroutine["name"],
            "parameters": "",
            "java_class": "",
            "java_method": "",
            "java_signature": "",
            "raw_response": str(response)
        }
        mappings.append(mapping)

    # Save mappings to JSON for manual review
    with open("subroutine_mappings.json", "w") as f:
        json.dump(mappings, f, indent=2)

    print(f"\nSaved {len(mappings)} mappings to subroutine_mappings.json")
    print("Review the file and fill in the parameters, java_class, java_method, and java_signature fields.")

What does this code do?

  • T24_SUBROUTINES lists the 10 most common INFOBASIC routines that need Java equivalents: F.READ, F.READU, F.WRITE, F.DELETE, OPF, CACHE.READ, EB.TRANS, TRANSACTION.ABORT, FATAL.ERROR, and EB.READLIST.
  • build_mappings() queries the knowledge base for each subroutine and saves the LLM's response alongside empty fields for you to fill in after manual review. The raw_response field captures the LLM's full answer so you can extract the correct Java class, method, and signature.
  • Below the build_mappings() function, add the lookup function and entry point:
def lookup_subroutine(name):
    """Look up a subroutine mapping by its INFOBASIC name and return the Java equivalent."""
    with open("subroutine_mappings.json", "r") as f:
        mappings = json.load(f)

    for mapping in mappings:
        if mapping["infobasic_subroutine"].upper() == name.upper():
            print(f"\nINFOBASIC: {mapping['infobasic_subroutine']}")
            print(f"Parameters: {mapping['parameters']}")
            print(f"Java Class: {mapping['java_class']}")
            print(f"Java Method: {mapping['java_method']}")
            print(f"Java Signature: {mapping['java_signature']}")
            return mapping

    print(f"No mapping found for '{name}'.")
    return None

if __name__ == "__main__":
    import sys
    if len(sys.argv) > 1:
        # If a subroutine name is passed, look it up
        lookup_subroutine(sys.argv[1])
    else:
        # Otherwise, build all mappings
        build_mappings()

What does this code do?

  • lookup_subroutine() reads the JSON file and prints the Java equivalent for a given INFOBASIC subroutine name. This gives you a quick command-line lookup tool.
  • The if __name__ == "__main__" block lets you either build all mappings (no arguments) or look up a single subroutine by passing its name as an argument.
  • Save build_mapping.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import json
import chromadb
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Configure LLM and embedding model
Settings.llm = Ollama(
    model="qwen2.5-coder:7b",
    base_url="http://localhost:11434",
    temperature=0.1,
    request_timeout=300.0
)

Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text",
    base_url="http://localhost:11434"
)

# Load the persisted ChromaDB index
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("infobasic_kb")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(vector_store)
query_engine = index.as_query_engine(similarity_top_k=5)

# The 10 common T24 subroutines to map to Java equivalents
T24_SUBROUTINES = [
    {"name": "F.READ", "query": "What Java class and method is equivalent to the INFOBASIC F.READ subroutine? Show the full method signature from the .jar files."},
    {"name": "F.READU", "query": "What Java class and method is equivalent to the INFOBASIC F.READU subroutine? Show the full method signature from the .jar files."},
    {"name": "F.WRITE", "query": "What Java class and method is equivalent to the INFOBASIC F.WRITE subroutine? Show the full method signature from the .jar files."},
    {"name": "F.DELETE", "query": "What Java class and method is equivalent to the INFOBASIC F.DELETE subroutine? Show the full method signature from the .jar files."},
    {"name": "OPF", "query": "What Java class and method is equivalent to the INFOBASIC OPF (open file) subroutine? Show the full method signature from the .jar files."},
    {"name": "CACHE.READ", "query": "What Java class and method is equivalent to the INFOBASIC CACHE.READ subroutine? Show the full method signature from the .jar files."},
    {"name": "EB.TRANS", "query": "What Java class and method is equivalent to the INFOBASIC EB.TRANS subroutine? Show the full method signature from the .jar files."},
    {"name": "TRANSACTION.ABORT", "query": "What Java class and method is equivalent to the INFOBASIC TRANSACTION.ABORT subroutine? Show the full method signature from the .jar files."},
    {"name": "FATAL.ERROR", "query": "What Java class and method is equivalent to the INFOBASIC FATAL.ERROR subroutine? Show the full method signature from the .jar files."},
    {"name": "EB.READLIST", "query": "What Java class and method is equivalent to the INFOBASIC EB.READLIST subroutine? Show the full method signature from the .jar files."},
]

def build_mappings():
    """Query the knowledge base for each T24 subroutine and build the mapping index."""
    mappings = []

    for subroutine in T24_SUBROUTINES:
        print(f"Looking up Java equivalent for {subroutine['name']}...")
        response = query_engine.query(subroutine["query"])

        # Store the raw response for manual review and refinement
        mapping = {
            "infobasic_subroutine": subroutine["name"],
            "parameters": "",
            "java_class": "",
            "java_method": "",
            "java_signature": "",
            "raw_response": str(response)
        }
        mappings.append(mapping)

    # Save mappings to JSON for manual review
    with open("subroutine_mappings.json", "w") as f:
        json.dump(mappings, f, indent=2)

    print(f"\nSaved {len(mappings)} mappings to subroutine_mappings.json")
    print("Review the file and fill in the parameters, java_class, java_method, and java_signature fields.")

def lookup_subroutine(name):
    """Look up a subroutine mapping by its INFOBASIC name and return the Java equivalent."""
    with open("subroutine_mappings.json", "r") as f:
        mappings = json.load(f)

    for mapping in mappings:
        if mapping["infobasic_subroutine"].upper() == name.upper():
            print(f"\nINFOBASIC: {mapping['infobasic_subroutine']}")
            print(f"Parameters: {mapping['parameters']}")
            print(f"Java Class: {mapping['java_class']}")
            print(f"Java Method: {mapping['java_method']}")
            print(f"Java Signature: {mapping['java_signature']}")
            return mapping

    print(f"No mapping found for '{name}'.")
    return None

if __name__ == "__main__":
    import sys
    if len(sys.argv) > 1:
        # If a subroutine name is passed, look it up
        lookup_subroutine(sys.argv[1])
    else:
        # Otherwise, build all mappings
        build_mappings()
Test the knowledge base and verify the subroutine mappings

Time to put it all together. You will test the query interface with targeted questions, generate the subroutine mappings, and verify the results against your actual .jar files.

  • Start the query interface by running:
python query_kb.py

You should see a message confirming the knowledge base loaded from ./chroma_db, followed by a prompt waiting for your question.

  • Test with these three targeted questions. Type each one at the Question: prompt and press Enter:
What is the syntax for F.READ in INFOBASIC?
What Java class handles file reading in the Temenos library?
How do I configure a PGM.FILE entry for a new subroutine?

What should I see?

Each query takes 50-100 seconds on CPU. The first question should return content from your Temenos guides describing F.READ syntax. The second should return matches from your .jar signature files. The third should return configuration guidance from your Temenos documentation or scraped references.

If answers are vague or cite no sources, your documents may not have been embedded properly. Re-run build_kb.py and check that the document count is greater than zero.

  • Type quit to exit the query loop.
  • Generate the subroutine mappings by running:
python build_mapping.py

The script queries the knowledge base for all 10 T24 subroutines. Each lookup takes 50-100 seconds, so the full run takes roughly 10-15 minutes.

  • Once it finishes, open subroutine_mappings.json in VS Code.
  • For each entry, read the raw_response field and fill in the parameters, java_class, java_method, and java_signature fields with the correct values from your .jar files.
  • Cross-check at least 3 entries (such as F.READ, F.WRITE, and EB.TRANS) against the signature files in docs/jar-signatures/ to make sure the Java class and method names match exactly.

Refine incorrect mappings manually

The LLM may not always find the exact Java equivalent. If a mapping looks wrong, open the matching signature file in docs/jar-signatures/ and copy the correct class name and method signature directly into subroutine_mappings.json.

  • Verify the lookup function works by testing a few entries:
python build_mapping.py F.READ
python build_mapping.py F.WRITE
python build_mapping.py EB.TRANS

Each command should print the INFOBASIC subroutine name alongside its matched Java class, method, and full signature.

  • Confirm the chroma_db/ folder is populated by running:
ls chroma_db

You should see files inside the folder, confirming the vector store persisted to disk.

Lookup returns empty fields?

  • The parameters, java_class, java_method, and java_signature fields start empty by design. You need to fill them in manually based on the raw_response content and your .jar signature files.
  • If raw_response is also empty or unhelpful, try running query_kb.py and asking the same question manually to see what the knowledge base returns.

Need more help? Help me verify my subroutine mappings in build_mapping.py. The lookup function returns empty fields for some T24 subroutines.

Secret mission

Pilot Translation Run

Your knowledge base can answer questions about INFOBASIC and your Java libraries. But the real goal is code translation. Can you feed it a real INFOBASIC subroutine and get a valid Java translation grounded in your actual library APIs? Time to find out.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Ollama models stored locally at %HOMEPATH%\.ollama (qwen2.5-coder:7b at ~4.7GB and nomic-embed-text at ~274MB). These are free and cost nothing to keep.
  • ChromaDB vector store at ./chroma_db (local files, no cost).
  • Python virtual environment and packages in ./venv.
  • Project files in the infobasic-kb folder on your Desktop (scripts, docs, mappings, and pilot results).

Keep everything running

No action needed. Choose this if you're still actively building or plan to continue into Project 2.

  • Ollama runs as a background service and uses minimal resources when idle. You can confirm it is running by opening a terminal and running ollama ls.
  • The ChromaDB store persists to disk in ./chroma_db. It takes no memory when your scripts are not running.
  • Your virtual environment in ./venv stays on disk and only uses memory when activated.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your models, ChromaDB data, and project files all remain on disk.

  • Right-click the Ollama icon in your system tray (bottom-right of your taskbar) and select Quit Ollama. This frees up the memory Ollama uses while running in the background.
  • If your virtual environment is active in a terminal, type deactivate and close the terminal window.
  • Your models (~5GB total) stay in %HOMEPATH%\.ollama and your ChromaDB index stays in ./chroma_db. When you come back, just reopen Ollama and activate your virtual environment.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your models, vector store, and project files.

  • Remove your Ollama models by running these commands in your terminal:
ollama rm qwen2.5-coder:7b
ollama rm nomic-embed-text
  • If your virtual environment is still active, deactivate it first:
deactivate
  • Delete the entire project folder. Navigate to your Desktop and remove it:

macOS

cd ~/Desktop
rm -rf infobasic-kb

Windows

cd ~\Desktop
rmdir /s /q infobasic-kb
  • This removes all scripts, your virtual environment, the ChromaDB vector store, your docs folders, subroutine_mappings.json, and pilot_results.md.

Nice Work!

Nice Work!

Nice work! You now have a fully offline, queryable knowledge base on your laptop that grounds a local LLM in your actual Temenos documentation, custom Java APIs, and INFOBASIC references.

This is the foundation that Project 2 will use to build an intelligent code translation agent.

You've learned how to:

  • Built a fully local RAG knowledge base powered by Ollama and ChromaDB, with a document ingestion pipeline that processes HTML, PDF, and text files from Temenos user guides and scraped web sources.
  • Extracted Java API signatures from .jar files using javap and indexed them as searchable vector embeddings alongside INFOBASIC coding standards scraped from jBASE docs, T24 programming resources, and Acosani references.
  • Built an interactive query interface and a structured subroutine mapping index that pairs INFOBASIC T24 routines like F.READ, F.WRITE, and EB.TRANS with their Java library equivalents.
  • Secret Mission: Ran a pilot translation of a real INFOBASIC subroutine through the knowledge base, evaluated the output quality, and documented findings in pilot_results.md to guide the translation agent design in Project 2.

Ready to quiz yourself?