Docs Decoder: API Docs to Diagrams

Parse API docs into structured JSON and Mermaid diagrams using Groq.

Introduction

30 Second Summary

Every API has documentation, but those pages are buried under layers of HTML noise. Extracting the actual endpoint details by hand is slow and does not scale.

In this project, you will build a command-line tool called Docs Decoder that turns messy API documentation into clean, structured data. It uses Groq's LLM to extract endpoints, validates every extraction against the source text to catch hallucinations, and generates Mermaid diagrams you can visualize instantly.

What You'll Build

Run Docs Decoder against any API docs URL and watch it deliver a clean JSON file of every endpoint and a Mermaid flowchart you can paste straight into a live renderer.

By the end of this project, you'll have:

Feed in a documentation URL and get back structured JSON with every endpoint, its HTTP method, path, description, and parameters.
See exactly which extractions the LLM fabricated vs. which are grounded in the actual docs.
Generate a Mermaid diagram that maps out the API's endpoint architecture, viewable in any Mermaid renderer.
Secret Mission: Enrich your diagrams with data-flow analysis that maps how endpoints connect to each other.

Are there any prerequisites?

This project assumes you are comfortable with Python and have Python 3.10+ installed along with a code editor. You will create a free Groq account in Step 1.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Tools and Groq Account

Your Docs Decoder pipeline needs three ingredients before it can do anything: a library to fetch web pages, another to strip HTML down to readable text, and a client to talk to Groq's LLM.

In this step, you'll install those packages, create a Groq account with an API key, and run a quick test script to confirm everything connects.

In this step, get ready to:

Install the Python packages for HTTP requests, HTML parsing, and LLM access.
Create a Groq account and set up your API key.
Test the connection with a minimal script.

Install the Python packages

You need three packages: requests to download web pages, beautifulsoup4 to parse HTML, and groq to call Groq's LLM.

Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
Install all three packages by running this command:

pip install groq==1.6.0 beautifulsoup4==4.15.0 requests==2.34.2

What are these packages?

requests handles HTTP requests so your script can download any web page. beautifulsoup4 (BeautifulSoup) parses HTML into a navigable structure you can strip down to just the text.

groq is Groq's official Python SDK. It gives you a client that sends prompts to their LLM and gets structured responses back.

Verify the packages installed correctly by running this command:

python -c "from groq import Groq; print('Ready')"

You should see Ready printed in your terminal.

Seeing ModuleNotFoundError?

If your terminal says it can't find the groq module, the install may have gone to a different Python version. Try running pip3 install instead of pip install and re-run the check.

help me fix ModuleNotFoundError for groq after pip install

Create a Groq account and get an API key

Groq provides ultra-fast LLM inference. Your pipeline will send documentation text to Groq's model and get structured JSON back. To call the API, you need an account and an API key.

Open https://console.groq.com in your browser and sign up for a free account.
After signing in, click API Keys in the left sidebar.
Click Create API Key.
Give your key a name (e.g. docs-decoder) and click Submit.
Copy your API key immediately and paste it somewhere safe. Groq only shows the full key once.

Why is this free?

Groq's developer plan gives you generous free-tier usage. The model you'll use, openai/gpt-oss-20b, costs $0.075 per million input tokens. This entire project will use a tiny fraction of that, well under a cent.

Now you need to make the API key available to your Python script. You do this by setting it as an environment variable in your terminal.

Switch back to your terminal and set the API key by running this command (replace your-api-key-here with the key you just copied):

macOS

export GROQ_API_KEY=your-api-key-here

Windows

$env:GROQ_API_KEY = "your-api-key-here"

Confirm the variable is set by running:

macOS

echo $GROQ_API_KEY

Windows

echo $env:GROQ_API_KEY

You should see your API key printed back. If you see a blank line, re-run the export command and make sure you pasted the key correctly.

Seeing a blank line?

Make sure there are no spaces around the = sign in the export command. On macOS, the format is export GROQ_API_KEY= followed directly by your key, with no wrapping quotes.

help me set my GROQ_API_KEY environment variable

Write a test script to verify the connection

Packages are installed and your API key is set. Before building the full pipeline, you need to confirm that your script can actually reach Groq and get a response.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir docs-decoder
cd docs-decoder

Confirm the folder was created by running:

ls

The folder is empty for now. That is expected.

Open your code editor by pressing Cmd+Space (macOS) or the Windows key (Windows), typing the name of your editor, and pressing Enter.
Open the docs-decoder folder from your Desktop in your editor.
Create a new file called docs_decoder.py inside the docs-decoder folder.
Copy the following code into docs_decoder.py:

import os
from groq import Groq

# Create a Groq client using the API key from environment variables
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

# Send a test message to verify the connection works
response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Say hello"}]
)
# Print the model's response
print(response.choices[0].message.content)

What does this code do?

Groq(api_key=os.environ.get("GROQ_API_KEY")) creates a client that authenticates using the environment variable you set earlier.
client.chat.completions.create() sends a chat message to the openai/gpt-oss-20b model. This model runs at 1000 tokens per second and costs $0.075 per million input tokens.
response.choices[0].message.content pulls the actual text response out of the API's return structure.

Save docs_decoder.py.

Before you run this, what do you expect to see in the terminal? Think about what you asked the model to do.

Switch back to your terminal (make sure you are still in the docs-decoder folder) and run the script:

python docs_decoder.py

You should see a greeting from the model printed to your terminal. The exact wording will vary, but you will see something like a friendly hello message.

Seeing an authentication error?

If you see an error mentioning an invalid API key or authentication failure, your GROQ_API_KEY environment variable may not be set in this terminal session. Run the export (macOS) or $env: (Windows) command again in the same terminal window where you run the script.

Also make sure you copied the full API key with no extra spaces. If you closed the Groq key dialog, you will need to create a new key.

help me fix my Groq API authentication error

✔️ Awesome, I've got everything!

Great. Double check you have saved docs_decoder.py and that the response printed to your terminal.

ⓧ I'd like to double check the full code

Here is the full docs_decoder.py file. Compare it against yours:

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Say hello"}]
)
print(response.choices[0].message.content)

Your tools are installed, your API key is working, and Groq is responding. Next up, you'll teach your script to fetch a real documentation page and strip it down to clean, readable text.

Fetch and Parse Documentation into Clean Text

Your Python packages are installed and your Groq connection is confirmed.

Now it's time to point your pipeline at real documentation. API docs pages are full of HTML noise: navigation bars, scripts, footers, and ads surrounding the actual content. Before your pipeline can extract anything useful, you need to strip all that away and get clean, readable text.

In this step, get ready to:

Write a function that fetches a documentation page and strips its HTML down to clean text.
Run the pipeline and see clean documentation text printed to the terminal with a character count.

Write the fetch_docs() function

Your docs_decoder.py currently has the Groq test script from Step 1. You'll replace it with a function that downloads a documentation page, removes all the noisy HTML tags, and returns just the meaningful text.

Select all the code in docs_decoder.py and delete it.
Replace it with the following code:

import os
import json
import requests
from bs4 import BeautifulSoup
from groq import Groq


DOCS_URL = "https://console.groq.com/docs/api-reference"


def fetch_docs(url):
    """Fetch and parse a documentation page into clean text."""
    response = requests.get(url, headers={"User-Agent": "DocsDecoder/1.0"})
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
        tag.decompose()

    text = soup.get_text(separator="\n", strip=True)
    lines = [line for line in text.splitlines() if line.strip()]
    return "\n".join(lines)

What does this code do?

The imports bring in everything the pipeline needs: requests for downloading web pages, BeautifulSoup for parsing HTML, and Groq for later LLM calls.

DOCS_URL points to Groq's own API reference page. Your pipeline will use Groq to extract Groq's own docs.

requests.get() downloads the page HTML. The User-Agent header identifies your script so the server doesn't block it as an unknown bot.
raise_for_status() stops the script immediately if the page returns an error (like a 403 or 404).
The for tag in soup(...): loop finds every script, style, nav, footer, header, and aside tag and removes it from the page entirely.
get_text(separator="\n", strip=True) extracts just the visible text from what remains. The list comprehension then filters out any blank lines.

Save docs_decoder.py. The next chunk adds a main() function that calls fetch_docs() and lets you run the pipeline.

Wire it into main() and run

The fetch_docs() function is ready, but nothing calls it yet. You need a main() function to kick off the pipeline and show you the results.

Add the following code at the bottom of docs_decoder.py, below the fetch_docs() function:

def main():
    source_text = fetch_docs(DOCS_URL)
    print(f"Fetched {len(source_text)} characters")
    print(f"Preview: {source_text[:500]}...")


if __name__ == "__main__":
    main()

What does this code do?

main() calls fetch_docs(DOCS_URL) to download and clean the Groq API reference page, then prints the total character count and the first 500 characters as a preview.
if __name__ == "__main__": ensures main() only runs when you execute the script directly. This is a standard Python pattern that prevents the code from running if the file is imported as a module later.

Save docs_decoder.py.

Before you run the script, take a moment to predict what you'll see. Will the output be raw HTML, or something cleaner?

Run your pipeline by executing this command in your terminal:

python docs_decoder.py

You should see clean, readable documentation text printed to the terminal. No HTML tags, no JavaScript, no navigation menus. Just the actual content of the page, followed by a character count like Fetched 12,847 characters.

Why does the character count matter?

The character count tells you how much content your parser extracted. A very small number (under 100) would mean the page blocked your request or the parser removed too much. A very large number means there's plenty of documentation for the LLM to work with in the next step.

Seeing an error instead of clean text?

If you see an HTTPError with a message mentioning a status code and URL, the page may be temporarily blocking requests. Wait a few seconds and try again.
If you see ConnectionError, check that you're connected to the internet.
If the output is empty or very short, the page structure may have changed. Check that DOCS_URL in your code matches https://console.groq.com/docs/api-reference exactly.

Help me debug my fetch_docs() output

✔️ Awesome, I've got everything!

Great. Double check you've saved docs_decoder.py and that your terminal is showing clean text output.

ⓧ I'd like to double check the full code

Scroll back up through your docs_decoder.py and confirm each section is present, in this order:

Imports for os, json, requests, BeautifulSoup, and Groq.
DOCS_URL constant set to "https://console.groq.com/docs/api-reference".
fetch_docs(url) function with requests.get(), BeautifulSoup parsing, tag removal loop, and text extraction.
main() function that calls fetch_docs(DOCS_URL) and prints the character count and preview.
if __name__ == "__main__": entry point at the very bottom.

You've got clean, readable text from a raw documentation URL. Next up, you'll send this text to Groq and extract structured API endpoint data as JSON.

Extract Structured API Data with Groq

Your pipeline can now fetch a raw documentation page and strip it down to clean, readable text. That text is great for humans, but it is not something you can query, filter, or feed into another system.

In this step, you will send that clean text to Groq's LLM and ask it to extract structured endpoint data as JSON. The model will turn prose into a queryable schema with HTTP methods, paths, descriptions, and parameters.

In this step, get ready to:

Write an extraction function that sends documentation text to Groq and receives structured JSON back.
Wire the extraction into your pipeline and see the endpoint data printed to the terminal.

Write the extract_api_data() function

Groq's API supports a JSON mode that forces the model to return valid JSON instead of free-form text. You will combine this with a carefully crafted prompt that defines the exact schema you want back.

In docs_decoder.py, add the first half of the extract_api_data() function directly below the fetch_docs() function. You will complete this function in the next code block:

def extract_api_data(client, text):
    """Extract structured API endpoint data using Groq LLM."""
    # Trim to 6000 chars to keep the extraction focused
    trimmed = text[:6000]

    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a documentation parser. Extract ONLY information "
                    "that is explicitly stated in the provided text. "
                    "Do not invent or guess any details. Return valid JSON."
                ),
            },

What does this code do so far?

trimmed = text[:6000] cuts the documentation down to 6,000 characters. This keeps the extraction focused and avoids accuracy issues that come with feeding very long text into a model.
The system message tells the model to only extract information that is explicitly stated. This is your first line of defense against hallucinations.

Still inside the extract_api_data() function, add the user message and closing code directly below the system message you just pasted:

            {
                "role": "user",
                "content": (
                    "Extract API endpoint information from this documentation.\n\n"
                    "Return a JSON object with an \"endpoints\" array. "
                    "Each endpoint should have:\n"
                    "- method: HTTP method (GET, POST, PUT, DELETE, etc.)\n"
                    "- path: the URL path\n"
                    "- description: what the endpoint does\n"
                    "- parameters: array of objects with name, type, and required (boolean)\n\n"
                    f"Documentation:\n{trimmed}"
                ),
            },
        ],
        # Force JSON output and deterministic responses
        response_format={"type": "json_object"},
        temperature=0,
    )

    return json.loads(response.choices[0].message.content)

What does the rest of this function do?

The user message defines the exact JSON schema you want: an endpoints array where each entry has a method, path, description, and parameters list.
response_format={"type": "json_object"} activates Groq's JSON mode. The model is guaranteed to return valid JSON instead of free-form text.
temperature=0 makes the output deterministic. For data extraction, you want consistency over creativity.
json.loads() parses the model's string response into a Python dictionary you can work with programmatically.

Why trim to 6,000 characters?

The openai/gpt-oss-20b model has a 131,072-token context window, so the full page could technically fit. Trimming keeps the extraction focused on the most relevant content and avoids accuracy degradation over very long inputs.

You can increase this limit later if you want to capture more endpoints from longer documentation pages.

Update main() to run extraction and display results

Your extraction function is ready, but it is not wired into the pipeline yet. You need to update main() to create a Groq client, call the new function, and print the structured results.

In docs_decoder.py, find the existing main() function:

def main():
    source_text = fetch_docs(DOCS_URL)
    print(f"Fetched {len(source_text)} characters")
    print(f"Preview: {source_text[:500]}...")

Replace the entire main() function with this updated version that creates a Groq client and runs the extraction:

def main():
    # Initialize the Groq client using the API key from environment
    client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

    print("1. Fetching documentation...")
    source_text = fetch_docs(DOCS_URL)
    print(f"   Fetched {len(source_text)} characters")
    print(f"   Preview: {source_text[:200]}...\n")

    print("2. Extracting API structure...")
    raw_data = extract_api_data(client, source_text)
    endpoint_count = len(raw_data.get("endpoints", []))
    print(f"   Found {endpoint_count} endpoints")
    print(f"   Raw extraction:\n{json.dumps(raw_data, indent=2)}\n")

What changed in main()?

Groq(api_key=os.environ.get("GROQ_API_KEY")) creates an authenticated Groq client using the environment variable you set in Step 1.
The function now runs a two-stage pipeline: fetch the docs, then extract structured data from the text.
json.dumps(raw_data, indent=2) pretty-prints the extracted JSON so you can inspect every endpoint the model found.

Save docs_decoder.py.

Before you run this, what do you think the output will look like? Will the model return a flat list of endpoints, or will it organize them in some other way?

Run the pipeline by executing this command in your terminal:

python docs_decoder.py

You should see the pipeline fetch the documentation text, then print a JSON object listing extracted endpoints with their HTTP methods, paths, descriptions, and parameters.

Seeing an error?

If you see an authentication error, make sure your GROQ_API_KEY environment variable is still set. Run echo $GROQ_API_KEY to check. If it is blank, re-export it.
If you see a JSON parsing error, make sure the response_format={"type": "json_object"} parameter is present in your client.chat.completions.create() call. Without it, the model may return text that is not valid JSON.
If you see an indentation or syntax error, check that the user message block is properly nested inside the messages list at the same indentation level as the system message.
Why is my extraction failing or returning empty results?

✔️ Awesome, I've got everything!

Great. Double check you have saved docs_decoder.py.

ⓧ I'd like to double check the full code

Your docs_decoder.py file has grown past 30 lines, so scroll through and confirm each section is present in this order:

The imports at the top: os, json, requests, BeautifulSoup, and Groq.
The DOCS_URL constant.
The fetch_docs(url) function from Step 2.
The new extract_api_data(client, text) function with system and user messages, response_format, and temperature=0.
The updated main() that creates a Groq client, fetches docs, extracts data, and prints the JSON.
The if __name__ == "__main__": block at the bottom.

Your pipeline now turns raw documentation into structured, queryable JSON. Take a look at the output. Does every endpoint in that JSON actually exist in the original docs? In the next step, you will find out by adding a grounding layer that catches hallucinations.

Catch Hallucinations with Source Grounding

Your pipeline now extracts structured endpoint data from raw documentation. The JSON output from the last step has methods, paths, descriptions, and parameters neatly organized.

But how much of that data is actually real? LLMs routinely fabricate plausible-looking endpoint paths, parameter names, and descriptions that never appeared in the source text. Without a validation layer, you would ship made-up data downstream. This step adds the safety net that separates a demo from a production pipeline.

In this step, get ready to:

Write a grounding function that checks each extracted endpoint against the original source text.
Update the pipeline to display validated vs. flagged results and save clean data to api_structure.json.

Write the grounding function

The core idea behind grounding is simple: if the LLM claims an endpoint exists, check whether that path actually appears in the text you fed it. If the path is nowhere in the source, the LLM likely invented it.

In docs_decoder.py, find the extract_api_data() function.
Add the ground_extractions() function below it by copying this code:

def ground_extractions(data, source_text):
    """Validate extracted data against the source text to catch hallucinations."""
    source_lower = source_text.lower()
    validated_endpoints = []
    flagged = []

    for endpoint in data.get("endpoints", []):
        path = endpoint.get("path", "")
        # Extract the last segment of the path for flexible matching
        path_fragment = path.split("/")[-1] if "/" in path else path

        if path_fragment.lower() in source_lower or path.lower() in source_lower:
            validated_endpoints.append(endpoint)
        else:
            flagged.append({
                "path": path,
                "reason": "Path not found in source text - possibly hallucinated",
            })

    return {"endpoints": validated_endpoints}, flagged

What does this code do?

This function takes the raw extraction and the original source text, then runs each endpoint through a simple but effective check.

source_lower = source_text.lower() converts the source to lowercase so matching is case-insensitive.
For each endpoint, it extracts the last segment of the path (e.g., completions from /openai/v1/chat/completions) and checks whether it appears anywhere in the source.
Endpoints that pass the check go into validated_endpoints. Endpoints that fail get added to flagged with a reason explaining why.

Save docs_decoder.py.

Wire grounding into the pipeline and run it

The function exists, but nothing calls it yet. You need to update main() to pass the raw extraction through grounding, print the results, and save the validated data to a file.

In docs_decoder.py, scroll to the main() function and find this line near the end:

    print(f"   Raw extraction:\n{json.dumps(raw_data, indent=2)}\n")

Add the grounding, display, and save logic below that line by copying this code:

    # Step 3: Validate extractions against the original source
    print("3. Validating against source text...")
    validated_data, flagged = ground_extractions(raw_data, source_text)
    print(f"   Validated: {len(validated_data['endpoints'])} endpoints")
    print(f"   Flagged:   {len(flagged)} possibly hallucinated")
    for item in flagged:
        print(f"     - {item['path']}: {item['reason']}")

    # Save only the validated endpoints to a JSON file
    with open("api_structure.json", "w") as f:
        json.dump(validated_data, f, indent=2)
    print("   Saved validated data to api_structure.json\n")

What does this code do?

ground_extractions(raw_data, source_text) returns two values: a dictionary of validated endpoints and a list of flagged ones.
The for loop prints each flagged endpoint with its path and reason, so you can see exactly what the LLM fabricated.
json.dump() writes only the validated endpoints to api_structure.json. Flagged data never makes it into the output file.

Save docs_decoder.py.

Before you run the script, think about this: do you expect all the endpoints from Step 3 to be validated, or will some get flagged as hallucinated?

Run the full pipeline by running this command:

python docs_decoder.py

You should see the familiar fetch and extraction output from before, followed by a new section. It splits the endpoints into two groups: validated (confirmed in the source text) and flagged (not found, possibly hallucinated). Each flagged endpoint is listed with its path and reason.

Confirm that api_structure.json was created by running:

ls

You should see api_structure.json alongside docs_decoder.py in the output.

Not seeing validated vs. flagged output?

Make sure the ground_extractions() function is placed above the main() function. Python needs to see the function definition before it gets called.
Check that you added the new code after the line that prints the raw extraction, not inside the extract_api_data() function.
If you see a NameError, double-check that the function name in your main() call matches the function definition exactly: ground_extractions.

Why is my grounding step not producing the expected output?

Why does grounding matter?

The extraction from Step 3 probably looked convincing. Every endpoint had a method, path, and description. But an LLM can generate a perfectly formatted endpoint that was never mentioned in the original documentation.

Grounding is the standard production pattern for catching this. If an extracted value cannot be traced back to the source text, it gets flagged instead of silently passed downstream. This is the difference between a demo and a real data pipeline.

✔️ Awesome, I've got everything!

Great. Double check you have saved docs_decoder.py and that api_structure.json exists in your project folder.

ⓧ I'd like to double check the full code

Your docs_decoder.py is now over 80 lines. Scroll through your file and confirm each of these sections is present, in order:

Imports (os, json, requests, BeautifulSoup, Groq) and the DOCS_URL constant at the top.
fetch_docs(url) function that downloads and cleans HTML.
extract_api_data(client, text) function that sends text to Groq and returns structured JSON.
ground_extractions(data, source_text) function that validates endpoints against the source (added in this step).
main() function that runs fetch, extract, grounding, and saves to api_structure.json.
if __name__ == "__main__": block at the very bottom.

Your pipeline now fetches, extracts, and validates. Only the endpoints that can be traced back to the source text make it into api_structure.json. Next up, you will turn that clean data into a visual diagram that maps out the entire API architecture at a glance.

Generate Mermaid Diagrams from Extracted Data

In the last step, you added a grounding layer that catches hallucinated endpoints before they contaminate your output. Your pipeline now produces clean, validated JSON saved to api_structure.json.

JSON is great for machines, but it is hard for a human to glance at a JSON file and understand how an API is structured. A visual diagram makes that architecture instantly scannable and shareable. In this step, you will generate a Mermaid flowchart from your validated data, completing the pipeline from raw docs URL to rendered diagram.

In this step, get ready to:

Write a generate_mermaid() function that sends validated data to Groq and gets back Mermaid diagram syntax.
Run the full pipeline end-to-end and visualize the generated diagram on mermaid.live.

Write the generate_mermaid() function

Mermaid is a text-based diagramming language. You write simple syntax like flowchart LR and renderers turn it into a visual diagram. Since it is plain text, an LLM can generate it directly as a response.

Why Mermaid instead of a Python diagramming library?

Python libraries like diagrams require Graphviz installed separately and are designed for cloud architecture diagrams. Mermaid needs zero install overhead. It renders natively in GitHub, VS Code, Notion, and the free online editor at mermaid.live.

Because Mermaid syntax is just text, the LLM can produce it in a single response. No rendering dependencies on your machine.

In docs_decoder.py, find the ground_extractions() function. Add the following new function below it:

def generate_mermaid(client, data):
    """Generate a Mermaid flowchart diagram from structured API data."""
    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[
            {
                "role": "system",
                # Instruct model to return raw Mermaid syntax only
                "content": (
                    "You are a diagram generator. Output ONLY valid Mermaid "
                    "diagram syntax. No markdown fences, no explanations."
                ),
            },
            {
                "role": "user",
                # Group endpoints by resource, label edges with HTTP methods
                "content": (
                    "Create a Mermaid flowchart diagram for this API structure.\n"
                    "Group endpoints by resource. Show HTTP methods as edge labels.\n"
                    "Use this format: flowchart LR\n\n"
                    f"API Data:\n{json.dumps(data, indent=2)}"
                ),
            },
        ],
        temperature=0,
    )

    return response.choices[0].message.content.strip()

What does this code do?

The system prompt tells the model to act as a diagram generator and return only raw Mermaid syntax. No markdown fences, no explanations. This keeps the output clean so you can save it directly to a file.
The user prompt passes in the validated endpoint JSON and asks for a flowchart LR (left-to-right) layout, grouped by resource with HTTP methods as edge labels.
The .strip() call trims any leading or trailing whitespace from the response before returning it.

Save docs_decoder.py. The next chunk wires this function into main() so you can run the full pipeline.

Wire the diagram into main() and run the full pipeline

The function is ready. Now you need to call it from main() after the grounding stage, save the output to a file, and print it to the terminal.

In docs_decoder.py, scroll down to main() and find this line:

    print("   Saved validated data to api_structure.json\n")

Add the following code below that line:

    # Stage 4: Generate a Mermaid flowchart from validated data
    print("4. Generating Mermaid diagram...")
    mermaid_code = generate_mermaid(client, validated_data)

    # Save the diagram to a .mmd file
    with open("api_diagram.mmd", "w") as f:
        f.write(mermaid_code)
    print("   Saved diagram to api_diagram.mmd\n")

    # Print the diagram code for easy copy-paste
    print("--- Mermaid Diagram ---")
    print(mermaid_code)
    print("\nPaste the diagram code into https://mermaid.live to visualize!")

What does this code do?

generate_mermaid(client, validated_data) sends the grounded endpoint data to Groq and gets back raw Mermaid syntax.
The Mermaid code is saved to api_diagram.mmd (the standard file extension for Mermaid diagrams) so you can open it later or commit it to version control.
The diagram is also printed to the terminal so you can copy-paste it directly into a renderer.

Save docs_decoder.py.

Before you run this, what do you expect the terminal output to look like now that all four pipeline stages are wired up?

Run the full pipeline by running this command:

python docs_decoder.py

You should see all four stages execute in sequence: fetching documentation, extracting API structure, validating against the source text, and generating the Mermaid diagram. At the bottom, the raw Mermaid syntax is printed between --- Mermaid Diagram --- markers.

Script failing on the diagram step?

If you see a NameError: name 'generate_mermaid' is not defined, make sure the generate_mermaid() function is defined above main() in your file, not inside it.
If the script hangs or times out, your Groq API rate limit may have been reached. Wait a minute and try again.
If the Mermaid output looks garbled or includes markdown fences, check that your system prompt says "No markdown fences, no explanations" exactly as shown.

Ask for help: why is my Mermaid diagram generation failing in docs_decoder.py?

Visualize your diagram on mermaid.live

The Mermaid code in your terminal is just text. To see it as a visual flowchart, you will paste it into an online renderer.

Copy the Mermaid code printed between the --- Mermaid Diagram --- markers in your terminal.
Open https://mermaid.live in your browser.
Clear the default code in the left-side editor panel and paste your copied Mermaid code.

On the right side of the screen, you should see a rendered flowchart showing the API's endpoints grouped by resource. Each edge is labeled with an HTTP method like GET or POST.

What if my diagram looks different from someone else's?

That is expected. The diagram depends on which endpoints your grounding step validated. Different runs against the same docs page may produce slightly different validated sets, and the LLM may lay out the same endpoints in different arrangements. The important thing is that the diagram reflects real, grounded data from the source page.

Confirm that api_diagram.mmd was created alongside api_structure.json by running:

ls

You should see docs_decoder.py, api_structure.json, and api_diagram.mmd all listed in the output.

✔️ Awesome, I've got everything!

Great. Double check you have saved docs_decoder.py and that all three output files are present in your project folder.

ⓧ I'd like to double check the full code

Your docs_decoder.py file is now too long to show in one block. Scroll up and confirm each of these sections is present in order:

Imports (os, json, requests, BeautifulSoup, Groq) and the DOCS_URL constant at the top.
fetch_docs(url) function that downloads and cleans HTML.
extract_api_data(client, text) function that sends text to Groq with a JSON schema prompt.
ground_extractions(data, source_text) function that validates endpoints against the source.
generate_mermaid(client, data) function that generates a Mermaid flowchart (added in this step).
main() function with all four stages: fetch, extract, ground, and generate diagram.
The if __name__ == "__main__": block calling main() at the bottom.

Your pipeline is complete. From a single docs URL, you now get clean text, structured JSON, grounded validation, and a visual diagram. Next up, the Secret Mission challenges you to enrich those diagrams with data-flow analysis that maps how endpoints connect to each other.

Secret mission

Enrich Diagrams with Data-Flow Analysis

Your basic diagram shows every endpoint as an isolated node. But real APIs have data flow: creating a resource feeds into retrieving it, and listing resources connects to individual lookups. In this secret mission, you'll add relationship analysis that reveals how the API actually works as a connected system.

Clean Up Your Resources

This project runs entirely on your local machine. The only external service is Groq's API, which costs fractions of a cent per run. There are no ongoing charges to worry about.

Decide whether to keep your project files and packages installed, or clean everything up.

Resources you used:

Project files: docs_decoder.py, api_structure.json, api_diagram.mmd, and api_diagram_enriched.mmd.
Python packages: groq, beautifulsoup4, and requests.
Environment variable: GROQ_API_KEY (session-only unless you added it to your shell config).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with Docs Decoder or want to point it at other API documentation pages.

Your project files stay on disk and are ready to run anytime.
The installed Python packages remain available for other projects.
Your Groq account stays active on the free tier with no ongoing charges.

Pause - I'll come back to this later

Clear your API key from the current terminal session to prevent accidental usage. Your files and packages stay in place for when you come back.

Remove the API key from your current session by running:

unset GROQ_API_KEY

When you return, set the variable again with export GROQ_API_KEY=your-api-key-here and you are ready to go.

Delete - I don't want to use this again

Remove all project files and uninstall the Python packages to start fresh.

Clear the API key from your current terminal session by running:

unset GROQ_API_KEY

If you added GROQ_API_KEY to your shell config file (such as .bashrc or .zshrc), open that file and remove the export line.
Delete the project files by running this on macOS/Linux:

rm docs_decoder.py api_structure.json api_diagram.mmd api_diagram_enriched.mmd

On Windows, run this instead:

del docs_decoder.py, api_structure.json, api_diagram.mmd, api_diagram_enriched.mmd

Uninstall the Python packages by running:

pip uninstall groq beautifulsoup4 requests -y

To delete your Groq API key from their servers, log in to the Groq console at console.groq.com, navigate to API Keys, and delete the key you created for this project.

Nice Work!

You just built Docs Decoder, a complete LLM-powered pipeline that turns messy API documentation into structured, validated JSON and visual architecture diagrams.

You learned how to:

Built a content extraction pipeline that fetches raw API documentation, strips HTML noise with BeautifulSoup, and uses Groq's LLM with JSON mode to extract structured endpoint data.
Added a grounding validation layer that catches LLM hallucinations by verifying extractions against the source text.
Generated Mermaid diagrams that visualize the API architecture from the extracted data.
Secret Mission: Enriched diagrams with data-flow analysis showing how endpoints connect to each other.

Ready to quiz yourself?