Invoice Extraction Reliability Harness

Build an Azure invoice pipeline with evaluation and human-review routing.

Introduction

30 Second Summary

An invoice can look perfectly readable while hiding a total that should never be approved. Treating a confident prediction as a business decision can turn one quiet mismatch into a costly exception.

In this project, you will build a Python reliability harness that uses Azure Document Intelligence to extract invoice fields from synthetic documents. It compares every prediction with ground truth before deterministic rules choose automatic acceptance or human review.

What You'll Build

A single pipeline run produces scored field results for two synthetic invoices while routing the inconsistent invoice to human review with a clear reason.

By the end of this project, you'll have:

  • A synthetic PDF test set you can regenerate locally to test extraction without exposing real customer or financial data.
  • Field-level evidence that compares every extracted value with its prewritten label. Each result includes its confidence score plus match status.
  • A review-ready evidence package that produces explicit decisions for a future Aurachain approval workflow. Its assessment separates demonstrated capabilities from unverified claims.
  • Secret Mission: Add a third invoice with no due date. Prove the policy routes it to human review with a missing_field issue for DueDate.

Are there any prerequisites?

You need Windows with Python 3.9 or later. The live path also needs an Azure subscription that offers the F0 pricing tier.

Before We Start

Before any hands-on work begins, commit to treating every Azure Document Intelligence result as a prediction that needs evidence and control. This keeps incorrect, incomplete, low-confidence, or internally inconsistent invoice data under accountable human review.

Create a Secure, Cost-Gated Workspace

A reliability assessment loses credibility when it starts with unverified costs or incompatible dependencies. Credentials saved in project files create another avoidable risk.

You will gate the live Azure Document Intelligence resource at F0. You will build a pinned Python workspace with session-only credentials.

In this step, get ready to:
  • Create an Azure Document Intelligence resource only when the portal offers the F0 pricing tier.
  • Create the project workspace with pinned dependencies and protected generated files.
  • Activate a virtual environment and verify the required imports with session-only credentials.
Create the F0 Azure resource

The pricing gate keeps this project within its verified cost boundary. The live path continues only when the resource form offers F0.

Stop if F0 is unavailable

Creating a paid resource would change the project cost. Leave the form without provisioning anything if F0 does not appear under Pricing tier.

  • Go to the Azure portal in your browser.
  • Sign in with your Azure account.
  • Select Create a resource.
  • Search for Document Intelligence.
  • Select Create on the Document Intelligence result.
  • Select the subscription you want to use.
  • Create a dedicated resource group named invoice-reliability-rg.
  • Select an available region.

Why use a dedicated resource group?

A dedicated resource group keeps this project separate from unrelated Azure resources. That boundary makes the final cleanup clear and verifiable.

  • Enter a unique resource name: your unique invoice reliability resource name.
  • Open the Pricing tier options.
  • Choose the tab below that matches what the portal offers.

✔️ F0 is available

The free pricing gate has passed. You can create the resource without selecting a paid tier.

  • Select F0 as the pricing tier.
  • Select Review + create.
  • Confirm that the review page still shows F0.

Provisioning can take a few minutes while Azure creates the resource. A short wait on the deployment screen is expected.

  • Select Create.
  • Select Go to resource after deployment succeeds.
  • Confirm that the resource overview shows the F0 pricing tier.
  • Open Keys and Endpoint.

The cost gate worked. Your F0 resource is ready for controlled invoice analysis.

ⓧ F0 is unavailable

The verified free path is unavailable for this subscription or region. Stop before creating the resource.

  • Leave the resource creation form without selecting a paid tier.
  • Keep the local workspace for later use when an eligible subscription offers F0.

Do not substitute S0

The project cost is verified only for F0. A paid tier falls outside this project path.

Build the local workspace

A virtual environment isolates this project's packages from other Python projects. Pinned package versions make the workspace reproducible.

  • Press the Windows key to open search.
  • Type File Explorer.
  • Press Enter.
  • Select Desktop in the left sidebar.
  • Select New.
  • Select Folder.
  • Name the folder invoice-extraction-reliability-harness.
  • Press Enter to create the folder.

Your project location

The invoice-extraction-reliability-harness folder is the project root. Every file in the harness belongs inside this folder.

  • Press the Windows key to open search.
  • Type Visual Studio Code.
  • Press Enter to open Visual Studio Code.
  • Select File in the top menu.
  • Select Open Folder.
  • Choose the invoice-extraction-reliability-harness folder on your Desktop.
  • Select Select Folder.

VS Code now shows the empty project root in its Explorer sidebar. The next check confirms whether your installed Python version can run both pinned packages.

  • Select Terminal in the top menu.
  • Select New Terminal.
  • Check your Python version in Windows PowerShell by running:
python --version

What does this check prove?

ReportLab 5.0.1 requires Python 3.9 or later. The displayed version determines whether you can continue with the pinned dependency set.

✔️ I see the required version or higher

Your Python version is 3.9 or later. Keep this PowerShell terminal open for the workspace commands.

ⓧ I see an older version

The active Python installation is below 3.9. Install a supported runtime before creating the virtual environment.

  • Open the official Python Windows installation guide.
  • Install the Python Install Manager using its documented Windows path.
  • Install a current Python runtime.
  • Close the existing VS Code terminal.
  • Create a new PowerShell terminal in VS Code.
  • Check the active version again by running:
python --version

What should you confirm?

The command must now report Python 3.9 or later. A newer runtime can coexist with an older installation when Windows manages the active runtime.

Still seeing the older version?

Windows may still resolve the previous runtime first. Use the official Python Windows troubleshooting guidance to confirm which runtime the python command selects.

Ask for help with the active runtime path: help me make the python command use my supported Windows installation

ⓧ Command not found

Windows cannot find a Python runtime from this PowerShell session. Install Python before creating the workspace environment.

  • Open the official Python Windows installation guide.
  • Install the Python Install Manager using its documented Windows path.
  • Install a current Python runtime.
  • Close the existing VS Code terminal.
  • Create a new PowerShell terminal in VS Code.
  • Confirm the installation by running:
python --version

What should you see?

PowerShell should report Python 3.9 or later. That confirms the runtime is available to the project.

Python still unavailable?

Close every PowerShell terminal after installation so a new session can discover the command. Check the official Windows guidance if the command still cannot find Python.

Get help with the Windows installation: help me troubleshoot why Python is unavailable in VS Code PowerShell

The workspace files define the complete base harness before any model inference occurs. Creating the labels now protects the evaluation from being rewritten to match later extraction results.

  • Create every file shown in the full-code tab inside invoice-extraction-reliability-harness.
  • Copy each file's contents exactly from the reference below.

✔️ Awesome, I've got everything!

Great. Save every file before creating the virtual environment.

ⓧ I'd like to double check the full code

Create requirements.txt with these pinned dependencies.

azure-ai-documentintelligence==1.0.2
reportlab==5.0.1

What does this file control?

This file pins the Azure SDK to 1.0.2. It pins ReportLab to 5.0.1.

Create .gitignore with these exclusions.

.venv/
__pycache__/
*.pyc
.env
.env.*
artifacts/
data/invoices/

What stays out of Git?

The ignore rules exclude the virtual environment and generated artifacts from Git. They also exclude local environment files if one is created accidentally.

Create generate_invoices.py with the synthetic invoice generator.

from pathlib import Path

from reportlab.lib.pagesizes import A4
from reportlab.pdfgen import canvas

OUTPUT_DIR = Path("data/invoices")

INVOICES = [
    {
        "filename": "invoice_001.pdf",
        "vendor": "Ubuntu Logistics Pty Ltd",
        "customer": "Demo Retail Pty Ltd",
        "invoice_id": "INV-1001",
        "invoice_date": "2026-10-01",
        "due_date": "2026-10-31",
        "description": "Freight coordination",
        "subtotal": 1000.00,
        "tax": 150.00,
        "total": 1150.00,
    },
    {
        "filename": "invoice_002.pdf",
        "vendor": "Jozi Office Supplies",
        "customer": "Demo Retail Pty Ltd",
        "invoice_id": "INV-1002",
        "invoice_date": "2026-10-05",
        "due_date": "2026-11-04",
        "description": "Office equipment",
        "subtotal": 500.00,
        "tax": 75.00,
        "total": 580.00,
    },
]


def money(value):
    return f"{value:,.2f}"


def write_invoice(invoice):
    path = OUTPUT_DIR / invoice["filename"]
    pdf = canvas.Canvas(str(path), pagesize=A4)
    width, height = A4

    pdf.setFont("Helvetica-Bold", 20)
    pdf.drawString(50, height - 60, "TAX INVOICE")

    pdf.setFont("Helvetica", 11)
    pdf.drawString(50, height - 100, f"Vendor: {invoice['vendor']}")
    pdf.drawString(50, height - 120, f"Bill To: {invoice['customer']}")
    pdf.drawString(50, height - 150, f"Invoice Number: {invoice['invoice_id']}")
    pdf.drawString(50, height - 170, f"Invoice Date: {invoice['invoice_date']}")
    pdf.drawString(50, height - 190, f"Due Date: {invoice['due_date']}")

    pdf.line(50, height - 220, width - 50, height - 220)
    pdf.drawString(50, height - 245, "Description")
    pdf.drawRightString(390, height - 245, "Subtotal")
    pdf.drawRightString(470, height - 245, "VAT")
    pdf.drawRightString(545, height - 245, "Total")
    pdf.line(50, height - 255, width - 50, height - 255)

    pdf.drawString(50, height - 280, invoice["description"])
    pdf.drawRightString(390, height - 280, money(invoice["subtotal"]))
    pdf.drawRightString(470, height - 280, money(invoice["tax"]))
    pdf.drawRightString(545, height - 280, money(invoice["total"]))

    pdf.line(320, height - 320, width - 50, height - 320)
    pdf.drawString(360, height - 345, "Subtotal")
    pdf.drawRightString(545, height - 345, f"ZAR {money(invoice['subtotal'])}")
    pdf.drawString(360, height - 365, "VAT 15%")
    pdf.drawRightString(545, height - 365, f"ZAR {money(invoice['tax'])}")
    pdf.setFont("Helvetica-Bold", 12)
    pdf.drawString(360, height - 390, "Invoice Total")
    pdf.drawRightString(545, height - 390, f"ZAR {money(invoice['total'])}")

    pdf.setFont("Helvetica", 9)
    pdf.drawString(50, 50, "Synthetic test document. Not payable.")
    pdf.showPage()
    pdf.save()
    return path


def main():
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    for invoice in INVOICES:
        print(f"Created {write_invoice(invoice)}")


if __name__ == "__main__":
    main()

What does this generator provide?

The generator uses ReportLab to create two reproducible PDFs. Every name and amount is synthetic.

Create data/ground_truth.json with the labels written before inference.

{
  "invoice_001.pdf": {
    "VendorName": "Ubuntu Logistics Pty Ltd",
    "CustomerName": "Demo Retail Pty Ltd",
    "InvoiceId": "INV-1001",
    "InvoiceDate": "2026-10-01",
    "DueDate": "2026-10-31",
    "SubTotal": 1000.0,
    "TotalTax": 150.0,
    "InvoiceTotal": 1150.0
  },
  "invoice_002.pdf": {
    "VendorName": "Jozi Office Supplies",
    "CustomerName": "Demo Retail Pty Ltd",
    "InvoiceId": "INV-1002",
    "InvoiceDate": "2026-10-05",
    "DueDate": "2026-11-04",
    "SubTotal": 500.0,
    "TotalTax": 75.0,
    "InvoiceTotal": 580.0
  }
}

Why create labels now?

This JSON file records the expected fields before Azure produces predictions. That order prevents the labels from being adjusted to flatter the model.

Create extractor.py with the Azure extraction boundary.

import os
from pathlib import Path

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential
from azure.core.exceptions import AzureError, HttpResponseError

FIELD_NAMES = (
    "VendorName",
    "CustomerName",
    "InvoiceId",
    "InvoiceDate",
    "DueDate",
    "SubTotal",
    "TotalTax",
    "InvoiceTotal",
)


def require_environment_variable(name):
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(f"Missing required environment variable: {name}")
    return value


def normalized_value(field):
    if field is None:
        return None
    if field.value_string is not None:
        return field.value_string
    if field.value_date is not None:
        return field.value_date.isoformat()
    if field.value_currency is not None:
        return round(field.value_currency.amount, 2)
    if field.value_number is not None:
        return round(field.value_number, 2)
    return field.content


def extract_invoice(pdf_path):
    endpoint = require_environment_variable("DOCUMENTINTELLIGENCE_ENDPOINT")
    key = require_environment_variable("DOCUMENTINTELLIGENCE_API_KEY")
    client = DocumentIntelligenceClient(
        endpoint=endpoint,
        credential=AzureKeyCredential(key),
    )
    operation_id = None

    try:
        with Path(pdf_path).open("rb") as document:
            poller = client.begin_analyze_document(
                "prebuilt-invoice",
                body=document,
            )
            result = poller.result()

        operation_id = poller.details.get("operation_id")
        if not result.documents:
            raise RuntimeError("Azure returned no analyzed invoice document.")

        fields = result.documents[0].fields or {}
        extracted = {}
        for field_name in FIELD_NAMES:
            field = fields.get(field_name)
            extracted[field_name] = {
                "value": normalized_value(field),
                "confidence": field.confidence if field else None,
            }
        return extracted
    except FileNotFoundError as exc:
        raise RuntimeError(f"Invoice file not found: {pdf_path}") from exc
    except HttpResponseError as exc:
        status = getattr(exc, "status_code", "unknown")
        raise RuntimeError(
            f"Azure request failed with status {status}: {exc.message}"
        ) from exc
    except AzureError as exc:
        raise RuntimeError(f"Azure SDK failure: {exc}") from exc
    finally:
        if operation_id:
            try:
                client.delete_analyze_result(
                    model_id="prebuilt-invoice",
                    result_id=operation_id,
                )
            except HttpResponseError as exc:
                print(f"Warning: could not delete Azure analysis result: {exc}")
        client.close()

Where does Azure stay isolated?

This module is the only base module that knows about Azure. It normalizes model fields and closes the client after each extraction.

Create evaluator.py with the field-level comparison logic.

NUMERIC_FIELDS = {"SubTotal", "TotalTax", "InvoiceTotal"}


def normalized_text(value):
    return " ".join(str(value).split()).casefold()


def values_match(field_name, expected, actual):
    if expected is None or actual is None:
        return expected is actual
    if field_name in NUMERIC_FIELDS:
        try:
            return abs(float(expected) - float(actual)) <= 0.01
        except (TypeError, ValueError):
            return False
    return normalized_text(expected) == normalized_text(actual)


def evaluate_fields(expected, extracted):
    field_results = []
    correct_fields = 0

    for field_name, expected_value in expected.items():
        actual_value = extracted.get(field_name, {}).get("value")
        confidence = extracted.get(field_name, {}).get("confidence")
        match = values_match(field_name, expected_value, actual_value)
        correct_fields += int(match)
        field_results.append(
            {
                "field": field_name,
                "expected": expected_value,
                "actual": actual_value,
                "confidence": confidence,
                "match": match,
            }
        )

    total_fields = len(expected)
    accuracy = correct_fields / total_fields if total_fields else 0.0
    return {
        "correct_fields": correct_fields,
        "total_fields": total_fields,
        "accuracy": round(accuracy, 4),
        "field_results": field_results,
    }

How are fields compared?

Text comparison normalizes whitespace and letter case. Monetary comparison uses the project-defined tolerance of 0.01.

Create validator.py with the deterministic business rules.

from datetime import date

REQUIRED_FIELDS = (
    "VendorName",
    "InvoiceId",
    "InvoiceDate",
    "DueDate",
    "SubTotal",
    "TotalTax",
    "InvoiceTotal",
)
MONEY_FIELDS = ("SubTotal", "TotalTax", "InvoiceTotal")


def field_value(extracted, field_name):
    return extracted.get(field_name, {}).get("value")


def validate_invoice(extracted, confidence_threshold=0.80):
    issues = []

    for field_name in REQUIRED_FIELDS:
        entry = extracted.get(field_name, {})
        value = entry.get("value")
        confidence = entry.get("confidence")
        if value is None or value == "":
            issues.append(
                {
                    "code": "missing_field",
                    "field": field_name,
                    "message": f"{field_name} is required.",
                }
            )
            continue
        if confidence is None or confidence < confidence_threshold:
            issues.append(
                {
                    "code": "low_confidence",
                    "field": field_name,
                    "message": (
                        f"{field_name} confidence is below "
                        f"{confidence_threshold:.2f}."
                    ),
                }
            )

    invoice_date = field_value(extracted, "InvoiceDate")
    due_date = field_value(extracted, "DueDate")
    if invoice_date and due_date:
        try:
            if date.fromisoformat(str(due_date)) < date.fromisoformat(
                str(invoice_date)
            ):
                issues.append(
                    {
                        "code": "invalid_date_order",
                        "field": "DueDate",
                        "message": "DueDate is earlier than InvoiceDate.",
                    }
                )
        except ValueError:
            issues.append(
                {
                    "code": "invalid_date",
                    "field": "InvoiceDate/DueDate",
                    "message": "One or both dates are not ISO-formatted dates.",
                }
            )

    amounts = {}
    for field_name in MONEY_FIELDS:
        value = field_value(extracted, field_name)
        if value is None:
            continue
        try:
            amounts[field_name] = float(value)
            if amounts[field_name] < 0:
                issues.append(
                    {
                        "code": "negative_amount",
                        "field": field_name,
                        "message": f"{field_name} cannot be negative.",
                    }
                )
        except (TypeError, ValueError):
            issues.append(
                {
                    "code": "invalid_amount",
                    "field": field_name,
                    "message": f"{field_name} is not numeric.",
                }
            )

    if all(field in amounts for field in MONEY_FIELDS):
        expected_total = round(amounts["SubTotal"] + amounts["TotalTax"], 2)
        if abs(expected_total - amounts["InvoiceTotal"]) > 0.01:
            issues.append(
                {
                    "code": "arithmetic_mismatch",
                    "field": "InvoiceTotal",
                    "message": (
                        f"Subtotal plus tax is {expected_total:.2f}, but the "
                        f"invoice total is {amounts['InvoiceTotal']:.2f}."
                    ),
                }
            )

    return issues

What does validation control?

The validator checks required fields and confidence. It also checks dates and monetary arithmetic without calling Azure.

Create router.py with the workflow payload contract.

def build_workflow_payload(
    source_file,
    expected_invoice_id,
    extracted,
    issues,
    evaluation,
):
    extracted_values = {
        field_name: entry.get("value")
        for field_name, entry in extracted.items()
    }
    extracted_invoice_id = extracted_values.get("InvoiceId")

    return {
        "source_file": source_file,
        "invoice_id": extracted_invoice_id or expected_invoice_id,
        "decision": "human_review" if issues else "auto_accept",
        "reasons": issues,
        "field_accuracy": evaluation["accuracy"],
        "extracted": extracted_values,
    }

What does the router produce?

The router turns evaluation evidence and validation issues into an explicit decision. Its payload is a future integration contract.

Create main.py with the pipeline orchestration.

import json
from pathlib import Path

from evaluator import evaluate_fields
from extractor import extract_invoice
from router import build_workflow_payload
from validator import validate_invoice

GROUND_TRUTH_PATH = Path("data/ground_truth.json")
INVOICE_DIR = Path("data/invoices")
ARTIFACT_DIR = Path("artifacts")


def write_json(path, value):
    path.write_text(json.dumps(value, indent=2), encoding="utf-8")


def run_pipeline():
    ground_truth = json.loads(GROUND_TRUTH_PATH.read_text(encoding="utf-8"))
    ARTIFACT_DIR.mkdir(parents=True, exist_ok=True)

    results = []
    workflow_payloads = []

    for filename, expected in ground_truth.items():
        service_error = None
        try:
            extracted = extract_invoice(INVOICE_DIR / filename)
        except RuntimeError as exc:
            extracted = {}
            service_error = str(exc)

        evaluation = evaluate_fields(expected, extracted)
        issues = validate_invoice(extracted)
        if service_error:
            issues.insert(
                0,
                {
                    "code": "service_error",
                    "field": None,
                    "message": service_error,
                },
            )

        payload = build_workflow_payload(
            source_file=filename,
            expected_invoice_id=expected.get("InvoiceId"),
            extracted=extracted,
            issues=issues,
            evaluation=evaluation,
        )
        workflow_payloads.append(payload)
        results.append(
            {
                "source_file": filename,
                "service_error": service_error,
                "evaluation": evaluation,
                "validation_issues": issues,
                "decision": payload["decision"],
            }
        )
        print(
            f"{filename}: accuracy={evaluation['accuracy']:.2%}, "
            f"decision={payload['decision']}"
        )

    average_accuracy = (
        sum(item["evaluation"]["accuracy"] for item in results) / len(results)
        if results
        else 0.0
    )
    report = {
        "dataset_size": len(results),
        "average_field_accuracy": round(average_accuracy, 4),
        "method": "Field-level comparison against labels created before inference.",
        "results": results,
    }
    review_queue = [
        payload
        for payload in workflow_payloads
        if payload["decision"] == "human_review"
    ]

    write_json(ARTIFACT_DIR / "evaluation_report.json", report)
    write_json(ARTIFACT_DIR / "workflow_payloads.json", workflow_payloads)
    write_json(ARTIFACT_DIR / "review_queue.json", review_queue)
    print(f"Wrote reports to {ARTIFACT_DIR.resolve()}")
    return report


if __name__ == "__main__":
    run_pipeline()

What does the pipeline coordinate?

The pipeline extracts and evaluates each labeled invoice. It records service failures as review evidence.

Create assessment.py with the evidence-based assessment generator.

import json
import unittest
from pathlib import Path

REPORT_PATH = Path("artifacts/evaluation_report.json")
ASSESSMENT_PATH = Path("artifacts/assessment.md")


def bullet_list(items):
    return "\n".join(f"- {item}" for item in items)


def build_assessment(report, tests_passed):
    live_integration_worked = all(
        item["service_error"] is None for item in report["results"]
    )

    demonstrated = [
        "Python modularity: extraction, evaluation, validation, routing, and assessment are separated.",
        "Data modelling: labeled expected values and workflow payloads use explicit JSON contracts.",
        "AI evaluation: field accuracy is calculated from labels created before inference.",
        "Deterministic controls: arithmetic, date, required-field, and confidence rules drive review decisions.",
    ]
    if live_integration_worked:
        demonstrated.append(
            "API integration: live Azure analyses completed and service-side results were marked for early deletion."
        )
    if tests_passed:
        demonstrated.append(
            "Automated testing: deterministic reliability tests passed without calling Azure."
        )

    partially_demonstrated = [
        "Security: secrets are session-only and excluded from Git, but managed identity and Key Vault are not implemented.",
        "Cloud operations: a managed Azure service is provisioned and cleaned up, but the application is not deployed.",
        "Aurachain integration: a workflow-ready contract exists, but no tenant API or live process connection is claimed.",
        "Evaluation quality: methodology is repeatable, but two synthetic invoices are too small for production conclusions.",
    ]
    if not live_integration_worked:
        partially_demonstrated.append(
            "API integration: the recorded Azure call failed and requires diagnosis before it can be claimed."
        )
    if not tests_passed:
        partially_demonstrated.append(
            "Automated testing: at least one deterministic test failed."
        )

    unverified = [
        "Production deployment, observability, concurrency, load testing, and service-level objectives.",
        "Performance on real invoice diversity, scans, handwriting, multiple languages, or multi-page documents.",
        "Live Aurachain authentication, API calls, role mapping, workflow execution, and audit evidence.",
        "Database persistence, RAG, agentic behavior, and production identity management.",
    ]

    return f"""# Practical Skills Assessment

Dataset size: {report['dataset_size']}

Average field accuracy: {report['average_field_accuracy']:.2%}

Automated tests passed: {tests_passed}

## Demonstrated

{bullet_list(demonstrated)}

## Partially Demonstrated

{bullet_list(partially_demonstrated)}

## Unverified

{bullet_list(unverified)}

## Method and Limitations

The harness compares each extracted field with labels written before inference, then applies deterministic business validation independently of the model. String comparison normalizes case and whitespace. Numeric comparison uses a project-defined tolerance of 0.01. A project-defined confidence threshold of 0.80 triggers review.

The dataset is intentionally small and synthetic. Its results assess implementation discipline, not production model quality. The second invoice contains a deliberately inconsistent total to prove that correct extraction and business validity are different questions.
"""


def main():
    if not REPORT_PATH.exists():
        raise SystemExit("Run python main.py before generating the assessment.")

    report = json.loads(REPORT_PATH.read_text(encoding="utf-8"))
    suite = unittest.defaultTestLoader.discover("tests")
    result = unittest.TextTestRunner(verbosity=2).run(suite)
    ASSESSMENT_PATH.write_text(
        build_assessment(report, result.wasSuccessful()),
        encoding="utf-8",
    )
    print(f"Wrote {ASSESSMENT_PATH}")
    raise SystemExit(0 if result.wasSuccessful() else 1)


if __name__ == "__main__":
    main()

How are skills classified?

The assessment reads recorded pipeline evidence. It separates demonstrated skills from partial evidence and unverified claims.

Create tests/test_reliability.py with the deterministic test suite.

import unittest

from evaluator import evaluate_fields
from router import build_workflow_payload
from validator import validate_invoice


def entry(value, confidence=0.99):
    return {"value": value, "confidence": confidence}


class ReliabilityTests(unittest.TestCase):
    def test_evaluator_counts_a_missing_field(self):
        expected = {"InvoiceId": "INV-1", "VendorName": "Demo Vendor"}
        extracted = {
            "InvoiceId": entry("INV-1"),
            "VendorName": entry(None, None),
        }

        evaluation = evaluate_fields(expected, extracted)

        self.assertEqual(evaluation["correct_fields"], 1)
        self.assertEqual(evaluation["accuracy"], 0.5)

    def test_arithmetic_mismatch_routes_human_review(self):
        extracted = {
            "VendorName": entry("Demo Vendor"),
            "InvoiceId": entry("INV-2"),
            "InvoiceDate": entry("2026-10-05"),
            "DueDate": entry("2026-11-04"),
            "SubTotal": entry(500.00),
            "TotalTax": entry(75.00),
            "InvoiceTotal": entry(580.00),
        }
        issues = validate_invoice(extracted)
        evaluation = {"accuracy": 1.0}

        payload = build_workflow_payload(
            "invoice_002.pdf",
            "INV-2",
            extracted,
            issues,
            evaluation,
        )

        self.assertIn("arithmetic_mismatch", {issue["code"] for issue in issues})
        self.assertEqual(payload["decision"], "human_review")

    def test_low_confidence_routes_human_review(self):
        extracted = {
            "VendorName": entry("Demo Vendor", 0.50),
            "InvoiceId": entry("INV-3"),
            "InvoiceDate": entry("2026-10-05"),
            "DueDate": entry("2026-11-04"),
            "SubTotal": entry(100.00),
            "TotalTax": entry(15.00),
            "InvoiceTotal": entry(115.00),
        }

        issues = validate_invoice(extracted)

        self.assertIn("low_confidence", {issue["code"] for issue in issues})


if __name__ == "__main__":
    unittest.main()

Why do these tests avoid Azure?

Python unittest exercises deterministic logic without making service calls. The tests stay fast and independent of credentials.

Create README.md with the project boundaries.

# Invoice Extraction Reliability Harness

This project evaluates Azure Document Intelligence invoice extraction against synthetic, labeled data. It does not authorize payments.

## Architecture

1. `generate_invoices.py` creates non-sensitive PDFs.
2. `extractor.py` is the only module that knows about Azure.
3. `evaluator.py` compares predictions with labels created before inference.
4. `validator.py` applies deterministic business rules.
5. `router.py` creates explicit workflow decisions and reasons.
6. `main.py` writes evaluation and workflow artifacts.
7. `assessment.py` classifies demonstrated, partial, and unverified skills.

## Reliability Method

- Labels are written before model inference.
- Accuracy is measured per field.
- Text comparison normalizes case and whitespace.
- Money comparison uses a project-defined tolerance of 0.01.
- Missing, low-confidence, invalid, or inconsistent data goes to human review.
- Azure service failures are review reasons, never successful extractions.

## Security Choices

- Synthetic documents only.
- API key and endpoint exist only in the current PowerShell session.
- Secret files and generated artifacts are ignored by Git.
- The client is closed after use.
- The analysis result is marked for early deletion after normalization.
- Debug logging with unredacted headers is not enabled.

## Known Limitations

- Two invoices cannot establish production accuracy.
- No real scans, handwriting, multiple languages, or multi-page cases are tested.
- Key-based authentication is used for a short learning exercise. Production should evaluate managed identity and secret-management requirements.
- There is no deployed service, database, monitoring system, or live Aurachain connection.

## Aurachain Boundary

`workflow_payloads.json` is an integration contract for the next project. The AI supplies extracted evidence. Deterministic rules supply the proposed route. Aurachain will own workflow state, authorization, task assignment, escalation, human approval, and audit history only after the learner's tenant version, API access, authentication, environment, and licensing are verified.

What does the README establish?

The README documents the architecture and reliability method. It also records the security choices and the boundary around any future Aurachain integration.

  • Save every file in VS Code.
  • Confirm that the Explorer sidebar lists each file from the reference.

The full base scaffold is now visible in VS Code. The next commands isolate its dependencies inside .venv.

  • Create the virtual environment by running:
python -m venv .venv

What does this command create?

Python creates an isolated environment in .venv. The folder is already excluded by .gitignore.

  • Activate the virtual environment by running:
.venv\Scripts\Activate.ps1

What does activation change?

Activation makes this terminal use the Python interpreter inside .venv. The environment name should appear at the start of the PowerShell prompt.

✔️ The environment is active

Your PowerShell prompt shows the .venv environment name. Package installation now stays inside the project.

ⓧ Script execution is blocked

PowerShell is blocking the activation script. Python documents a current-user execution policy for this Windows case.

  • Allow locally created activation scripts for your Windows user by running:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

What does this policy target?

The command changes the execution policy for the current Windows user. It allows the documented PowerShell virtual-environment activation path.

  • Confirm the policy change if PowerShell asks for approval.
  • Activate the virtual environment again by running:
.venv\Scripts\Activate.ps1

What confirms activation?

The PowerShell prompt should now show the .venv environment name. Future package commands in this terminal use the isolated interpreter.

Activation still blocked?

Confirm that the terminal shell is PowerShell. Check that the .venv folder exists in the Explorer sidebar.

Get help with your execution policy: help me activate this Windows virtual environment safely

  • Install the pinned dependencies into the active environment by running:
python -m pip install -r requirements.txt

What gets installed?

The command reads the two exact package pins from requirements.txt. The active environment keeps both packages local to this project.

Dependency installation failed?

Confirm that the PowerShell prompt shows the active .venv environment. Confirm that requirements.txt contains both pins exactly.

Ask for environment-specific help: help me diagnose this pinned Python package installation

Set session-only credentials and verify

Current-session environment variables disappear when this PowerShell process closes. This keeps the endpoint and API key out of project files.

Keep the API key off screen

The next command contains a live credential after you replace the placeholder. Do not save it in a file or include it in a screenshot.

  • Switch back to the Azure portal Keys and Endpoint page from earlier.
  • Copy the resource endpoint.
  • Copy one API key.
  • Replace your-endpoint-here with the copied endpoint in the first command below.
  • Replace your-key-here with the copied key in the second command below.
  • Set both values in the current PowerShell session by running:
$Env:DOCUMENTINTELLIGENCE_ENDPOINT = "your-endpoint-here"
$Env:DOCUMENTINTELLIGENCE_API_KEY = "your-key-here"

Where do the credentials live?

PowerShell stores both values in the current process environment. Closing this terminal removes them from that session.

The final check tests the runtime and both pinned imports. Before you run it, do you expect the active environment to report a supported version and print the readiness message?

  • Verify the complete local setup by running:
python --version
python -c "import azure.ai.documentintelligence, reportlab; print('ready')"

What does the final check prove?

The first command confirms Python 3.9 or later. The second command proves that both pinned libraries import from the active environment.

You should see a supported Python version followed by ready. That's the foundation complete: the workspace is isolated and the Azure cost gate is proven.

The import check did not print ready?

Confirm that the PowerShell prompt still shows the active .venv environment. Rerun the dependency installation if the output mentions a missing package.

Get help with the exact output: help me diagnose why the workspace import check failed

Your secure workspace now has pinned code and session-only credentials. Next, you will generate synthetic invoices and inspect the first live extraction.

Generate Invoices and Run an Extraction

Your secure workspace is ready with pinned dependencies, session-only credentials, and an F0 resource. Now you can make the first live call to Azure Document Intelligence.

A raw extraction can look convincing without proving that any field is correct. You need a controlled baseline before you can measure reliability.

This step generates two synthetic PDF files from known data. You then extract one invoice while keeping the existing labels unchanged.

In this step, get ready to:
  • Review the synthetic invoice definitions and pre-inference labels.
  • Generate two safe invoice PDFs without real financial data.
  • Analyze the first invoice and inspect its normalized fields.
Review the synthetic source and labels

The existing generate_invoices.py uses ReportLab to turn fixed Python dictionaries into invoice files. The ground truth in data/ground_truth.json records the expected answers before inference.

  • In VS Code, select generate_invoices.py in the Explorer sidebar.
  • Confirm INVOICES defines invoice_001.pdf and invoice_002.pdf.
  • Select data/ground_truth.json in the Explorer sidebar.
  • Confirm each invoice label contains VendorName, CustomerName, InvoiceId, InvoiceDate, DueDate, SubTotal, TotalTax, and InvoiceTotal.

Why freeze the labels first?

Ground truth supplies a fixed target for later evaluation. Writing it before inference prevents model output from influencing the expected answers.

The labels stay unchanged during this extraction. That separation makes the next accuracy measurement meaningful.

Generate the synthetic invoices

Both invoices use invented organizations and controlled monetary values. Each page includes a footer that identifies it as a synthetic document that is not payable.

  • Switch back to the activated PowerShell terminal in VS Code.
  • Generate both invoice PDFs by running:
python generate_invoices.py

What does this command create?

  • main() creates the data/invoices directory when needed.
  • write_invoice() renders each invoice as an A4 document.
  • money() formats the monetary values with two decimal places.
  • The footer clearly separates these test documents from payable invoices.

PowerShell should report two created file paths. The output should name invoice_001.pdf and invoice_002.pdf.

  • Return to the Explorer sidebar in VS Code.
  • Expand the data folder.
  • Expand the invoices folder.
  • Select invoice_001.pdf.
  • Confirm the first invoice shows Synthetic test document. Not payable. in its footer.
  • Select invoice_002.pdf.
  • Confirm the second invoice shows the same synthetic-document footer.

PDF files missing?

  • Check that the PowerShell prompt points to the folder containing generate_invoices.py.
  • Confirm the prompt still shows the activated .venv environment.
  • Run the generation command again after correcting the folder or environment.

Ask for help with the exact output if the files still do not appear: help me troubleshoot why generate_invoices.py did not create the two PDF files

Run the first Azure extraction

The existing extractor.py is the boundary between your local harness and Azure Document Intelligence. It sends one invoice to the prebuilt-invoice model.

The extractor converts model-specific fields into plain values with confidence scores. Accuracy and business validity remain unmeasured at this stage.

  • Switch back to the activated PowerShell terminal in VS Code.

Before you run the extraction, do you expect the returned object to contain raw field objects or normalized values with confidence scores?

  • Analyze invoice_001.pdf by running:
python -c "from extractor import extract_invoice; print(extract_invoice(r'data\invoices\invoice_001.pdf'))"

What happens during extraction?

  • extract_invoice() reads DOCUMENTINTELLIGENCE_ENDPOINT and DOCUMENTINTELLIGENCE_API_KEY from the current session.
  • The prebuilt-invoice model analyzes the local document.
  • normalized_value() converts dates, currency amounts, numbers, and text into plain values.
  • The cleanup block requests early deletion of the analysis result and closes the Azure client.

PowerShell should print a Python dictionary containing all eight normalized field entries. Every entry should contain value and confidence.

The returned values should describe the synthetic Ubuntu Logistics invoice you just inspected.

Extraction did not return the fields?

  • Check whether the message names a missing session environment variable.
  • Set the endpoint or API key again from the existing Azure resource if the PowerShell session changed.
  • Confirm data/invoices/invoice_001.pdf still exists in the Explorer sidebar.
  • Confirm the endpoint and key belong to the same Azure Document Intelligence resource.

Share only the redacted error when asking for help: help me diagnose why extract_invoice did not return normalized invoice fields

You have your first live extraction. The harness now turns a synthetic invoice into structured evidence while keeping credentials out of project files.

Your synthetic dataset now produces normalized model output. Next, you will compare those predictions with the frozen labels to measure field-level accuracy.

Measure Field-Level Accuracy

The first live Azure Document Intelligence extraction returned normalized invoice fields with confidence scores. You now need evidence that shows which predictions match labels created before inference.

This step uses field-level scoring across both synthetic invoices. The resulting report preserves the model output beside the expected answer for every field.

A plausible extraction can still contain missing or incorrect fields. The second invoice reveals a deeper limit: extraction accuracy cannot establish business validity.

In this step, get ready to:
  • Review the field comparison policy.
  • Run the two-invoice evaluation pipeline.
  • Challenge the accuracy result with the second invoice's arithmetic.
Understand the field comparison policy

The labels in data/ground_truth.json form your ground truth. They existed before inference, so the model could not influence the expected answers.

How is field accuracy calculated?

  • The normalized_text() function collapses repeated whitespace. It also makes text comparison case-insensitive.
  • The values_match() function allows a monetary difference of up to 0.01.
  • A missing expected or extracted value matches only when both values are None.
  • The evaluate_fields() function records every field result. It calculates the proportion of matching fields.
  • Select evaluator.py from the Visual Studio Code file list.
  • Locate values_match() to identify the text comparison policy.
  • Locate evaluate_fields() to trace how each field contributes to the final accuracy.
Run the two-invoice evaluation

The main.py pipeline pairs each PDF with its pre-inference labels. A failed service call becomes service_error evidence.

Every successful extraction passes through evaluate_fields(). The pipeline then writes the complete comparison to artifacts/evaluation_report.json.

  • Return to the activated PowerShell environment from the previous step.

Before you run the pipeline, do you expect both invoices to produce the same field accuracy?

  • Run the two-invoice evaluation pipeline by entering this command:
python main.py

What does this command prove?

  • The run_pipeline() function reads the pre-inference labels.
  • The extract_invoice() function processes each synthetic PDF through the live model.
  • The evaluate_fields() function scores every labeled field.
  • The write_json() function stores the evaluation evidence in the artifacts folder.

You should see one line containing invoice_001.pdf with accuracy=. A second accuracy line should name invoice_002.pdf.

You should then see a line beginning with Wrote reports to. That confirms the runtime artifacts now exist.

That is the evidence trail working: every invoice now has a measurable field score.

Pipeline stopped before writing the report?

Your credentials stay in the current process. Re-entering them in a replacement PowerShell window keeps them out of project files.

  • Confirm the PowerShell prompt still shows the activated .venv.
  • Re-enter the DOCUMENTINTELLIGENCE_ENDPOINT value if the PowerShell window changed.
  • Re-enter the DOCUMENTINTELLIGENCE_API_KEY value if the PowerShell window changed.
  • Run python main.py again from the activated environment.

Help me diagnose why the invoice evaluation pipeline stopped before writing its report.

Inspect the report and challenge the score

The JSON report preserves the evidence behind each accuracy score. This lets you inspect individual predictions without relying on a single percentage.

  • Select artifacts/evaluation_report.json from the Visual Studio Code file list.
  • Confirm dataset_size is 2.
  • Confirm method says Field-level comparison against labels created before inference.
  • Locate the average_field_accuracy value.
  • Inspect the field_results entries for invoice_001.pdf.
  • Inspect the field_results entries for invoice_002.pdf.
  • Locate the service_error value for each invoice.

How do I read each result?

  • The expected value comes from the label written before inference.
  • The actual value comes from the normalized model output.
  • The confidence value records the model's confidence for that field.
  • The match value records the outcome of your comparison policy.
  • A missing extracted value appears as null under actual.
  • A successful service call records null under service_error.

Before you calculate, do you think a high field-matching score guarantees that the invoice total is valid?

  • Select data/invoices/invoice_002.pdf from the Visual Studio Code file list.
  • Read the displayed subtotal of 500.00.
  • Read the displayed VAT amount of 75.00.
  • Calculate 500.00 + 75.00.
  • Compare your sum with the displayed total of 580.00.

You get 575.00. The invoice states 580.00.

The shortfall is intentional. Field-level accuracy can still be high because the model copied the document faithfully.

The document itself contains invalid arithmetic. Extraction accuracy cannot approve it.

Your report now measures model output against pre-inference labels. Next, deterministic rules will turn invoice problems into human-review reasons.

Validate Invoices and Route Review

Your evaluation report measures how closely Azure Document Intelligence matches the labels. The inconsistent total on invoice_002.pdf proves that a correct prediction can still describe an invalid invoice.

Business rules must now decide whether each result can continue automatically. This step applies deterministic checks before building a review-ready integration contract.

In this step, get ready to:
  • Confirm the deterministic validation rules.
  • Trace each invoice from extraction evidence to a workflow decision.
  • Generate the workflow payloads plus the filtered review queue.
Trace the deterministic checks

Extraction confidence estimates how strongly the model supports a field prediction. Deterministic validation applies the same policy to every extracted result.

  • In Visual Studio Code, select validator.py from the project file list.
  • Locate validate_invoice() to confirm the policy uses a confidence threshold of 0.80.
  • Check REQUIRED_FIELDS to confirm the seven fields that must be present.
  • Trace the date checks to confirm that DueDate cannot precede InvoiceDate.
  • Trace MONEY_FIELDS to confirm that each available amount must be numeric.
  • Trace the arithmetic check to confirm that SubTotal plus TotalTax must match InvoiceTotal within 0.01.

Why Validation Stays Independent

The validator.py module does not call Azure. It receives normalized fields from the extractor.

This separation makes every business rule repeatable. The same input always produces the same findings.

Trace the workflow contract

A workflow payload turns technical findings into a stable message for another system. It carries the extracted evidence plus an explicit routing decision.

  • In Visual Studio Code, select router.py from the project file list.
  • Locate build_workflow_payload() to confirm that an empty issue list produces auto_accept.
  • Trace the same function to confirm that any issue produces human_review.
  • Switch back to main.py in your project workspace.
  • Locate the call to validate_invoice(extracted) to confirm that validation runs after field evaluation.
  • Locate the service_error handling to confirm that a failed extraction becomes a review reason.
  • Locate review_queue to confirm that it keeps only payloads with the human_review decision.

What the Contract Proves

Each payload contains the source file plus the invoice identifier. It also contains extracted values plus field accuracy.

The payload is an integration contract for a future Aurachain workflow. A live Aurachain connection remains outside this project.

Run the controlled routing pipeline

Before you run the pipeline, which invoice do you expect the deterministic policy to route for review?

  • Reprocess both invoices through evaluation plus validation by running this command:
python main.py

What Does This Run Produce?

The pipeline refreshes the extraction evidence for both source invoices. It applies deterministic validation before building each routing decision.

  • The artifacts/evaluation_report.json file records the refreshed field-level evidence.
  • The artifacts/workflow_payloads.json file records one explicit payload per source invoice.
  • The artifacts/review_queue.json file keeps only invoices that require human review.

You'll see one summary line for each source file. The line for invoice_002.pdf should show a decision of human_review.

That is the control gap closed. The inconsistent invoice now stops at human review instead of flowing forward automatically.

Reports Missing or Showing Service Issues?

  • Stay in the activated PowerShell session from earlier so the session-only credentials remain available.
  • Confirm the two expected PDFs remain in data/invoices.
  • Confirm PowerShell is still inside the project folder containing main.py before rerunning the command.

Help me diagnose the pipeline output.

  • In Visual Studio Code, select artifacts/workflow_payloads.json from the project file list.
  • Confirm the file contains two payloads with source files plus invoice identifiers.
  • Confirm each payload contains extracted values plus field accuracy.
  • Confirm each payload contains a decision plus its review reasons.
  • Select artifacts/review_queue.json from the project file list.
  • Confirm the filtered array includes invoice_002.pdf with an arithmetic_mismatch reason for InvoiceTotal.

The workflow payload file provides one contract per invoice. The review queue isolates the exceptions that need human accountability.

Your pipeline now measures model output before enforcing deterministic business rules. Next, you'll prove that logic with repeatable tests and turn the recorded evidence into a skills assessment.

Test the Harness and Assess Skills

Your pipeline already catches the inconsistent total in invoice_002.pdf. It routes that invoice to human review.

A professional reliability claim needs repeatable tests. It also needs documented limits that separate proven skills from future work.

In this step, get ready to:
  • Prove the deterministic reliability logic with three service-independent tests.
  • Audit the architecture documentation against the project evidence.
  • Generate an evidence-based skills assessment.
Challenge the reliability rules

The Python unittest suite supplies controlled inputs directly to the evaluator, validator, and router. This isolates your business rules from Azure Document Intelligence behavior.

Before you run the suite, predict whether any test should make a live Azure request.

  • Run the three deterministic tests from your activated PowerShell environment by running:
python -m unittest discover -s tests -v

What does this command test?

  • The discover command finds the tests in tests.
  • The -s option sets the test discovery directory.
  • The -v option lists each test result.
  • The suite checks missing extraction evaluation.
  • The suite checks arithmetic mismatch routing.
  • The suite checks low-confidence validation.

You should see three tests listed as successful. That result proves the deterministic reliability logic behaves as expected.

Seeing a failed test?

A failure in the missing-field test points to the comparison logic in evaluate_fields().

A routing failure points to validate_invoice() or build_workflow_payload().

Use the failing test name to narrow the problem. Help me diagnose a failing reliability test.

Audit the project claims

The test result proves the deterministic logic. The project documentation defines what that evidence allows you to claim.

The Aurachain payload remains a future integration contract. The project does not claim a live workflow connection.

  • Return to README.md in Visual Studio Code.
  • Confirm the Architecture section assigns one responsibility to each module.
  • Confirm the Reliability Method section records labels before inference.
  • Confirm the Reliability Method section treats service failures as review reasons.
  • Confirm the Security Choices section keeps credentials in the current PowerShell session.
  • Confirm the Known Limitations section identifies the missing production evidence.
  • Confirm the Aurachain Boundary section describes a future integration contract.

The documentation should make the claim boundary visible. The harness demonstrates a workflow contract while leaving the live connection unverified.

Why document the boundary?

Reliable engineering includes honest reporting. A clear boundary prevents a generated payload from being presented as a deployed workflow.

The limitations also show where future testing must focus. Evidence keeps the assessment defensible.

Turn evidence into an assessment

The assessment.py script reads the recorded evaluation report. It reruns the deterministic tests before writing artifacts/assessment.md.

Before you run the script, predict which evidence category should contain a live Aurachain integration.

  • Generate the skills assessment from the recorded project evidence by running:
python assessment.py

How is the assessment produced?

  • REPORT_PATH points to artifacts/evaluation_report.json.
  • The test loader discovers the suite in tests.
  • build_assessment() classifies the recorded evidence.
  • A failed test makes the script exit with a nonzero status.
  • A completed run writes artifacts/assessment.md.

You should see the same three tests succeed. PowerShell should then confirm that the assessment file was written.

Assessment file missing?

Confirm that artifacts/evaluation_report.json still exists before rerunning the assessment.

If a deterministic test fails, use its verbose result to find the affected module. Fix that failure before treating the assessment as current evidence.

Help me diagnose why the assessment was not generated.

  • Return to the project editor after the command finishes.
  • Select artifacts/assessment.md.

What should the assessment contain?

  • The Demonstrated section records the modular Python design.
  • The Demonstrated section records explicit JSON contracts.
  • The Demonstrated section records pre-inference field evaluation.
  • The Demonstrated section records deterministic controls.
  • The Demonstrated section records passing automated tests.
  • The Partially Demonstrated section limits the session-only security claim.
  • The Partially Demonstrated section limits the cloud operations claim.
  • The Partially Demonstrated section limits the Aurachain integration claim.
  • The Partially Demonstrated section limits conclusions from the small synthetic dataset.
  • The Unverified section includes production deployment.
  • The Unverified section includes diverse real documents.
  • The Unverified section includes a live Aurachain integration.
  • The Unverified section includes production identity management.
  • The Method and Limitations section explains how the evidence was measured.

You now have repeatable test evidence plus a documented skills boundary. Your reliability harness can be demonstrated without overstating what it proves.

Secret mission

Add a Missing-Date Edge Case

Extend the synthetic dataset with a third invoice that omits its due date. Then prove that accurate ground truth and deterministic review policy can reach different conclusions about the same field.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your work available, pause your API activity, or delete the project entirely. This project has no ongoing paid-tier charge while the Azure Document Intelligence resource remains on F0.

Resources you used:

  • A dedicated Azure resource group containing one Azure Document Intelligence resource on the F0 pricing tier.
  • A current Windows PowerShell session containing the endpoint and API key in session-only environment variables.
  • A local project workspace containing the source code, the .venv/ environment, three synthetic PDFs in data/invoices/, the labels in data/ground_truth.json, and the generated evidence in artifacts/.

Keep everything running

No action is needed. Choose this if you plan to keep testing the reliability harness.

  • Keep the dedicated Azure resource group on the F0 pricing tier.
  • Keep the local workspace so you retain the synthetic invoices and reliability evidence.
  • Keep credentials in the current PowerShell session only while you are actively working.
  • Return to the resource overview before future API use to confirm that Pricing tier still shows F0.

Pause - I'll come back to this later

Stop making API calls while preserving the Azure resource and local workspace. Azure Document Intelligence has no pause action in this project, so clearing the current session credentials prevents further calls from that PowerShell process.

  • Clear both session-only credential values by running these commands:
$Env:DOCUMENTINTELLIGENCE_ENDPOINT = $null
$Env:DOCUMENTINTELLIGENCE_API_KEY = $null

What do these commands do?

  • The first command removes DOCUMENTINTELLIGENCE_ENDPOINT from the current PowerShell session.
  • The second command removes DOCUMENTINTELLIGENCE_API_KEY from the current PowerShell session.
  • The Azure resource and local project files remain available for your return.
  • Close the current PowerShell window after the commands finish.

Your code and generated evidence remain intact. You can resume later by activating .venv/ and setting fresh session values.

Delete - I don't want to use this again

Remove the Azure resource group and local workspace if you have finished with the harness. This option permanently removes the cloud resource and your local project evidence.

Deletion is irreversible

Deleting the dedicated resource group permanently removes its Azure Document Intelligence resource. Save any evidence you want to keep before continuing.

Delete the Azure resource group:

  • Return to the Azure portal tab from earlier.
  • Select the dedicated resource group containing your F0 Azure Document Intelligence resource.
  • Check that the resource group contains only the resource created for this project.
  • Select Delete resource group.
  • Type the displayed resource group name into the confirmation field.
  • Confirm the resource group deletion.

Azure processes the deletion after confirmation. You should no longer see the dedicated resource group once the deletion finishes.

Clear the current PowerShell credentials:

  • Remove both session-only credential values by running these commands:
$Env:DOCUMENTINTELLIGENCE_ENDPOINT = $null
$Env:DOCUMENTINTELLIGENCE_API_KEY = $null

What does this remove?

These commands remove the Azure endpoint and API key from the current PowerShell process. They do not write either credential to a file.

  • Close the current PowerShell window after the commands finish.

Remove the local workspace:

Finding the correct folder deserves an extra check because deleting it removes the source code and all generated evidence.

  • Press the Windows key to open system search.
  • Enter main.py in the search field.
  • Open the file location for the project result.
  • Confirm that the containing folder also includes assessment.py, router.py, data/, artifacts/, and .venv/.
  • Move one level above the confirmed project folder.
  • Select the confirmed project folder.
  • Press the Delete key to remove it.
  • Empty the Windows Recycle Bin after checking that you selected the correct folder.

The Azure resource group and local reliability harness are now removed. Your synthetic invoice data and generated assessment no longer remain on this machine.

Nice Work!

Nice Work!

You brought the invoice reliability harness across the finish line. Your tested pipeline now turns model predictions into evidence-backed workflow decisions.

You've learned how to:

  • Build a secure extraction pipeline that sends synthetic invoice PDFs to Azure Document Intelligence. Session-only credentials remain outside project files.
  • Measure field-level accuracy against labels created before inference. Deterministic validation routes invalid or uncertain invoices to human review.
  • Generate workflow contracts as JSON reports. Azure-independent tests prove the business rules repeatably. assessment.md separates demonstrated skills from unverified claims.
  • Secret Mission: Add a missing-date edge case with invoice_003.pdf. The required-field policy records DueDate as missing. The router sends the invoice to human review.

You can now demonstrate why accurate extraction still needs business validation before workflow authorization.

Ready to quiz yourself?