AI Fraud Detection for MPLADS Funds

Build a hybrid ML dashboard to detect anomalies in 96k MPLADS payment records.

Introduction

30 Second Summary

Every year, Members of Parliament across India collectively spend thousands of crores of public money on local development projects through the MPLADS scheme. With over 96,000 vendor-level payment records flowing through hundreds of constituencies, spotting suspicious spending patterns manually is practically impossible.

In this project, you will build an AI-powered anomaly detection platform that ingests real MPLADS expenditure data from the 18th Lok Sabha, runs hybrid detection combining government compliance rules with Isolation Forest machine learning, and surfaces explainable risk alerts through an interactive Streamlit dashboard.

What You'll Build

Run the finished dashboard, filter by state or MP, and instantly see which records are flagged as suspicious. Click any alert and read a plain-English explanation of exactly why it was raised, backed by the original data row.

By the end of this project, you'll have:

  • A data ingestion pipeline that cleans 96,541 real MPLADS payment records. Upload the CSV and see instant quality stats: missing fields handled, inconsistent values fixed, and duplicate records flagged automatically.
  • A hybrid anomaly detection engine that layers rule-based compliance checks (vendor concentration, statistical outliers, Benford's Law) on top of Isolation Forest ML. You can view composite risk scores that catch suspicious patterns no single rule would find.
  • An interactive Streamlit monitoring dashboard with role-based views, a state-wise risk bar chart, and drill-down investigation panels. Filter by state, constituency, or severity, then click any flagged record to see the full explanation and underlying data row.
  • Secret Mission: Add NLP-powered duplicate work detection using TF-IDF cosine similarity to catch copy-paste projects across constituencies.

Are there any prerequisites?

This is a hard-difficulty project. You should be comfortable with Python, pandas, and basic scikit-learn concepts before starting. Familiarity with Streamlit is helpful but not required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This project tackles a real government monitoring challenge, so understanding the problem space first will make every technical decision ahead feel intentional.

Set Up the Project and Install Dependencies

You know what you're building: a hybrid detection platform that combines rule-based compliance checks with machine learning to flag suspicious MPLADS expenditures. Before you can write any detection logic or dashboard code, the right tools and data need to be in place.

In this step, you'll scaffold the project folder, create a virtual environment, install the key Python packages, and download the real MPLADS dataset with 96,541 vendor-level payment records.

In this step, get ready to:
  • Create the project folder structure and Python virtual environment.
  • Install Streamlit, scikit-learn, Plotly, pandas, and openpyxl.
  • Download the MPLADS 18th Lok Sabha vendor-wise expenditure dataset.
Create the project folder and virtual environment

Your detection pipeline needs a clean workspace with separate directories for raw data, the detection engine code, and the dashboard code.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder and its subdirectories by running these commands:
mkdir mplads_monitor
cd mplads_monitor
mkdir data
mkdir engine
mkdir dashboard
  • Confirm the folder structure is in place by running:
ls

You should see dashboard, data, and engine listed.

  • Create a Python virtual environment by running:
python -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python workspace. Packages you install inside it stay separate from your system-wide Python, so different projects can use different library versions without conflicting.

  • Activate the virtual environment:

macOS

  • Run this command to activate:
source venv/bin/activate

Windows

  • Run this command to activate:
venv\Scripts\activate

Linux

  • Run this command to activate:
source venv/bin/activate

Your terminal prompt should now show (venv) at the beginning. This confirms packages will install into the isolated environment.

Don't see (venv) in your prompt?

  • On macOS or Linux, make sure you typed source before the path. Without it, the script runs in a subshell and the activation does not persist.
  • On Windows PowerShell, you may need to allow script execution first. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned then try the activation command again.
  • If python -m venv venv failed earlier, check that python --version returns 3.11 or higher. If your system uses python3 instead of python, run python3 -m venv venv and use python3 for all commands going forward.

Help me activate my Python virtual environment.

Install the project dependencies

The detection pipeline relies on five packages. Streamlit powers the monitoring dashboard. scikit-learn provides the Isolation Forest algorithm for ML-based anomaly detection.

  • Install all five packages by running:
pip install streamlit==1.62.0 scikit-learn==1.9.0 plotly==7.0.0 pandas openpyxl

What do these packages do?

  • streamlit builds the interactive web dashboard for exploring alerts and risk scores.
  • scikit-learn provides the Isolation Forest model that detects statistical anomalies in expenditure patterns.
  • plotly renders interactive charts like state-wise risk bar charts.
  • pandas handles all the data loading, cleaning, and analysis of the 96k+ expenditure records.
  • openpyxl is a pandas dependency that adds Excel file support.
  • Verify the installation succeeded by running:
python -c "import streamlit; import sklearn; import plotly; print('All good')"

You should see All good printed in your terminal.

Seeing an import error or install failure?

  • If pip install failed with a message about no matching distribution for scikit-learn, your Python version may be below 3.11. Run python --version to check. scikit-learn==1.9.0 requires Python 3.11 or higher.
  • Make sure your (venv) prefix is visible. If it is not, re-activate your virtual environment before running pip install.

Help me fix my package installation.

Download the MPLADS dataset

The pipeline needs real expenditure data to analyze. The 18th Lok Sabha vendor-wise dataset on Dataful.in contains 96,541 payment records covering 508 constituencies and 14,754 vendors.

  • Open the MPLADS 18th Lok Sabha vendor-wise dataset page in your browser.
  • Scroll down past the data preview to the download buttons.
  • Click the CSV button to download the dataset.
  • Move or save the downloaded file into the data/ folder inside your mplads_monitor project.
  • Rename the file to mplads_vendor_data.csv if it downloaded with a different name.

What fields does this dataset contain?

Each row is a single vendor payment for an MPLADS work. The key fields you will use throughout this project are:

  • state, implementing_district, loksabha_constituency for geographic grouping.
  • loksabha_MP_name and vendor_name for tracking who recommended and who received funds.
  • expenditure_amount and expenditure_date for detecting spending outliers and temporal patterns.
  • work for categorizing and comparing project types across constituencies.
  • Confirm the file is in place by running:
ls data/

You should see mplads_vendor_data.csv listed.

Don't see the file?

  • Check that you moved the file into the data/ folder inside mplads_monitor, not the project root.
  • Make sure the file is named exactly mplads_vendor_data.csv. The data loading code in later steps expects this exact filename.
  • If your browser saved the file with a .csv.download or .csv (1) suffix, rename it to remove the extra extension.

Help me get the MPLADS dataset file into the right location.

Before you run the next command, what do you expect to see? Will the output appear in your terminal, or somewhere else?

  • Verify that Streamlit is working by running:
streamlit hello

What should I see?

A Streamlit demo app opens automatically in your browser at http://localhost:8501. The page shows interactive demo widgets you can play with.

Your terminal stays busy serving the app. When you are done exploring, press Ctrl+C in your terminal to stop the Streamlit server.

Streamlit didn't open?

  • If your terminal says the command was not found, make sure your virtual environment is still active. Look for the (venv) prefix in your prompt.
  • If the browser did not open automatically, copy http://localhost:8501 from the terminal output and paste it into your browser manually.

Help me get Streamlit running.

  • Press Ctrl+C in your terminal to stop the Streamlit server before continuing.

Your project scaffold is ready, your packages are installed, and 96,541 real MPLADS payment records are waiting in your data folder. Next up, you'll build the data ingestion pipeline that cleans this messy government data into something your detection engine can analyze.

Ingest and Clean the MPLADS Data

Your project folder is set up and all dependencies are installed. Now it's time to bring in the real data.

The MPLADS vendor dataset has expenditure amounts stored as comma-separated strings, inconsistent date formats, and missing values scattered across key fields. If you feed this directly into your anomaly detection engine, it will flag data quality problems as fraud, producing noise instead of real signals.

In this step, get ready to:
  • Write a load_and_clean() function that handles all the messy data transformations.
  • Create a get_quality_report() function that produces dimension stats and data quality metrics.
  • Run the pipeline on the real MPLADS dataset and verify the cleaned output.
Create the data cleaning function

The load_and_clean() function is the foundation of your entire detection pipeline. It reads the raw CSV, fixes type mismatches, handles missing data, and returns a clean DataFrame ready for analysis.

  • Create a new file called data_loader.py inside the engine/ folder of your mplads_monitor/ project, then paste the following imports and function start:
import pandas as pd
import numpy as np
from pathlib import Path


def load_and_clean(filepath="data/mplads_vendor_data.csv"):
    """Load and clean the MPLADS vendor-wise expenditure dataset."""
    df = pd.read_csv(filepath)

    cleaning_log = []

    # Strip whitespace from string columns
    str_cols = df.select_dtypes(include="object").columns
    for col in str_cols:
        df[col] = df[col].str.strip()

What does this code do?

  • pd.read_csv(filepath) loads the raw CSV into a pandas DataFrame with all 96,541 records.
  • The whitespace stripping loop removes invisible leading and trailing spaces. Without this, "Unknown " and "Unknown" would be treated as different vendors.
  • cleaning_log is a list that will record every transformation the function performs, so you can trace what happened to the data later.
  • Save engine/data_loader.py. The function is not complete yet. You will finish it in the next chunk.
  • Still in engine/data_loader.py, add the following code directly below the last line (df[col] = df[col].str.strip()), keeping the same indentation level inside the function:
    # Convert expenditure_amount from string to numeric
    original_count = len(df)
    df["expenditure_amount"] = (
        df["expenditure_amount"]
        .astype(str)
        .str.replace(",", "", regex=False)
    )
    df["expenditure_amount"] = pd.to_numeric(df["expenditure_amount"], errors="coerce")
    failed = df["expenditure_amount"].isna().sum()
    cleaning_log.append(
        f"Converted {original_count} amount strings to numeric, {failed} failed"
    )

    # Parse expenditure_date to datetime
    df["expenditure_date"] = pd.to_datetime(df["expenditure_date"], format="mixed", dayfirst=True, errors="coerce")
    date_failed = df["expenditure_date"].isna().sum()
    cleaning_log.append(f"Parsed dates, {date_failed} could not be parsed")

    # Drop rows where both amount and vendor are missing
    before_drop = len(df)
    df = df.dropna(subset=["expenditure_amount", "vendor_name"], how="all")
    cleaning_log.append(f"Dropped {before_drop - len(df)} rows with both amount and vendor missing")

What does this code do?

  • str.replace(",", "") removes commas from amount strings so "1,50,000" becomes "150000". Then pd.to_numeric() converts it to a proper number. Values that cannot be converted become NaN.
  • pd.to_datetime() with format="mixed" handles multiple date formats in one column. The dayfirst=True flag tells pandas that 15/03/2024 means March 15, not October 3.
  • dropna(how="all") only removes rows where BOTH amount and vendor are missing. Rows missing only one field are kept for analysis.
  • Save engine/data_loader.py.
  • Verify the module loads without syntax errors by running this command from the mplads_monitor/ directory:
python -c "import engine.data_loader; print('Syntax OK')"

You should see Syntax OK printed. If you see a SyntaxError, double-check that the indentation in your function is consistent.

Seeing a ModuleNotFoundError?

  • Make sure you are running the command from the mplads_monitor/ directory, not from inside engine/.
  • Check that your file is saved as data_loader.py (not dataloader.py or data-loader.py).
  • Help me debug this.
  • Still in engine/data_loader.py, add the final section of the load_and_clean() function directly below the last cleaning_log.append(...) line, keeping the same indentation inside the function:
    # Fill missing categorical fields
    cat_fills = {
        "vendor_name": "Unknown",
        "implementing_agency_name": "Unknown",
        "work": "Unspecified",
        "state": "Unknown",
        "loksabha_constituency": "Unknown",
        "loksabha_MP_name": "Unknown",
        "payment_status": "Unknown",
    }
    for col, fill_val in cat_fills.items():
        if col in df.columns:
            count = df[col].isna().sum()
            df[col] = df[col].fillna(fill_val)
            if count > 0:
                cleaning_log.append(f"Filled {count} missing values in '{col}' with '{fill_val}'")

    # Add record_id for traceability
    df = df.reset_index(drop=True)
    df["record_id"] = df.index

    for entry in cleaning_log:
        print(f"  [CLEAN] {entry}")

    return df

What does this code do?

  • The cat_fills dictionary maps each categorical column to its fill value. Most fields get "Unknown", while work gets "Unspecified" to distinguish missing work descriptions from missing vendor names.
  • record_id gives every row a unique numeric identifier. This makes individual records traceable when the rule engine flags anomalies later.
  • The final loop prints every cleaning action so you can see exactly what the function did to the data.
  • Save engine/data_loader.py.
  • Test that the cleaning function works end-to-end on your real dataset by running:
python -c "from engine.data_loader import load_and_clean; df = load_and_clean(); print(len(df), 'records loaded')"

You should see several [CLEAN] log lines showing each transformation, followed by a record count. The exact count depends on how many rows had both amount and vendor missing.

Seeing a FileNotFoundError?

  • Make sure data/mplads_vendor_data.csv exists inside your mplads_monitor/ folder.
  • Confirm you are running the command from mplads_monitor/, not from engine/. The file path data/mplads_vendor_data.csv is relative to your current working directory.
  • Help me fix the file path.
Add data quality metrics

The cleaning function transforms the raw data, but you also need a way to measure the quality of what comes out. A quality report gives you dimension counts and data integrity stats at a glance.

  • In engine/data_loader.py, add the get_quality_report() function below the load_and_clean() function, at the same indentation level (not inside load_and_clean):
def get_quality_report(df):
    """Generate a data quality report dictionary."""
    report = {
        "total_records": len(df),
        "missing_values_per_column": df.isna().sum().to_dict(),
        "duplicate_count": df.duplicated(
            subset=["vendor_name", "expenditure_amount", "expenditure_date", "work"], keep=False
        ).sum(),
        "distinct_states": df["state"].nunique(),
        "distinct_constituencies": df["loksabha_constituency"].nunique(),
        "distinct_mps": df["loksabha_MP_name"].nunique(),
        "distinct_vendors": df["vendor_name"].nunique(),
    }
    return report

What does this code do?

  • df.duplicated() checks for rows where vendor_name, expenditure_amount, expenditure_date, and work are all identical. These could indicate duplicate payment records in the original data.
  • nunique() counts distinct values for key dimensions. This tells you how many states, constituencies, MPs, and vendors the dataset covers.
  • Save engine/data_loader.py. You will test this function alongside the main runner in the next chunk.
Run the full pipeline and verify the output

Both functions are ready. Adding a __main__ block makes the script runnable as a standalone command, calling both functions and printing everything to your terminal.

  • In engine/data_loader.py, add the following __main__ block at the very bottom of the file, outside all functions:
if __name__ == "__main__":
    print("Loading and cleaning MPLADS data...")
    df = load_and_clean()
    report = get_quality_report(df)
    print(f"\n  Data Quality Report:")
    print(f"  Total records: {report['total_records']}")
    print(f"  Duplicates: {report['duplicate_count']}")
    print(f"  States: {report['distinct_states']}, Constituencies: {report['distinct_constituencies']}")
    print(f"  MPs: {report['distinct_mps']}, Vendors: {report['distinct_vendors']}")
    print(f"\n  Sample rows:")
    print(df.head())

What does this code do?

  • if __name__ == "__main__" ensures this code only runs when you execute the file directly. When another module imports data_loader, this block is skipped.
  • It calls load_and_clean() first, then passes the cleaned DataFrame to get_quality_report() to produce the stats.
  • df.head() prints the first 5 rows so you can visually confirm the cleaning worked on real data.
  • Save engine/data_loader.py.

Before you run this, what do you expect to see? Think about how many records survived cleaning, how many states and MPs the dataset covers, and what the cleaned amount values should look like compared to the raw comma-separated strings.

  • Run the full data pipeline by running:
python engine/data_loader.py

You should see [CLEAN] log lines showing each transformation, followed by a Data Quality Report with record counts, duplicate count, and dimension stats. The last section shows 5 sample rows with proper numeric amounts and parsed dates.

Not seeing the quality report?

  • If you see no output at all, make sure the if __name__ == "__main__" block is at the bottom of the file and not indented inside a function.
  • If amounts show NaN, some values in the CSV could not be converted to numbers. The [CLEAN] log shows how many failed.
  • Help me debug this.

✔️ Awesome, I've got everything!

Great. Double-check you have saved engine/data_loader.py and your terminal shows the full quality report output.

ⓧ I'd like to double check the full code

Your engine/data_loader.py file is longer than 30 lines, so scroll through it and confirm each section is present, in order:

  • Imports at the top: pandas, numpy, pathlib.
  • The load_and_clean() function with: CSV loading, whitespace stripping, amount string-to-numeric conversion, date parsing, dropping rows where both amount and vendor are missing, filling missing categoricals from cat_fills, adding record_id, and printing the cleaning log.
  • The get_quality_report() function returning a dictionary with total_records, missing_values_per_column, duplicate_count, and four distinct-value counts.
  • The if __name__ == "__main__" block that calls both functions and prints the report plus df.head().

Your data pipeline is built and verified on real MPLADS data. Next, you will use this clean data to build the rule-based compliance engine that flags suspicious spending patterns.

Build the Rule-Based Compliance Engine

Your data pipeline now loads and cleans 96,541 MPLADS payment records into a structured DataFrame. But clean data alone does not tell you which of those records are suspicious.

The MPLADS Guidelines 2023 define specific compliance rules that can be checked programmatically. Rule-based checks catch known violations with zero false positives, giving you a credible baseline before adding machine learning in the next step.

In this step, get ready to:
  • Create a RuleEngine class with three compliance checks: vendor concentration, amount outliers, and Benford's Law violations.
  • Generate human-readable alert explanations for every flag, making the system fully auditable.
  • Run the engine on real data and verify flagged records with severity breakdowns.
Create the rule engine scaffold and vendor concentration check

The first rule targets vendor concentration. If a single vendor receives more than 40% of an MP's total expenditure, that pattern is a red flag for potential favoritism or collusion.

  • Create a new file called engine/rules.py and add the following imports and class scaffold:
import pandas as pd
import numpy as np
from collections import Counter
from engine.data_loader import load_and_clean


BENFORD_EXPECTED = {d: np.log10(1 + 1 / d) for d in range(1, 10)}


class RuleEngine:
    def __init__(self, df):
        self.df = df
        self.alerts = []

    def run_all(self):
        self.alerts = []
        self._check_vendor_concentration()
        self._check_amount_outliers()
        self._check_benford()
        return self.alerts

What does this code do?

BENFORD_EXPECTED precomputes the expected frequency of each leading digit (1 through 9) according to Benford's Law. You will use this lookup table in the third rule.

The RuleEngine class takes a cleaned DataFrame and stores an empty alerts list. run_all() calls all three rule methods in sequence and returns the collected alerts.

  • Add the vendor concentration check method below run_all, still inside the RuleEngine class:
    def _check_vendor_concentration(self):
        mp_groups = self.df.groupby("loksabha_MP_name")
        for mp_name, group in mp_groups:
            total_exp = group["expenditure_amount"].sum()
            if total_exp == 0:
                continue
            vendor_totals = group.groupby("vendor_name")["expenditure_amount"].sum()
            for vendor, vendor_total in vendor_totals.items():
                share = vendor_total / total_exp
                if share > 0.40:
                    flagged_records = group[group["vendor_name"] == vendor]["record_id"].tolist()
                    self.alerts.append({
                        "record_ids": flagged_records,
                        "mp_name": mp_name,
                        "rule_name": "VENDOR_CONCENTRATION",
                        "severity": "HIGH" if share > 0.70 else "MEDIUM",
                        "explanation": (
                            f"Vendor '{vendor}' received {share:.0%} of MP {mp_name}'s "
                            f"total expenditure (Rs. {vendor_total:,.0f} of Rs. {total_exp:,.0f}). "
                            f"Expected: below 40%."
                        ),
                        "data_evidence": {
                            "vendor": vendor,
                            "vendor_share": round(share, 4),
                            "vendor_total": float(vendor_total),
                            "mp_total": float(total_exp),
                        },
                    })

What does this code do?

For each MP, the method sums up expenditure per vendor and computes each vendor's share. Any vendor receiving more than 40% of the MP's total spend triggers an alert.

Every alert carries a human-readable explanation string that names the vendor, the percentage, and the actual rupee amounts. This is what makes every flag auditable and traceable to the underlying data.

  • Save engine/rules.py.
  • Verify the file has no syntax errors by running this command:
python -c "from engine.rules import RuleEngine; print('Import OK')"

You should see Import OK printed in your terminal.

Seeing an ImportError?

  • Make sure you are running the command from the mplads_monitor project folder, not from inside the engine/ subdirectory. Python needs to find the engine package from the project root.
  • Check that data_loader.py exists in the engine/ folder. The import from engine.data_loader import load_and_clean depends on it.
  • If you see a SyntaxError, check the indentation of _check_vendor_concentration. It must be indented one level inside the RuleEngine class, matching run_all.

Still stuck? Help me debug my engine/rules.py import error.

Add the statistical outlier and Benford's Law checks

The second rule catches payments that are statistical outliers within their work category. A payment more than 3 standard deviations above the category mean is unusual enough to warrant investigation.

  • Add the amount outlier method below _check_vendor_concentration, still inside the RuleEngine class:
    def _check_amount_outliers(self):
        for work_type, group in self.df.groupby("work"):
            amounts = group["expenditure_amount"].dropna()
            if len(amounts) < 10:
                continue
            mean_amt = amounts.mean()
            std_amt = amounts.std()
            if std_amt == 0:
                continue
            threshold = mean_amt + 3 * std_amt
            outliers = group[group["expenditure_amount"] > threshold]
            for _, row in outliers.iterrows():
                z_score = (row["expenditure_amount"] - mean_amt) / std_amt
                self.alerts.append({
                    "record_ids": [row["record_id"]],
                    "mp_name": row["loksabha_MP_name"],
                    "rule_name": "AMOUNT_OUTLIER",
                    "severity": "HIGH" if z_score > 5 else "MEDIUM",
                    "explanation": (
                        f"Payment of Rs. {row['expenditure_amount']:,.0f} for '{work_type}' "
                        f"is {z_score:.1f} standard deviations above the category mean "
                        f"(Rs. {mean_amt:,.0f}). Threshold: Rs. {threshold:,.0f}."
                    ),
                    "data_evidence": {
                        "amount": float(row["expenditure_amount"]),
                        "work_type": work_type,
                        "z_score": round(z_score, 2),
                        "category_mean": round(mean_amt, 2),
                    },
                })

What does this code do?

The method groups all records by work type, then computes the mean and standard deviation for each category. Any payment exceeding 3 standard deviations above the mean is flagged.

Categories with fewer than 10 records are skipped because the statistical measures would be unreliable. Categories where every payment is the same amount (standard deviation of zero) are also skipped.

The third rule applies Benford's Law, a forensic accounting technique. In naturally occurring financial data, the leading digit "1" appears about 30% of the time, "2" about 17%, and so on in a logarithmic distribution. Fabricated or rounded figures break this pattern.

  • Add the first part of the Benford's Law check below _check_amount_outliers. This extracts leading digits and computes the chi-squared test statistic:
    def _check_benford(self):
        mp_groups = self.df.groupby("loksabha_MP_name")
        for mp_name, group in mp_groups:
            amounts = group["expenditure_amount"].dropna()
            amounts = amounts[amounts > 0]
            if len(amounts) < 50:
                continue
            leading_digits = amounts.apply(lambda x: int(str(int(x))[0]))
            observed_counts = Counter(leading_digits)
            total = sum(observed_counts.values())

            chi_sq = 0
            for d in range(1, 10):
                observed = observed_counts.get(d, 0)
                expected = BENFORD_EXPECTED[d] * total
                if expected > 0:
                    chi_sq += (observed - expected) ** 2 / expected
  • Save engine/rules.py and verify the file still imports cleanly by running:
python -c "from engine.rules import RuleEngine; print('3 checks defined')"

You should see 3 checks defined printed. The method is syntactically valid at this point. It computes the chi-squared statistic but does not generate alerts yet.

  • Still inside the _check_benford method, directly below the code you just added (inside the for mp_name, group loop), add the alert generation logic:
            # Chi-squared critical value for 8 df at p=0.05 is 15.507
            if chi_sq > 15.507:
                flagged_records = group["record_id"].tolist()[:5]
                self.alerts.append({
                    "record_ids": flagged_records,
                    "mp_name": mp_name,
                    "rule_name": "BENFORD_VIOLATION",
                    "severity": "MEDIUM" if chi_sq < 30 else "HIGH",
                    "explanation": (
                        f"MP {mp_name}'s expenditure amounts show a leading-digit distribution "
                        f"that deviates from Benford's Law (chi-squared={chi_sq:.1f}, "
                        f"threshold=15.5). This may indicate fabricated or rounded figures."
                    ),
                    "data_evidence": {
                        "chi_squared": round(chi_sq, 2),
                        "sample_size": len(amounts),
                        "observed_distribution": dict(observed_counts),
                    },
                })

What is the chi-squared threshold?

The chi-squared critical value for 8 degrees of freedom at p=0.05 is 15.507. The test has 8 degrees of freedom because there are 9 possible leading digits (1 through 9) minus 1.

If an MP's leading-digit distribution deviates beyond this threshold, the explanation string calls it out with the exact chi-squared value. MPs with fewer than 50 payments are skipped because the sample is too small for a reliable test.

Why are human-readable explanations important?

Every alert carries an explanation string that names the specific MP, vendor, amounts, and statistical values. An auditor reviewing a flag can read the explanation and immediately understand what triggered it without digging through raw data.

This is what separates a useful detection system from a black box that just says "anomaly detected."

Run the rule engine and review the results
  • Add the get_summary method below _check_benford (still inside the class), and then add the __main__ block below the class definition:
    def get_summary(self):
        severity_counts = Counter(a["severity"] for a in self.alerts)
        rule_counts = Counter(a["rule_name"] for a in self.alerts)
        return {"by_severity": dict(severity_counts), "by_rule": dict(rule_counts), "total": len(self.alerts)}


if __name__ == "__main__":
    print("Loading data...")
    df = load_and_clean()
    print("Running rule engine...")
    engine = RuleEngine(df)
    alerts = engine.run_all()
    summary = engine.get_summary()
    print(f"\n  Rule Engine Summary:")
    print(f"  Total alerts: {summary['total']}")
    print(f"  By severity: {summary['by_severity']}")
    print(f"  By rule: {summary['by_rule']}")
    if alerts:
        print(f"\n  Sample alert:")
        print(f"    {alerts[0]['explanation']}")

What does this code do?

get_summary() counts alerts by severity (HIGH/MEDIUM) and by rule name, returning a quick overview dictionary.

The __main__ block loads the cleaned data, runs all three rules, prints the total alert count with breakdowns, and shows one sample alert explanation so you can verify the output format.

  • Save engine/rules.py.

Before you run the engine, what do you expect? Will most of the 96,541 records be flagged, or just a small fraction?

  • Run the rule engine on the real MPLADS data:
python engine/rules.py

You should see the data loading and cleaning output from data_loader.py, followed by a Rule Engine Summary showing total alerts, a breakdown by severity (HIGH and MEDIUM counts), a breakdown by rule (VENDOR_CONCENTRATION, AMOUNT_OUTLIER, BENFORD_VIOLATION counts), and a sample alert with a full human-readable explanation string.

Seeing an AttributeError or no output?

  • If you see AttributeError: 'RuleEngine' object has no attribute '_check_amount_outliers', one of the three rule methods is missing or misspelled. Check that all three method names match exactly: _check_vendor_concentration, _check_amount_outliers, _check_benford.
  • If the script runs but shows 0 alerts, check that data/mplads_vendor_data.csv is the 18th Lok Sabha vendor-wise dataset with 96,541 records, not a different file.
  • If you see an IndentationError, make sure every method inside RuleEngine is indented with exactly 4 spaces (one level inside the class).

Still stuck? Help me debug my rule engine output.

✔️ Awesome, I've got everything!

Great. Double check you have saved engine/rules.py and that the terminal output shows alerts across all three rule types.

ⓧ I'd like to double check the full code

The full engine/rules.py file is built from these chunks, in order. Scroll up and confirm each one is present in your file:

  • Imports (pandas, numpy, Counter, load_and_clean) and the BENFORD_EXPECTED constant.
  • RuleEngine class with __init__ and run_all methods.
  • _check_vendor_concentration method (flags vendors above 40% share).
  • _check_amount_outliers method (flags payments above 3 standard deviations).
  • _check_benford method (leading-digit extraction, chi-squared computation, and alert generation).
  • get_summary method and the if __name__ == "__main__" block.

Your rule engine now flags known compliance violations with clear, auditable explanations for every alert. Next, you will add Isolation Forest ML detection to catch the anomalous patterns that no single rule can find.

Add Isolation Forest ML Detection and Risk Scoring

Your rule engine catches known compliance violations like vendor concentration and amount outliers. But an MP could have subtly unusual spending that does not trigger any single rule while still looking anomalous when you examine multiple features together.

Isolation Forest solves this by finding records that are statistically "easy to isolate" from the normal population. In this step, you'll engineer spending features per MP, train an Isolation Forest model, and combine rule-based alerts with ML scores into a hybrid risk scoring system.

In this step, get ready to:
  • Engineer per-MP spending features from the raw payment data.
  • Train an Isolation Forest model to detect statistical anomalies.
  • Combine rule-based alerts and ML scores into composite risk scores with full explainability.
Create the feature engineering module

Before the ML model can analyze anything, you need to transform 96,000+ individual payment records into meaningful per-MP feature vectors. Each MP gets a numerical profile capturing their spending behaviour across multiple dimensions.

  • Create a new file called engine/detector.py and add these imports:
import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from engine.data_loader import load_and_clean
from engine.rules import RuleEngine

What are these imports for?

IsolationForest is the unsupervised ML algorithm that detects anomalies. StandardScaler normalizes your features so no single dimension dominates the model. The last two lines import your own modules from the previous steps.

  • Add the FeatureEngineer class below the imports. This class transforms raw payment records into per-MP feature vectors:
class FeatureEngineer:
    def __init__(self, df):
        self.df = df

    def compute_mp_features(self):
        mp_features = []
        for mp_name, group in self.df.groupby("loksabha_MP_name"):
            amounts = group["expenditure_amount"].dropna()
            vendor_shares = group.groupby("vendor_name")["expenditure_amount"].sum()
            total_exp = vendor_shares.sum()
            hhi = ((vendor_shares / total_exp) ** 2).sum() if total_exp > 0 else 0
            dates = group["expenditure_date"].dropna()
            if len(dates) > 1:
                date_range_days = (dates.max() - dates.min()).days
                payments_per_month = len(dates) / max(date_range_days / 30, 1)
            else:
                payments_per_month = 0
            mp_features.append({
                "loksabha_MP_name": mp_name,
                "total_expenditure": float(total_exp),
                "vendor_count": group["vendor_name"].nunique(),
                "vendor_concentration_hhi": round(hhi, 4),
                "mean_payment": float(amounts.mean()) if len(amounts) > 0 else 0,
                "std_payment": float(amounts.std()) if len(amounts) > 1 else 0,
                "unique_work_types": group["work"].nunique(),
                "payment_frequency": round(payments_per_month, 2),
                "total_payments": len(group),
            })

        return pd.DataFrame(mp_features)

What does this code do?

compute_mp_features() groups the raw data by MP name and computes eight numerical features for each MP.

  • vendor_concentration_hhi is the Herfindahl-Hirschman Index. It measures how concentrated an MP's spending is across vendors. A score near 1.0 means almost all money goes to one vendor.
  • payment_frequency captures how many payments per month the MP processes. Unusual bursts or gaps in payment timing can signal anomalies.
  • The remaining features (total_expenditure, vendor_count, mean_payment, std_payment) give Isolation Forest a multi-dimensional view of each MP's spending profile.
  • Save engine/detector.py.
  • Verify the feature engineering works by running this command:
python -c "from engine.detector import FeatureEngineer; from engine.data_loader import load_and_clean; df = load_and_clean(); mf = FeatureEngineer(df).compute_mp_features(); print(f'{len(mf)} MPs, {len(mf.columns)} features')"

You should see the cleaning log from your data loader, followed by a line showing the number of MPs and 9 feature columns.

Seeing an ImportError?

  • Make sure you are running the command from the mplads_monitor project root folder, not from inside the engine/ directory.
  • Confirm your virtual environment is activated. The scikit-learn import will fail if the packages are not installed in the active environment.

Still stuck? Ask for help.

Add Isolation Forest ML detection

The features are ready. Now you need a model that can examine all eight dimensions simultaneously and find MPs whose spending profiles look statistically unusual compared to the rest.

Why Isolation Forest?

The MPLADS dataset has no labeled fraud cases. You cannot train a supervised classifier because there is no ground truth to learn from. Isolation Forest is an unsupervised algorithm designed for exactly this situation.

It works by building random decision trees and measuring how many splits it takes to isolate each data point. Anomalies are "easy to isolate" because they sit far from the crowd. Normal points take many splits to separate.

  • In engine/detector.py, add the MLDetector class below the FeatureEngineer class:
class MLDetector:
    FEATURE_COLS = [
        "total_expenditure", "vendor_count", "vendor_concentration_hhi",
        "mean_payment", "std_payment", "unique_work_types",
        "payment_frequency", "total_payments",
    ]

    def __init__(self):
        self.model = IsolationForest(contamination=0.05, n_estimators=200, random_state=42)
        self.scaler = StandardScaler()

    def fit_predict(self, mp_features_df):
        X = mp_features_df[self.FEATURE_COLS].fillna(0)
        X_scaled = self.scaler.fit_transform(X)
        labels = self.model.fit_predict(X_scaled)
        scores = self.model.decision_function(X_scaled)

        # Map decision_function scores to 0-100 risk scale (lower decision = higher risk)
        min_score, max_score = scores.min(), scores.max()
        if max_score != min_score:
            risk_scores = 100 * (1 - (scores - min_score) / (max_score - min_score))
        else:
            risk_scores = np.zeros(len(scores))

        mp_features_df = mp_features_df.copy()
        mp_features_df["ml_anomaly_label"] = labels
        mp_features_df["ml_anomaly_score"] = np.round(risk_scores, 2)
        return mp_features_df

What does this code do?

  • contamination=0.05 tells the model to expect roughly 5% of MPs to be anomalous. This is a conservative starting point for unsupervised detection.
  • n_estimators=200 builds 200 random isolation trees. More trees improve stability of the anomaly scores.
  • StandardScaler normalizes features before training. Without scaling, total_expenditure (millions) would dominate vendor_count (single digits).
  • decision_function() returns a continuous anomaly score which gets mapped to a 0-100 risk scale. Lower decision function values mean higher risk.
  • Save engine/detector.py.
  • Verify the ML detector imports correctly by running:
python -c "from engine.detector import MLDetector; print('MLDetector ready')"

You should see MLDetector ready printed with no errors. This confirms the file has valid syntax and the IsolationForest import resolves correctly.

Seeing a NameError or IndentationError?

  • Check that the MLDetector class is at the same indentation level as FeatureEngineer (top level, not nested inside it).
  • Make sure the FEATURE_COLS list closes with ] and a comma after the last entry.

Need more help debugging?

Combine rules and ML into hybrid risk scores

You now have two independent detection signals: rule-based alerts from Step 3 and ML anomaly scores from the Isolation Forest. The hybrid approach combines them into a single composite risk score per MP, which is what separates a basic compliance checker from an intelligent detection system.

  • Add the HybridScorer class below MLDetector. Start with the class definition, severity weights, and the explanation helper method:
class HybridScorer:
    SEVERITY_WEIGHTS = {"HIGH": 1.0, "MEDIUM": 0.5, "LOW": 0.25}

    def __init__(self, mp_scores_df, rule_alerts):
        self.mp_scores = mp_scores_df
        self.rule_alerts = rule_alerts

    def _build_explanation(self, row):
        parts = []
        if row["ml_anomaly_score"] > 70:
            parts.append(f"ML anomaly score: {row['ml_anomaly_score']:.0f}/100 (statistical outlier)")
        for flag in row.get("rule_flags", []):
            parts.append(f"[{flag['severity']}] {flag['explanation']}")
        if not parts:
            parts.append("No significant anomalies detected.")
        return " | ".join(parts)

What does this code do?

SEVERITY_WEIGHTS maps each alert severity to a numerical weight. HIGH violations count as 1.0, MEDIUM as 0.5, and LOW as 0.25.

_build_explanation() generates a human-readable string for each MP's risk assessment. It concatenates the ML anomaly score (if high enough) with every rule flag that fired. This is what makes each alert auditable and traceable.

  • Now add the main scoring method. In the HybridScorer class, add compute_composite_scores() between __init__ and _build_explanation:
    def compute_composite_scores(self):
        # Compute rule severity score per MP
        mp_rule_scores = {}
        mp_rule_flags = {}
        for alert in self.rule_alerts:
            mp = alert["mp_name"]
            weight = self.SEVERITY_WEIGHTS.get(alert["severity"], 0)
            mp_rule_scores[mp] = max(mp_rule_scores.get(mp, 0), weight)
            if mp not in mp_rule_flags:
                mp_rule_flags[mp] = []
            mp_rule_flags[mp].append({
                "rule": alert["rule_name"],
                "severity": alert["severity"],
                "explanation": alert["explanation"],
            })
        results = self.mp_scores.copy()
        results["rule_severity_score"] = results["loksabha_MP_name"].map(
            lambda mp: mp_rule_scores.get(mp, 0)
        )
        results["rule_flags"] = results["loksabha_MP_name"].map(
            lambda mp: mp_rule_flags.get(mp, [])
        )
        # Normalize ML score to 0-1
        normalized_ml = results["ml_anomaly_score"] / 100
        # Composite: 40% ML + 60% rules
        results["composite_risk_score"] = np.round(
            0.4 * normalized_ml + 0.6 * results["rule_severity_score"], 4
        ) * 100
        results = results.sort_values("composite_risk_score", ascending=False)
        results["explanation"] = results.apply(
            lambda row: self._build_explanation(row), axis=1
        )
        return results

How does the composite score work?

The composite formula is 0.4 * normalized_ml_score + 0.6 * rule_severity_score. Rules carry 60% of the weight because they represent known, documented compliance violations. ML carries 40% because it catches statistical outliers that no single rule would flag.

An MP flagged by both the rule engine AND the ML model gets a much higher composite score than one flagged by either alone. This dual-signal approach reduces false positives while catching more genuine anomalies.

  • Save engine/detector.py.
  • Verify the HybridScorer imports correctly by running:
python -c "from engine.detector import HybridScorer; print('HybridScorer ready')"

You should see HybridScorer ready with no errors.

  • Now add the if __name__ == "__main__" block at the bottom of engine/detector.py to run the full hybrid pipeline:
if __name__ == "__main__":
    print("Loading data...")
    df = load_and_clean()

    print("Running rule engine...")
    engine = RuleEngine(df)
    alerts = engine.run_all()

    print("Engineering features...")
    fe = FeatureEngineer(df)
    mp_features = fe.compute_mp_features()

    print("Running Isolation Forest...")
    detector = MLDetector()
    mp_scores = detector.fit_predict(mp_features)

    print("Computing hybrid scores...")
    scorer = HybridScorer(mp_scores, alerts)
    results = scorer.compute_composite_scores()

    print(f"\n  Hybrid Detection Results:")
    print(f"  Total MPs analyzed: {len(results)}")
    ml_anomalies = (results["ml_anomaly_label"] == -1).sum()
    print(f"  ML anomalies: {ml_anomalies}")
    print(f"  Rule alerts: {len(alerts)}")
    print(f"\n  Top 10 riskiest MPs:")
    top10 = results.head(10)[["loksabha_MP_name", "composite_risk_score", "ml_anomaly_score", "rule_severity_score"]]
    print(top10.to_string(index=False))

What does this code do?

This block runs the entire hybrid detection pipeline end-to-end. It loads the cleaned data, runs the rule engine from Step 3, engineers per-MP features, trains Isolation Forest, and combines everything into composite risk scores.

The final output shows the top 10 riskiest MPs with three scores side by side: the composite score, the ML anomaly score, and the rule severity score. This lets you see exactly which signal contributed most to each MP's risk ranking.

  • Save engine/detector.py.

Before you run the pipeline, think about this: you have about 800 MPs in the dataset and the model is configured with contamination=0.05 (5%). Roughly how many ML anomalies do you expect to see?

  • Run the full hybrid pipeline by running:
python engine/detector.py

You should see the cleaning log, progress messages for each pipeline stage, then the Hybrid Detection Results summary. The output shows the total MPs analyzed, the number of ML anomalies, the total rule alerts, and a table of the top 10 riskiest MPs with their composite, ML, and rule scores.

Seeing a KeyError or empty results?

  • A KeyError on a column name usually means the CSV column headers do not match. Open data/mplads_vendor_data.csv and verify the column names match what data_loader.py expects (e.g. expenditure_amount, loksabha_MP_name).
  • If compute_composite_scores is not found, make sure you placed it inside the HybridScorer class, indented one level under the class definition.
  • If the script runs but shows 0 ML anomalies, check that FEATURE_COLS in MLDetector matches the column names produced by compute_mp_features().

Still stuck? Ask for help.

✔️ Awesome, I've got everything!

Great. Make sure you have saved engine/detector.py.

ⓧ I'd like to double check the full code

engine/detector.py is a large file. Scroll through it and confirm these sections are present in this order:

  • Imports at the top (pandas, numpy, IsolationForest, StandardScaler, load_and_clean, RuleEngine).
  • FeatureEngineer class with compute_mp_features() returning a DataFrame with 9 columns.
  • MLDetector class with FEATURE_COLS, IsolationForest(contamination=0.05, n_estimators=200, random_state=42), and fit_predict().
  • HybridScorer class with SEVERITY_WEIGHTS, compute_composite_scores(), and _build_explanation().
  • if __name__ == "__main__" block that runs all four classes and prints the top 10 riskiest MPs.

Your detection engine now produces composite risk scores that combine the best of both worlds: deterministic compliance checks and statistical anomaly detection. Next up, you'll build the Streamlit dashboard to visualize these results and make every flag explorable.

Build the Streamlit Monitoring Dashboard

Your hybrid detection engine now produces composite risk scores that combine rule-based alerts with ML anomaly scores. Running python engine/detector.py prints the top 10 riskiest MPs to your terminal.

Terminal output is useful for debugging, but it is invisible to SIH evaluators and real MPLADS stakeholders. They need an interactive interface where they can filter by state or MP, see risk visualizations, and drill into individual alerts to understand exactly why each flag was raised. Without a dashboard, you have a script. With one, you have a platform.

In this step, get ready to:
  • Create the dashboard layout with a cached data pipeline, sidebar filters, and three role-based tabs.
  • Build the Ministry Overview tab with KPI cards, a state-wise risk bar chart, and a riskiest-MPs leaderboard.
  • Build the Alert Investigation tab with filterable alerts, an investigation drill-down panel, and full data auditability.
Create the dashboard layout and data pipeline

The dashboard needs to run the entire detection pipeline (data loading, rules, ML, hybrid scoring) once and reuse the results across every tab and filter. Streamlit re-runs your script on every interaction by default, so you will cache the pipeline to keep the app fast.

  • Create a new file called app.py inside the dashboard/ folder.
  • Add the imports, page configuration, and cached pipeline function by pasting this code:
import streamlit as st
import pandas as pd
import plotly.express as px
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent.parent))  # add project root to path

from engine.data_loader import load_and_clean, get_quality_report
from engine.rules import RuleEngine
from engine.detector import FeatureEngineer, MLDetector, HybridScorer

st.set_page_config(page_title="MPLADS Monitor", layout="wide")
st.title("MPLADS Anomaly Detection Dashboard")
st.caption("SIH26102 - AI-Powered Fraud and Inefficiency Detection for MPLAD Scheme")

# Cache the full detection pipeline so it only runs once
@st.cache_data
def run_pipeline():
    df = load_and_clean()
    quality = get_quality_report(df)
    engine = RuleEngine(df)
    alerts = engine.run_all()
    summary = engine.get_summary()
    fe = FeatureEngineer(df)
    mp_features = fe.compute_mp_features()
    detector = MLDetector()
    mp_scores = detector.fit_predict(mp_features)
    scorer = HybridScorer(mp_scores, alerts)
    results = scorer.compute_composite_scores()
    return df, quality, alerts, summary, results

What does this code do?

sys.path.insert(0, ...) adds the project root to Python's import path. Without this, importing from engine/ would fail because Streamlit runs from the dashboard/ directory.

@st.cache_data tells Streamlit to run run_pipeline() once, then reuse the cached results on every subsequent interaction. This prevents re-processing 96,541 records every time someone clicks a filter.

The pipeline function is defined, but the dashboard has no visible interface yet. Next, you will add the sidebar filters and tab structure that give the dashboard its interactive skeleton.

  • Add the sidebar filters, state filtering logic, and tab creation below the run_pipeline function:
# Run the pipeline and unpack results
df, quality, alerts, rule_summary, results = run_pipeline()

# Sidebar filters for role-based views
st.sidebar.header("Filters")
role = st.sidebar.selectbox("View As", ["Ministry (DIID)", "State Nodal Agency", "District Authority"])
states = ["All"] + sorted(df["state"].unique().tolist())
selected_state = st.sidebar.selectbox("State", states)
severity_filter = st.sidebar.multiselect("Alert Severity", ["HIGH", "MEDIUM", "LOW"], default=["HIGH", "MEDIUM"])

# Apply state filter to narrow down results
if selected_state != "All":
    filtered_df = df[df["state"] == selected_state]
    filtered_results = results[
        results["loksabha_MP_name"].isin(filtered_df["loksabha_MP_name"].unique())
    ]
else:
    filtered_df = df
    filtered_results = results

# Create the three role-based tabs
tab1, tab2, tab3 = st.tabs(["Ministry Overview", "State/District View", "Alert Investigation"])

What does this code do?

The sidebar gives stakeholders three filter controls. The role selectbox switches between Ministry, State Nodal Agency, and District Authority perspectives. The state selectbox narrows all data to a single state. The severity multiselect controls which alert levels appear.

st.tabs() creates the three-tab layout. Each tab will hold a different granularity of view: national overview, state and constituency drill-down, and individual alert investigation.

  • Save dashboard/app.py.
  • Before you run the dashboard, what do you expect to see? Three populated tabs, or something simpler?
  • Launch the dashboard by running this command from the mplads_monitor/ project root:
streamlit run dashboard/app.py

Your browser opens at http://localhost:8501. You should see the title "MPLADS Anomaly Detection Dashboard", a sidebar with three filter controls (View As, State, Alert Severity), and three empty tabs. The tabs have no content yet because you have not added any code inside them.

Why is Streamlit still running?

Streamlit keeps a development server running and watches for file changes. When you edit app.py and save, Streamlit detects the change and offers to rerun. You can click **Always rerun** so updates appear automatically.

Dashboard not opening?

  • Make sure you are running the command from the mplads_monitor/ project root, not from inside dashboard/. The pipeline imports depend on relative paths from the project root.
  • Check that your virtual environment is activated. If streamlit is not found, activate your environment and try again.
  • If you see an import error mentioning engine, confirm that the sys.path.insert line is present near the top of app.py.

Help me debug why my Streamlit dashboard won't start

Build the Ministry Overview tab

The Ministry Overview tab gives national-level stakeholders an instant snapshot: how many records were analyzed, how many alerts fired, and which states and MPs carry the most risk.

  • In dashboard/app.py, add the KPI cards and state-wise bar chart inside the tab1 block, below the line tab1, tab2, tab3 = st.tabs(...):
with tab1:
    # KPI metric cards across the top
    col1, col2, col3, col4 = st.columns(4)
    col1.metric("Total Records", f"{quality['total_records']:,}")
    col2.metric("Total Alerts", f"{rule_summary['total']}")
    high_risk = len(filtered_results[filtered_results["composite_risk_score"] > 50])
    col3.metric("High-Risk MPs", high_risk)
    col4.metric("States Covered", quality["distinct_states"])

    # State-wise risk bar chart using Plotly
    st.subheader("State-wise Anomaly Density")
    state_risk = filtered_results.merge(
        df[["loksabha_MP_name", "state"]].drop_duplicates(),
        on="loksabha_MP_name",
        how="left",
    )
    state_agg = state_risk.groupby("state").agg(
        avg_risk=("composite_risk_score", "mean"),
        mp_count=("loksabha_MP_name", "nunique"),
    ).reset_index()
    fig = px.bar(
        state_agg.sort_values("avg_risk", ascending=False).head(20),
        x="state", y="avg_risk", color="avg_risk",
        color_continuous_scale="RdYlGn_r",
        title="Average Risk Score by State (Top 20)",
        labels={"avg_risk": "Avg Risk Score", "state": "State"},
    )
    st.plotly_chart(fig, use_container_width=True)

What does this code do?

st.columns(4) creates four side-by-side metric cards. Each one calls st.metric() to display a KPI: total records analyzed, total rule-based alerts, MPs with a composite risk score above 50, and the number of states covered.

The bar chart merges each MP's risk score with their state, aggregates by state to compute an average risk score, sorts descending, and renders the top 20 states. The RdYlGn_r color scale maps higher risk to red and lower risk to green.

  • Save dashboard/app.py and switch to your browser.
  • Click **Always rerun** if Streamlit prompts you (or click **Rerun** manually).
  • You should see four KPI metric cards at the top of the Ministry Overview tab and a color-coded bar chart of the top 20 states by average risk score below them.

The Ministry Overview tab still needs a riskiest-MPs leaderboard, and the State/District View tab is empty. You will add both in the next chunk.

  • Add the Top 10 MPs table below the bar chart (still inside with tab1:), then add the with tab2: block right after. Paste this below the st.plotly_chart line:
    # Top 10 riskiest MPs leaderboard
    st.subheader("Top 10 Riskiest MPs")
    top_mps = filtered_results.head(10)[
        ["loksabha_MP_name", "composite_risk_score", "ml_anomaly_score", "total_expenditure", "vendor_count"]
    ]
    st.dataframe(top_mps, use_container_width=True, hide_index=True)

with tab2:
    # Constituency-level risk table
    st.subheader("Constituency-Level Risk")
    constituency_data = filtered_results.merge(
        df[["loksabha_MP_name", "state", "loksabha_constituency"]].drop_duplicates(),
        on="loksabha_MP_name", how="left",
    )
    if selected_state != "All":
        constituency_data = constituency_data[constituency_data["state"] == selected_state]

    st.dataframe(
        constituency_data[
            ["loksabha_constituency", "loksabha_MP_name", "composite_risk_score",
             "vendor_count", "vendor_concentration_hhi", "total_expenditure"]
        ].sort_values("composite_risk_score", ascending=False),
        use_container_width=True, hide_index=True,
    )

    # Rule type and severity breakdown
    st.subheader("Rule Breakdown")
    st.write(f"By rule type: {rule_summary['by_rule']}")
    st.write(f"By severity: {rule_summary['by_severity']}")

What does this code do?

The first section (still inside with tab1:) renders a sortable table of the 10 highest-risk MPs with their composite score, ML anomaly score, total expenditure, and vendor count.

The with tab2: block adds the State/District View. It merges MP risk scores with their constituency and state, respects the sidebar state filter, and displays a constituency-level risk table. The Rule Breakdown section prints alert counts by rule type and severity.

  • Save dashboard/app.py and let Streamlit rerun.
  • In the Ministry Overview tab, scroll down past the bar chart. You should see a "Top 10 Riskiest MPs" table with five columns.
  • Click the **State/District View** tab. You should see a constituency-level risk table and a Rule Breakdown section showing alert counts by rule type and severity.
  • Try selecting a specific state from the sidebar **State** filter. Both tabs should update to show only data from that state.

Charts or tables not rendering?

  • Check the indentation. The Top 10 MPs code must be indented inside with tab1: (4 spaces). The with tab2: block must start at the same indentation level as with tab1: (no extra indentation).
  • If you see a KeyError mentioning a column name, verify your CSV has the expected column headers by checking the first row of data/mplads_vendor_data.csv.

Help me debug why my Streamlit tabs are not rendering correctly

Build the Alert Investigation tab

The Alert Investigation tab is the most critical view for auditors. It lets them pick any individual alert, see the full explanation and data evidence, and trace the flag back to the original CSV records.

  • Add the alert filtering, selectbox, and alert table inside the tab3 block. Paste this below the with tab2: block you just added:
with tab3:
    st.subheader("Alert Investigation Panel")
    # Filter alerts by severity and state selections
    filtered_alerts = [a for a in alerts if a["severity"] in severity_filter]
    if selected_state != "All":
        state_mps = df[df["state"] == selected_state]["loksabha_MP_name"].unique()
        filtered_alerts = [a for a in filtered_alerts if a["mp_name"] in state_mps]

    if filtered_alerts:
        alert_df = pd.DataFrame([
            {
                "MP": a["mp_name"],
                "Rule": a["rule_name"],
                "Severity": a["severity"],
                "Explanation": a["explanation"][:120] + "...",
            }
            for a in filtered_alerts
        ])
        # Let the user pick a specific alert to investigate
        selected_idx = st.selectbox(
            "Select an alert to investigate",
            range(len(filtered_alerts)),
            format_func=lambda i: f"{filtered_alerts[i]['mp_name']} - {filtered_alerts[i]['rule_name']} ({filtered_alerts[i]['severity']})",
        )
        st.dataframe(alert_df, use_container_width=True, hide_index=True)

What does this code do?

This code filters the full alerts list down to only the severities and state the user selected in the sidebar. It then builds a summary DataFrame showing each alert's MP, rule, severity, and a truncated explanation.

The st.selectbox dropdown uses format_func to show each alert as a human-readable label (e.g., "MP Name - VENDOR_CONCENTRATION (HIGH)") while internally tracking the numeric index.

The alert list appears, but selecting an alert does nothing yet. The investigation drill-down panel comes next.

  • Add the investigation drill-down panel and the sidebar data quality summary. Paste this directly below the st.dataframe(alert_df, ...) line you just added:
        # Show detailed investigation panel for selected alert
        if selected_idx is not None:
            alert = filtered_alerts[selected_idx]
            st.markdown("---")
            st.subheader(f"Investigation: {alert['mp_name']}")
            inv_col1, inv_col2 = st.columns(2)
            inv_col1.metric("Rule", alert["rule_name"])
            inv_col2.metric("Severity", alert["severity"])
            st.write("Full Explanation:")
            st.info(alert["explanation"])
            st.write("Data Evidence:")
            st.json(alert["data_evidence"])

            # Show the raw records for full auditability
            mp_data = df[df["loksabha_MP_name"] == alert["mp_name"]]
            st.write(f"All {len(mp_data)} records for this MP:")
            st.dataframe(mp_data.head(20), use_container_width=True, hide_index=True)
    else:
        st.info("No alerts match the current filters.")

# Sidebar data quality summary
st.sidebar.markdown("---")
st.sidebar.caption("Data Quality")
st.sidebar.write(f"Records: {quality['total_records']:,}")
st.sidebar.write(f"Duplicates: {quality['duplicate_count']}")
st.sidebar.write(f"Vendors: {quality['distinct_vendors']:,}")

What does the investigation panel show?

When a user selects an alert, the panel reveals four layers of detail. Two metric cards show the triggered rule and severity. st.info() displays the full human-readable explanation string your RuleEngine generated. st.json() renders the raw data evidence dictionary so auditors can inspect exact numbers.

The final table pulls all payment records for that MP from the original dataset. This is what makes the system auditable. Every flag can be traced back to the source data rows.

  • Save dashboard/app.py and let Streamlit rerun.
  • Click the **Alert Investigation** tab.
  • Before you click an alert, what do you expect the drill-down panel to show? A summary, or the full original data?
  • Select any alert from the dropdown. You should see a horizontal rule, an "Investigation" subheader with the MP's name, two metric cards (Rule and Severity), the full explanation in a blue info box, a JSON data evidence block, and a table of that MP's raw payment records.

Check the sidebar too. Below the filters, you should now see a "Data Quality" section showing the total record count, duplicate count, and vendor count.

Investigation panel not appearing when you select an alert?

  • Verify that the if selected_idx is not None: block is indented 8 spaces (inside both with tab3: and if filtered_alerts:).
  • The else: st.info("No alerts match...") block must be at the same level as if filtered_alerts: (4 spaces). If it is misaligned, the tab will show an error.

Help me fix my Alert Investigation tab

✔️ Awesome, I've got everything!

Great. Make sure you have saved dashboard/app.py.

ⓧ I'd like to double check the full code

The full dashboard/app.py file exceeds 30 lines, so scroll through and confirm these sections are present in order:

  • Imports (streamlit, pandas, plotly.express, sys, Path) plus the sys.path.insert line and engine imports.
  • st.set_page_config, st.title, and st.caption page configuration.
  • @st.cache_data decorated run_pipeline() function returning df, quality, alerts, summary, results.
  • Pipeline call unpacking into df, quality, alerts, rule_summary, results.
  • Sidebar filters: View As selectbox, State selectbox, Alert Severity multiselect.
  • State filter if/else block creating filtered_df and filtered_results.
  • st.tabs() creating tab1, tab2, tab3.
  • with tab1: block with KPI columns, bar chart, and Top 10 MPs table.
  • with tab2: block with constituency-level risk table and rule breakdown.
  • with tab3: block with alert filtering, selectbox, alert table, investigation drill-down, and else clause.
  • Sidebar data quality section at the bottom (Records, Duplicates, Vendors).

Secret mission

NLP-Powered Duplicate Work Detection

One of the most impactful fraud patterns in MPLADS is duplicate works. An identical project gets recommended across multiple constituencies with the same description and similar costs. In this mission, you will use TF-IDF cosine similarity to catch these copy-paste projects and surface them in a dedicated dashboard tab.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally with open-source tools, so there are no ongoing costs or cloud resources to worry about.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The mplads_monitor/ project folder on your Desktop, containing all Python files (data_loader.py, rules.py, detector.py, duplicate_detector.py, app.py), the Python virtual environment, and the downloaded MPLADS dataset.

Keep everything running

No action needed. Choose this if you are still actively building on the project or planning to extend it for your SIH submission.

  • Your mplads_monitor/ folder stays on your Desktop with all code, data, and the virtual environment intact.
  • The MPLADS dataset and all installed packages remain available for immediate use.
  • To restart the dashboard later, activate your virtual environment and run streamlit run dashboard/app.py from the project folder.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files and virtual environment stay on disk for later.

  • If the Streamlit dashboard is still running in your terminal, press Ctrl+C to stop it.
  • Deactivate the virtual environment by running:
deactivate
  • To resume later, navigate back to mplads_monitor/, activate the virtual environment, and run the dashboard again.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the downloaded dataset from your machine.

  • If the Streamlit dashboard is still running, press Ctrl+C to stop it.
  • Deactivate the virtual environment if it is still active:
deactivate
  • Delete the entire project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf mplads_monitor
  • On Windows, run:
cd $HOME\Desktop
Remove-Item -Recurse -Force mplads_monitor
  • Confirm the folder is gone by running ls (macOS/Linux) or dir (Windows) on your Desktop. You should no longer see mplads_monitor in the output.

Nice Work!

Nice Work!

You just built an AI-powered anomaly detection platform for the MPLADS scheme, running hybrid detection across 96,541 real government expenditure records. You now have an end-to-end prototype that ingests messy data, flags suspicious patterns with explainable alerts, and surfaces everything through an interactive dashboard ready for a compelling SIH demo.

You've learned how to:

  • Build a data ingestion pipeline that cleans real, messy government data (string-to-numeric conversion, date parsing, missing value handling) and produces auditable quality reports with dimension stats.
  • Create a hybrid anomaly detection engine combining rule-based compliance checks (vendor concentration, amount outliers, Benford's Law) with Isolation Forest ML to produce explainable composite risk scores for every MP.
  • Build an interactive Streamlit dashboard with role-based views, state-wise risk visualizations, and drill-down investigation panels showing exactly why each flag was raised and the underlying data row that triggered it.
  • Secret Mission: Add NLP-powered duplicate work detection using TF-IDF cosine similarity to catch copy-paste fraud across constituencies.

Ready to quiz yourself?