Audit Grant-Text Model Risk
Audit a grant-text classifier with NLP metrics, tests, and Azure ML execution.
Introduction
30 Second Summary
A convincing score can make a risky decision feel objective. In public funding, that confidence can hide the people or proposals a system consistently gets wrong.
In this project, you will build a command-line risk audit that evaluates predictions of historical SkatteFUNN approval outcomes from public application text. You will produce evidence showing why the model must remain an evaluation exercise instead of deciding real funding.
What You'll Build
Run one command and watch the audit turn public application text into an evidence pack that exposes weak predictions and supports a clear no-deployment recommendation.
By the end of this project, you'll have:
- Run a majority-class baseline beside an NLP classifier. Receive machine-readable JSON metrics for both approaches.
- Inspect class-specific failures through precision, recall, macro-F1, a confusion matrix, and text-length slices. Use the generated model card to explain why the classifier must not automate funding decisions.
- Run automated tests that check safe schema handling and end-to-end artifact generation. Submit the same workload to Azure Machine Learning and download the tracked audit outputs.
- Secret Mission: add a counterfactual audit that compares neutral and promotional versions of the same project text to measure changes in approval probability.
Are there any prerequisites?
You need a Linux terminal with internet access, Git, and a code editor. You also need permission to create or use an Azure subscription.
Before We Start
This educational audit evaluates associations between public SkatteFUNN application text and historical approval outcomes. Before any hands-on work, commit to never using its predictions to approve, reject, rank, prioritize, score, or otherwise automate funding decisions.
Create a Reproducible Environment
Your responsible-use boundary defines how this audit may be used. A credible audit also needs results that another machine can reproduce.
An unknown system Python runtime can change program behavior. Unpinned packages can change evaluation results between installations.
This step creates a reproducible environment with exact runtime versions. A local Git repository gives those files a version-controlled home.
In this step, get ready to:
- Prepare a local Git repository with uv available.
- Create an activated .venv with Python 3.12.16.
- Install the exact dependencies from requirements.txt.
Prepare the local repository
The uv tool manages the exact Python runtime for this project. Checking its version first reveals whether your machine already has the required release.
- Use your Linux application launcher to start a terminal window.
- Check the available uv version by running this command:
uv --versionWhat does this command do?
The command prints the uv release available in your current shell. Match your result to one of the scenarios below.
✔️ I see version 0.13.7
Your shell already has the required uv release. That is the first reproducibility check complete.
ⓧ I see an older version
The project uses uv 0.13.7 so each machine follows the same environment workflow.
- Install the pinned uv release by running these commands:
curl -LsSf https://astral.sh/uv/0.13.7/install.sh | sh
source "$HOME/.local/bin/env"
uv --versionWhat do these commands do?
- The installer downloads uv 0.13.7 from the official versioned installer.
- The environment script makes the installed command available in the current shell.
- The final check prints the uv release now in use.
You should see uv 0.13.7 in the final line.
Still seeing the older release?
Close the current terminal window. Start a fresh terminal window so your shell reloads its command path.
Help me resolve the uv version used by my Linux shell.
ⓧ Command not found
Your shell needs the pinned uv release before it can manage the project runtime.
- Install uv 0.13.7 by running these commands:
curl -LsSf https://astral.sh/uv/0.13.7/install.sh | sh
source "$HOME/.local/bin/env"
uv --versionWhat do these commands do?
- The installer places the pinned uv release on your machine.
- The environment script exposes uv to the current shell.
- The version check confirms that uv 0.13.7 is available.
You should see uv 0.13.7 in the final line.
Installer did not finish?
Confirm that your machine can reach the internet. Check that the installer command matches the command above.
Help me install the pinned uv release.
A fixed project location makes the repository easy to find during every later step. The project lives in a grant-risk-audit folder on your Desktop.
- Move to your Desktop by running this command:
cd ~/DesktopWhat does this command do?
This changes the terminal location to your Linux Desktop directory. The project folder created next appears there.
- Create the grant-risk-audit folder and initialize its Git repository by running these commands:
mkdir grant-risk-audit
cd grant-risk-audit
git initWhat do these commands do?
- The first command creates the grant-risk-audit directory.
- The second command moves the terminal into that directory.
- The final command creates the hidden .git repository data.
You should see confirmation that Git initialized an empty repository. Your audit now has a dedicated version-controlled home.
Repository initialization failed?
Confirm that your terminal can run Git. Check that the terminal is inside ~/Desktop/grant-risk-audit before retrying.
Help me initialize the local Git repository.
Ignore rules keep generated environments outside Git history. They also prevent cache files from cluttering future commits.
- Use your code editor's folder-opening control to select ~/Desktop/grant-risk-audit.
- Use your editor's new-file control to create .gitignore inside grant-risk-audit.
- Copy these ignore rules into .gitignore:
.venv/
__pycache__/
*.pyc
artifacts/
azure-download/What do these rules exclude?
- The environment rule keeps installed packages outside version control.
- The cache rules exclude generated Python bytecode.
- The artifact rules exclude local audit outputs plus downloaded cloud outputs.
- Save .gitignore.
- Confirm that .gitignore appears in your editor sidebar.
Ignore file missing?
Confirm that the filename starts with one full stop. Check that the file sits directly inside grant-risk-audit.
Help me place the Git ignore file correctly.
A separate upload ignore file controls which local content Azure Machine Learning receives during the later cloud run. This keeps the local environment plus Git history outside that upload.
- Use your editor's new-file control to create .amlignore inside grant-risk-audit.
- Copy these upload ignore rules into .amlignore:
.venv/
.git/
__pycache__/
*.pyc
artifacts/
azure-download/What does this file exclude?
The rules exclude the local environment plus Git history from the later code upload. They also exclude caches plus generated outputs.
- Save .amlignore.
- Confirm that .amlignore appears beside .gitignore in your editor sidebar.
Upload ignore file missing?
Check that the filename is exactly .amlignore. Confirm that it sits directly inside grant-risk-audit.
Help me place the upload ignore file correctly.
Install the pinned Python runtime
The project uses Python 3.12.16 regardless of the system runtime. A quick check shows which starting state applies to your machine.
- Check the system Python version by running this command:
python --versionWhat does this command do?
The command prints the Python runtime currently available through your shell. The uv-managed installation below provides the exact project runtime.
✔️ I see version 3.12.16
Your system command already reports the required Python release. The shared installation below also records that runtime under uv management.
ⓧ I see an older version
Your system command points to an older runtime. uv installs Python 3.12.16 for this project without changing the system runtime.
ⓧ Command not found
Your shell has no Python command available. uv supplies Python 3.12.16 through the shared installation below.
- Install Python 3.12.16 through uv by running this command:
uv python install 3.12.16What does this command do?
uv downloads Python 3.12.16 into its managed runtime collection. The system Python remains separate.
You should see confirmation that the requested Python runtime was installed.
Python installation failed?
Confirm that uv 0.13.7 is available in this terminal. Check that your internet connection remains active.
Help me install the pinned Python runtime.
A virtual environment isolates this project's interpreter context plus installed packages. The local environment lives in .venv.
- Create .venv with the pinned interpreter by running this command:
uv venv --python 3.12.16What does this command do?
uv creates a local virtual environment inside the current repository. It selects the managed Python 3.12.16 interpreter.
- Confirm that your editor sidebar shows the .venv directory.
Virtual environment missing?
Confirm that your terminal is inside ~/Desktop/grant-risk-audit. Check that the Python installation completed successfully.
Help me create the pinned virtual environment.
- Activate .venv in the current Linux shell by running this command:
source .venv/bin/activateWhat does activation change?
Activation makes the environment's Python executable the default for this terminal. Future package installations target .venv.
You should see the environment name at the start of your terminal prompt. That prefix confirms the isolated runtime is active.
Environment name missing?
Confirm that the .venv directory exists. Check that your terminal remains inside grant-risk-audit.
Help me activate the local virtual environment.
Pin and verify the dependencies
The audit uses openpyxl to read the public workbook. pandas prepares its tabular data.
scikit-learn supplies the later evaluation tools. Exact pins keep the package boundary consistent across compatible machines.
- Use your editor's new-file control to create requirements.txt inside grant-risk-audit.
- Copy these dependency pins into requirements.txt:
openpyxl==3.1.5
pandas==3.1.1
scikit-learn==1.9.1Why pin every dependency?
Each equality pin selects one exact package release. Reinstalling this file produces the same direct dependency versions on another compatible machine.
- Save requirements.txt.
- Confirm that requirements.txt appears beside both ignore files in your editor sidebar.
Dependency file looks different?
Check that every package occupies its own line. Confirm that each version uses two equals signs.
Help me correct the pinned requirements file.
✔️ Awesome, I've got everything!
Great. Double-check that all three files are saved directly inside grant-risk-audit.
ⓧ I'd like to double check the full code
.venv/
__pycache__/
*.pyc
artifacts/
azure-download/What should match in .gitignore?
Your file should contain these five rules in this order. They exclude the local environment plus generated project outputs.
.venv/
.git/
__pycache__/
*.pyc
artifacts/
azure-download/What should match in .amlignore?
Your file should contain these six rules in this order. The .git/ rule also excludes repository history from the later cloud upload.
openpyxl==3.1.5
pandas==3.1.1
scikit-learn==1.9.1What should match in requirements.txt?
Your file should contain the three exact dependency pins shown above. These versions define the reproducible package boundary.
- Install the pinned dependencies into the active environment by running this command:
uv pip install -r requirements.txtWhat does this command do?
uv reads each pin from requirements.txt. It installs those releases into the active .venv environment.
You should see the installation complete without a package resolution failure.
Dependency installation failed?
Confirm that the environment name appears in your terminal prompt. Check that requirements.txt matches the reference tab exactly.
Help me install the pinned audit dependencies.
Before you run the final check, which five version values do you expect the terminal to report?
- Verify the active runtime plus every pinned package by running this command:
python --version && uv --version && python -c "import pandas, sklearn, openpyxl; print(pandas.__version__, sklearn.__version__, openpyxl.__version__)"What should you see?
- The Python check reports 3.12.16.
- The uv check reports 0.13.7.
- The package check reports pandas 3.1.1.
- The package check reports scikit-learn 1.9.1.
- The package check reports openpyxl 3.1.5.
That locks down the environment. Your repository now uses the required Python runtime plus the exact dependency set for the audit.
Version does not match?
Confirm that .venv is active in this terminal. Check that the three dependency pins match the full-file reference.
Help me fix an environment version mismatch.
Your reproducible Linux environment is ready. Next, you will expose how a majority-class baseline can hide complete failure on one application outcome.
Expose the Misleading Baseline
Your pinned environment is ready. You can now turn that reproducible foundation into a command-line audit of historical SkatteFUNN application outcomes.
A single accuracy score can look useful when class imbalance lets a classifier ignore one outcome. This step makes that failure visible before you build the NLP pipeline.
In this step, get ready to:
- Load the official workbook through a defensive data pipeline.
- Detect the outcome column from its values without relying on a fixed header.
- Run a majority-class baseline with class-specific metrics.
Create the package and load the workbook
The official workbook can change its literal column headers. Your loader uses pandas with openpyxl to read the file. Defensive schema validation then identifies columns from their contents.
- Create a folder named grant_risk_audit inside your existing project repository using your code editor's file controls.
You should see the new grant_risk_audit folder beside requirements.txt in your file tree.
- Create grant_risk_audit/__init__.py with the code below:
"""Responsible audit tools for a grant-text classifier."""What does this code do?
This package docstring states the purpose of the audit tools. The file also makes the package boundary clear to readers and development tools.
- Save grant_risk_audit/__init__.py.
- Confirm that __init__.py appears inside the grant_risk_audit folder in your code editor.
Package file in the wrong place?
Check that grant_risk_audit sits in the same repository folder as requirements.txt. Move the package folder there if it appears somewhere else.
Confirm that the filename contains two underscores before init and two underscores after it.
Help me check my Python package structure.
✔️ Awesome, I've got everything!
Great. Your grant_risk_audit package now has its identifying file.
ⓧ I'd like to double check the full code
"""Responsible audit tools for a grant-text classifier."""The loader needs a stable data address and an explicit mapping between Norwegian outcomes and numeric labels. Keeping these values together makes every later evaluation use the same class order.
- Create grant_risk_audit/train.py with the loader foundation below:
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
DATA_URL = (
"https://www.forskningsradet.no/siteassets/skattefunn/tall/"
"skattefunn-innsendte-soknader-2002-2024.xlsx"
)
LABEL_MAP = {"avslått": 0, "godkjent": 1}
LABEL_NAMES = ["avslått", "godkjent"]
RANDOM_STATE = 42
def load_data(source: str) -> pd.DataFrame:
"""Load the official workbook or a compatible local workbook."""
return pd.read_excel(source, engine="openpyxl")What does this code do?
- The imports provide command-line parsing, JSON output, file paths, data handling, baseline classification, evaluation metrics, and dataset splitting.
- The constants preserve the official workbook address, numeric label mapping, class order, and fixed random state.
- The load_data() function reads either the official URL or a compatible local workbook through pandas.read_excel.
- The explicit engine="openpyxl" setting selects the workbook reader used by this project.
- Save grant_risk_audit/train.py.
- Check the file's syntax by running this command in the activated terminal:
python -m py_compile grant_risk_audit/train.pyWhat does this check prove?
The Python compiler parses train.py without running the workbook loader. A successful check returns you to the terminal prompt without a syntax error.
Your terminal should return to the prompt without printing an error. This confirms the loader foundation is valid Python.
Seeing a syntax error?
Compare the quotation marks around the workbook URL with the code block. Check that both lines remain inside the parentheses assigned to DATA_URL.
Confirm that every import appears at the top of train.py.
Help me fix the syntax in my loader foundation.
Header names are less dependable than the documented outcome values. The next function accepts exactly one column whose populated values are dominated by godkjent and avslått.
- Append the schema detector below load_data() in grant_risk_audit/train.py using this code:
def detect_schema(frame: pd.DataFrame) -> dict[str, str]:
"""Detect the binary outcome and longest substantial public text column."""
outcome_candidates: list[str] = []
for column in frame.columns:
values = frame[column].dropna().astype(str).str.strip().str.casefold()
if len(values) < 10:
continue
recognized = values.isin(LABEL_MAP)
recognized_values = set(values[recognized].unique())
if recognized.mean() >= 0.80 and recognized_values == set(LABEL_MAP):
outcome_candidates.append(str(column))
if len(outcome_candidates) != 1:
raise ValueError(
"Expected exactly one outcome column dominated by godkjent/avslått; "
f"found {outcome_candidates}."
)
outcome_column = outcome_candidates[0]
text_scores: list[tuple[float, str]] = []
for column in frame.columns:
column_name = str(column)
if column_name == outcome_column:
continue
values = frame[column].dropna().astype(str).str.strip()
if len(values) >= 10 and float(values.str.len().median()) >= 40:
text_scores.append((float(values.str.len().median()), column_name))
if not text_scores:
raise ValueError("No substantial text column was found.")
text_scores.sort(reverse=True)
return {"outcome": outcome_column, "text": text_scores[0][1]}How does schema detection work?
- The first loop normalizes populated values before measuring how many match the two documented outcomes.
- The exact-candidate check fails closed when the workbook contains no convincing outcome column or several possible columns.
- The second loop scores substantial text columns by median character length.
- The returned schema names the detected outcome column and the longest qualifying text column.
- Save grant_risk_audit/train.py.
- Check the schema detector's syntax by running:
python -m py_compile grant_risk_audit/train.pyWhat does this check prove?
This check parses the new function together with the loader foundation. Returning to the prompt confirms that the type annotations, conditions, and error paths are syntactically valid.
You should return to the terminal prompt without a syntax error.
Schema detector not compiling?
Check the indentation under both for loops. Confirm that the closing parentheses for the first ValueError align with its raise statement.
Help me fix my schema detector.
Detected columns still contain missing values, inconsistent capitalization, and repeated summaries. Cleaning these records creates one reliable dataset for the split and baseline.
- Append the dataset cleaner below detect_schema() using this code:
def prepare_dataset(frame: pd.DataFrame, schema: dict[str, str]) -> pd.DataFrame:
"""Normalize labels, remove empty text, and deduplicate public summaries."""
dataset = pd.DataFrame({
"text": frame[schema["text"]].astype("string").str.strip(),
"label_text": frame[schema["outcome"]].astype("string").str.strip().str.casefold(),
})
dataset["label"] = dataset["label_text"].map(LABEL_MAP)
dataset = dataset.dropna(subset=["text", "label"])
dataset = dataset[dataset["text"].str.len() >= 40]
dataset = dataset.drop_duplicates(subset=["text"]).reset_index(drop=True)
dataset["label"] = dataset["label"].astype(int)
counts = dataset["label"].value_counts()
if set(counts.index) != {0, 1} or int(counts.min()) < 10:
raise ValueError("Both outcome classes must contain at least 10 cleaned rows.")
return dataset[["text", "label"]]What does this code do?
- The new data frame extracts only the detected summary and outcome columns.
- The label mapping converts normalized outcome text into the fixed numeric order used by the metrics.
- The filters remove missing records, short summaries, and duplicate text.
- The class-count guard stops the audit when either cleaned outcome has fewer than ten rows.
- Save grant_risk_audit/train.py.
- Check the complete data-preparation layer by running:
python -m py_compile grant_risk_audit/train.pyWhat does this check prove?
The compiler now parses the loader, schema detector, and dataset cleaner as one module. A clean return confirms that the complete preparation layer is valid Python.
You should return to the prompt without an error. Your module can now load, detect, and clean the workbook defensively.
Cleaner not compiling?
Check that the dictionary passed to pd.DataFrame has a closing brace. Confirm that the final return contains the two columns in double brackets.
Help me fix my dataset cleaner.
Measure the majority-class baseline
Accuracy counts the share of correct predictions. Macro-F1 gives each class equal weight. A confusion matrix shows which actual outcomes landed in each predicted class.
The evaluator stores all three views in one JSON-ready structure. This keeps the majority baseline from hiding behind its strongest number.
- Append the evaluator below prepare_dataset() using this code:
def evaluate_predictions(y_true: Any, y_pred: Any) -> dict[str, Any]:
"""Return metrics that expose class-specific performance."""
report = classification_report(y_true, y_pred, labels=[0, 1], target_names=LABEL_NAMES, output_dict=True, zero_division=0)
return {
"accuracy": float(accuracy_score(y_true, y_pred)),
"macro_f1": float(report["macro avg"]["f1-score"]),
"per_class": {name: {"precision": float(report[name]["precision"]), "recall": float(report[name]["recall"]), "f1": float(report[name]["f1-score"]), "support": int(report[name]["support"])} for name in LABEL_NAMES},
"confusion_matrix": confusion_matrix(y_true, y_pred, labels=[0, 1]).tolist(),
"confusion_matrix_order": LABEL_NAMES,
}How are the metrics structured?
- The classification report calculates precision, recall, F1, and support for both ordered labels.
- The macro average produces one F1 score that treats both outcomes equally.
- The confusion matrix uses the explicit order avslått followed by godkjent.
- The returned dictionary contains values that can be serialized directly to JSON.
- Save grant_risk_audit/train.py.
- Check the evaluator's syntax by running:
python -m py_compile grant_risk_audit/train.pyWhat does this check prove?
This check parses the nested metric dictionary and its comprehension. Returning to the prompt confirms that the evaluator can be imported with the rest of the module.
You should return to the terminal prompt without a syntax error.
Evaluator not compiling?
Check the closing braces around per_class. Confirm that classification_report receives both labels and target_names.
Help me fix my metrics dictionary.
The command-line entry point connects preparation to a fixed stratified holdout. It then fits a DummyClassifier that always predicts the most frequent training label.
- Append the command-line entry point below evaluate_predictions() using this code:
def main() -> None:
parser = argparse.ArgumentParser(description="Audit a sensitive grant-text classifier.")
parser.add_argument("--source", default=DATA_URL)
parser.add_argument("--output-dir", default="artifacts")
parser.add_argument("--baseline-only", action="store_true")
args = parser.parse_args()
frame = load_data(args.source)
schema = detect_schema(frame)
dataset = prepare_dataset(frame, schema)
X_train, X_test, y_train, y_test = train_test_split(dataset["text"], dataset["label"], test_size=0.20, random_state=RANDOM_STATE, stratify=dataset["label"])
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit([[0] for _ in range(len(X_train))], y_train)
baseline = evaluate_predictions(y_test, dummy.predict([[0] for _ in range(len(X_test))]))
payload = {"schema": schema, "rows": int(len(dataset)), "class_counts": {LABEL_NAMES[int(label)]: int(count) for label, count in dataset["label"].value_counts().items()}, "baseline": baseline}
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "baseline_metrics.json").write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"Detected text column: {schema['text']}")
print(f"Detected outcome column: {schema['outcome']}")
print(f"Baseline accuracy: {baseline['accuracy']:.3f}")
print(f"Baseline macro-F1: {baseline['macro_f1']:.3f}")
if __name__ == "__main__":
main()How does the command work?
- The argument parser accepts a data source, an output directory, and the baseline-only flag.
- The fixed split preserves class proportions through stratification and uses RANDOM_STATE for reproducibility.
- The majority classifier learns the most frequent training outcome from dummy feature values.
- The command writes the schema, class counts, evaluation metrics, and ordered confusion matrix to baseline_metrics.json.
- Save grant_risk_audit/train.py.
- Check the completed baseline module by running:
python -m py_compile grant_risk_audit/train.pyWhat does this check prove?
The compiler now parses the complete command-line module. A clean return confirms that the entry point and all supporting functions form valid Python.
You should return to the prompt without an error. The baseline command is ready to run against the workbook.
Command module not compiling?
Check the parentheses around train_test_split and both DummyClassifier calls. Confirm that the final main() call is indented beneath its condition.
Help me fix my baseline entry point.
✔️ Awesome, I've got everything!
Your completed train.py module now contains the defensive loader and baseline-only command-line audit.
ⓧ I'd like to double check the full code
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
DATA_URL = (
"https://www.forskningsradet.no/siteassets/skattefunn/tall/"
"skattefunn-innsendte-soknader-2002-2024.xlsx"
)
LABEL_MAP = {"avslått": 0, "godkjent": 1}
LABEL_NAMES = ["avslått", "godkjent"]
RANDOM_STATE = 42
def load_data(source: str) -> pd.DataFrame:
"""Load the official workbook or a compatible local workbook."""
return pd.read_excel(source, engine="openpyxl")
def detect_schema(frame: pd.DataFrame) -> dict[str, str]:
"""Detect the binary outcome and longest substantial public text column."""
outcome_candidates: list[str] = []
for column in frame.columns:
values = frame[column].dropna().astype(str).str.strip().str.casefold()
if len(values) < 10:
continue
recognized = values.isin(LABEL_MAP)
recognized_values = set(values[recognized].unique())
if recognized.mean() >= 0.80 and recognized_values == set(LABEL_MAP):
outcome_candidates.append(str(column))
if len(outcome_candidates) != 1:
raise ValueError(
"Expected exactly one outcome column dominated by godkjent/avslått; "
f"found {outcome_candidates}."
)
outcome_column = outcome_candidates[0]
text_scores: list[tuple[float, str]] = []
for column in frame.columns:
column_name = str(column)
if column_name == outcome_column:
continue
values = frame[column].dropna().astype(str).str.strip()
if len(values) >= 10 and float(values.str.len().median()) >= 40:
text_scores.append((float(values.str.len().median()), column_name))
if not text_scores:
raise ValueError("No substantial text column was found.")
text_scores.sort(reverse=True)
return {"outcome": outcome_column, "text": text_scores[0][1]}
def prepare_dataset(frame: pd.DataFrame, schema: dict[str, str]) -> pd.DataFrame:
"""Normalize labels, remove empty text, and deduplicate public summaries."""
dataset = pd.DataFrame({
"text": frame[schema["text"]].astype("string").str.strip(),
"label_text": frame[schema["outcome"]].astype("string").str.strip().str.casefold(),
})
dataset["label"] = dataset["label_text"].map(LABEL_MAP)
dataset = dataset.dropna(subset=["text", "label"])
dataset = dataset[dataset["text"].str.len() >= 40]
dataset = dataset.drop_duplicates(subset=["text"]).reset_index(drop=True)
dataset["label"] = dataset["label"].astype(int)
counts = dataset["label"].value_counts()
if set(counts.index) != {0, 1} or int(counts.min()) < 10:
raise ValueError("Both outcome classes must contain at least 10 cleaned rows.")
return dataset[["text", "label"]]
def evaluate_predictions(y_true: Any, y_pred: Any) -> dict[str, Any]:
"""Return metrics that expose class-specific performance."""
report = classification_report(y_true, y_pred, labels=[0, 1], target_names=LABEL_NAMES, output_dict=True, zero_division=0)
return {
"accuracy": float(accuracy_score(y_true, y_pred)),
"macro_f1": float(report["macro avg"]["f1-score"]),
"per_class": {name: {"precision": float(report[name]["precision"]), "recall": float(report[name]["recall"]), "f1": float(report[name]["f1-score"]), "support": int(report[name]["support"])} for name in LABEL_NAMES},
"confusion_matrix": confusion_matrix(y_true, y_pred, labels=[0, 1]).tolist(),
"confusion_matrix_order": LABEL_NAMES,
}
def main() -> None:
parser = argparse.ArgumentParser(description="Audit a sensitive grant-text classifier.")
parser.add_argument("--source", default=DATA_URL)
parser.add_argument("--output-dir", default="artifacts")
parser.add_argument("--baseline-only", action="store_true")
args = parser.parse_args()
frame = load_data(args.source)
schema = detect_schema(frame)
dataset = prepare_dataset(frame, schema)
X_train, X_test, y_train, y_test = train_test_split(dataset["text"], dataset["label"], test_size=0.20, random_state=RANDOM_STATE, stratify=dataset["label"])
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit([[0] for _ in range(len(X_train))], y_train)
baseline = evaluate_predictions(y_test, dummy.predict([[0] for _ in range(len(X_test))]))
payload = {"schema": schema, "rows": int(len(dataset)), "class_counts": {LABEL_NAMES[int(label)]: int(count) for label, count in dataset["label"].value_counts().items()}, "baseline": baseline}
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "baseline_metrics.json").write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"Detected text column: {schema['text']}")
print(f"Detected outcome column: {schema['outcome']}")
print(f"Baseline accuracy: {baseline['accuracy']:.3f}")
print(f"Baseline macro-F1: {baseline['macro_f1']:.3f}")
if __name__ == "__main__":
main()Run the audit and inspect the failure
The baseline now faces the real workbook and a stratified holdout. Before you run it, do you think the accuracy score alone will reveal whether both outcomes are recognized?
- Run the baseline-only audit from your repository by entering:
python -m grant_risk_audit.train --baseline-onlyWhat does this command do?
The command reads the official workbook, detects its schema, cleans its records, and creates the fixed holdout. It writes the complete baseline audit to artifacts/baseline_metrics.json.
The terminal prints the detected column names with the headline accuracy and macro-F1 scores. The JSON artifact preserves the class-level evidence behind those summaries.
You should see a detected text column and a detected outcome column in the terminal. You should also see nonzero baseline accuracy beside a lower macro-F1 score.
Baseline command failed?
Confirm that your .venv remains active. Check that your internet connection can reach the official workbook address.
If schema detection stops the audit, compare train.py with the full-code tab. The detector intentionally stops when it cannot identify exactly one outcome column or any substantial text column.
Help me diagnose my baseline run.
- Open artifacts/baseline_metrics.json from your code editor's file tree.
- Locate the per_class object inside baseline.
- Compare each class's precision, recall, and f1 values.
- Inspect the rows in confusion_matrix using the order listed in confusion_matrix_order.
You should find nonzero accuracy even though the class that is never predicted has zero precision, zero recall, and zero F1. Its actual examples collect in the other prediction column of the confusion matrix.
Why is the accuracy misleading?
The majority predictor earns every correct decision from one outcome. Accuracy rewards those matches without requiring any success on the other outcome.
Macro-F1 gives both outcomes equal influence. The per-class metrics and confusion matrix reveal the failure that accuracy compresses into one appealing number.
You have exposed the trap: a classifier can achieve nonzero accuracy while completely failing one outcome. Your audit now makes that professional risk impossible to overlook.
Your baseline now proves why one headline metric cannot support a sensitive decision. Next, you'll train the text pipeline and compare its performance across both outcomes.
Train and Audit the NLP Pipeline
Your majority-class baseline has already exposed the danger of trusting accuracy alone. It can score correctly on the dominant outcome while completely missing the other class.
Now you'll train a scikit-learn Pipeline that keeps TF-IDF vectorization inside the fitted workflow. A class-balanced logistic regression model then faces the same holdout rows as the baseline.
In this step, get ready to:
- Train a text-classification pipeline on a fixed stratified split.
- Compare the model with the baseline using class-specific metrics.
- Generate text-length slice metrics and a responsible-use model card.
Map the fixed comparison
A fair comparison requires both predictors to face the same held-out applications. The fixed random state and stratification preserve that shared test set.
- Switch back to grant_risk_audit/train.py from the previous step.
- Locate RANDOM_STATE near the top of the file.
- Locate evaluate_predictions() beneath the data-cleaning functions.
Why use one fitted Pipeline?
The pipeline learns its vocabulary from the training rows during fit(). The held-out rows stay outside that learning process.
The vectorizer converts words and two-word phrases into weighted features. The classifier uses those features with inverse class-frequency weighting, which gives the less common outcome more influence during training.
What will the comparison measure?
- Accuracy measures the overall fraction of correct predictions.
- Macro-F1 gives both outcome classes equal importance.
- Per-class precision and recall expose which outcome the model handles poorly.
- The confusion matrix keeps avslått first and godkjent second throughout the audit.
Replace the runner with the complete audit
The complete runner adds three connected responsibilities. It trains the model, audits short and long summaries, and writes documentation that limits how the predictions may be used.
A model card records the intended educational use of the classifier. It also states that historical approval is an administrative outcome while proposal quality remains unmeasured.
- Select the entire contents of grant_risk_audit/train.py in your editor.
- Replace the selected content with the exact full-file reference in the second tab below.
✔️ Awesome, I've got everything!
Great. Save grant_risk_audit/train.py once your file matches the full reference.
ⓧ I'd like to double check the full code
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
DATA_URL = (
"https://www.forskningsradet.no/siteassets/skattefunn/tall/"
"skattefunn-innsendte-soknader-2002-2024.xlsx"
)
LABEL_MAP = {"avslått": 0, "godkjent": 1}
LABEL_NAMES = ["avslått", "godkjent"]
RANDOM_STATE = 42
def load_data(source: str) -> pd.DataFrame:
"""Load the official workbook or a compatible local workbook."""
return pd.read_excel(source, engine="openpyxl")
def detect_schema(frame: pd.DataFrame) -> dict[str, str]:
"""Detect the binary outcome and longest substantial public text column."""
outcome_candidates: list[str] = []
for column in frame.columns:
values = frame[column].dropna().astype(str).str.strip().str.casefold()
if len(values) < 10:
continue
recognized = values.isin(LABEL_MAP)
recognized_values = set(values[recognized].unique())
if recognized.mean() >= 0.80 and recognized_values == set(LABEL_MAP):
outcome_candidates.append(str(column))
if len(outcome_candidates) != 1:
raise ValueError(
"Expected exactly one outcome column dominated by godkjent/avslått; "
f"found {outcome_candidates}."
)
outcome_column = outcome_candidates[0]
text_scores: list[tuple[float, str]] = []
for column in frame.columns:
column_name = str(column)
if column_name == outcome_column:
continue
values = frame[column].dropna().astype(str).str.strip()
if len(values) < 10:
continue
median_length = float(values.str.len().median())
if median_length >= 40:
text_scores.append((median_length, column_name))
if not text_scores:
raise ValueError("No substantial text column was found.")
text_scores.sort(reverse=True)
return {"outcome": outcome_column, "text": text_scores[0][1]}
def prepare_dataset(
frame: pd.DataFrame, schema: dict[str, str]
) -> pd.DataFrame:
"""Normalize labels, remove empty text, and deduplicate public summaries."""
dataset = pd.DataFrame(
{
"text": frame[schema["text"]].astype("string").str.strip(),
"label_text": (
frame[schema["outcome"]].astype("string").str.strip().str.casefold()
),
}
)
dataset["label"] = dataset["label_text"].map(LABEL_MAP)
dataset = dataset.dropna(subset=["text", "label"])
dataset = dataset[dataset["text"].str.len() >= 40]
dataset = dataset.drop_duplicates(subset=["text"]).reset_index(drop=True)
dataset["label"] = dataset["label"].astype(int)
counts = dataset["label"].value_counts()
if set(counts.index) != {0, 1} or int(counts.min()) < 10:
raise ValueError(
"Both outcome classes must contain at least 10 cleaned rows."
)
return dataset[["text", "label"]]
def evaluate_predictions(y_true: Any, y_pred: Any) -> dict[str, Any]:
"""Return metrics that expose class-specific performance."""
report = classification_report(
y_true,
y_pred,
labels=[0, 1],
target_names=LABEL_NAMES,
output_dict=True,
zero_division=0,
)
return {
"accuracy": float(accuracy_score(y_true, y_pred)),
"macro_f1": float(report["macro avg"]["f1-score"]),
"per_class": {
name: {
"precision": float(report[name]["precision"]),
"recall": float(report[name]["recall"]),
"f1": float(report[name]["f1-score"]),
"support": int(report[name]["support"]),
}
for name in LABEL_NAMES
},
"confusion_matrix": confusion_matrix(
y_true, y_pred, labels=[0, 1]
).tolist(),
"confusion_matrix_order": LABEL_NAMES,
}
def build_model() -> Pipeline:
"""Create the production-shaped text pipeline."""
return Pipeline(
[
(
"tfidf",
TfidfVectorizer(
lowercase=True,
ngram_range=(1, 2),
min_df=2,
max_features=10_000,
sublinear_tf=True,
),
),
(
"classifier",
LogisticRegression(
class_weight="balanced",
l1_ratio=0.0,
max_iter=500,
),
),
]
)
def split_dataset(dataset: pd.DataFrame) -> tuple[Any, Any, Any, Any]:
"""Create one fixed, stratified holdout split."""
return train_test_split(
dataset["text"],
dataset["label"],
test_size=0.20,
random_state=RANDOM_STATE,
stratify=dataset["label"],
)
def build_slice_metrics(
texts: Any, y_true: Any, y_pred: Any
) -> dict[str, Any]:
"""Compare errors for shorter and longer held-out summaries."""
audit = pd.DataFrame(
{
"text": list(texts),
"truth": list(y_true),
"prediction": list(y_pred),
}
).reset_index(drop=True)
audit["length"] = audit["text"].str.len()
median_length = float(audit["length"].median())
audit["slice"] = audit["length"].map(
lambda value: "short" if value <= median_length else "long"
)
slices: dict[str, Any] = {"median_character_length": median_length}
for slice_name in ["short", "long"]:
rows = audit[audit["slice"] == slice_name]
if rows.empty:
continue
slices[slice_name] = {
"rows": int(len(rows)),
**evaluate_predictions(rows["truth"], rows["prediction"]),
}
return slices
def write_json(path: Path, payload: dict[str, Any]) -> None:
path.write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
def model_card(metrics: dict[str, Any], slices: dict[str, Any]) -> str:
"""Render the decision-focused documentation artifact."""
return f"""# Model Card: SkatteFUNN Text Risk Audit
## Purpose
This model is an educational risk-audit artifact. It tests whether public application text contains signals associated with historical approval outcomes.
## Forbidden use
Do not use this model to approve, reject, rank, prioritize, or score real applications. It does not measure project quality or eligibility and must not replace human assessment.
## Data
Source: {DATA_URL}
Detected text column: {metrics['schema']['text']}
Detected outcome column: {metrics['schema']['outcome']}
Cleaned rows: {metrics['rows']}
## Evaluation
Baseline accuracy: {metrics['baseline']['accuracy']:.3f}
Baseline macro-F1: {metrics['baseline']['macro_f1']:.3f}
Model accuracy: {metrics['model']['accuracy']:.3f}
Model macro-F1: {metrics['model']['macro_f1']:.3f}
Confusion-matrix order: {', '.join(LABEL_NAMES)}
Median held-out text length: {slices['median_character_length']:.0f} characters
## Limitations and risks
- Historical decisions are not objective ground truth for future funding quality.
- The workbook schema and collection process can change over time.
- Text may encode organization, sector, language, geography, and writing-style proxies.
- A random holdout does not prove temporal generalization or fairness across applicant groups.
- Public availability does not remove the need for data minimization and purpose limitation.
- Probability outputs are associations, not causal explanations.
## Human oversight
Any real decision-support proposal would require a documented legal basis, data-protection review, stakeholder consultation, temporal validation, subgroup analysis, appeal mechanisms, monitoring, and accountable human decision makers.
## Recommendation
Do not deploy this classifier for application screening. Use the repository only to demonstrate reproducible NLP evaluation, testing, cloud execution, and responsible model governance.
"""
def run_audit(
frame: pd.DataFrame,
output_dir: Path,
baseline_only: bool = False,
) -> tuple[Pipeline | None, dict[str, Any]]:
"""Run the baseline and, unless disabled, the full NLP audit."""
output_dir.mkdir(parents=True, exist_ok=True)
schema = detect_schema(frame)
dataset = prepare_dataset(frame, schema)
X_train, X_test, y_train, y_test = split_dataset(dataset)
dummy = DummyClassifier(strategy="most_frequent")
dummy_X_train = [[0] for _ in range(len(X_train))]
dummy_X_test = [[0] for _ in range(len(X_test))]
dummy.fit(dummy_X_train, y_train)
baseline = evaluate_predictions(y_test, dummy.predict(dummy_X_test))
baseline_payload = {
"schema": schema,
"rows": int(len(dataset)),
"class_counts": {
LABEL_NAMES[int(label)]: int(count)
for label, count in dataset["label"].value_counts().items()
},
"baseline": baseline,
}
write_json(output_dir / "baseline_metrics.json", baseline_payload)
print(f"Detected text column: {schema['text']}")
print(f"Detected outcome column: {schema['outcome']}")
print(f"Baseline accuracy: {baseline['accuracy']:.3f}")
print(f"Baseline macro-F1: {baseline['macro_f1']:.3f}")
if baseline_only:
return None, baseline_payload
model = build_model()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
model_metrics = evaluate_predictions(y_test, predictions)
slices = build_slice_metrics(X_test, y_test, predictions)
metrics = {
**baseline_payload,
"random_state": RANDOM_STATE,
"test_fraction": 0.20,
"model": model_metrics,
}
write_json(output_dir / "metrics.json", metrics)
write_json(output_dir / "slice_metrics.json", slices)
(output_dir / "MODEL_CARD.md").write_text(
model_card(metrics, slices), encoding="utf-8"
)
print(f"Model accuracy: {model_metrics['accuracy']:.3f}")
print(f"Model macro-F1: {model_metrics['macro_f1']:.3f}")
print(f"Artifacts written to: {output_dir}")
return model, metrics
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Audit a sensitive grant-text classifier."
)
parser.add_argument("--source", default=DATA_URL)
parser.add_argument("--output-dir", default="artifacts")
parser.add_argument("--baseline-only", action="store_true")
return parser.parse_args()
def main() -> None:
args = parse_args()
frame = load_data(args.source)
run_audit(
frame,
Path(args.output_dir),
baseline_only=args.baseline_only,
)
if __name__ == "__main__":
main()
How does the complete audit fit together?
- The build_model() function combines text vectorization with class-balanced classification.
- The split_dataset() function creates one reproducible holdout for both predictors.
- The build_slice_metrics() function compares summaries at or below the median length with summaries above it.
- The run_audit() function writes the baseline, model, slice, and governance artifacts from one command.
- Save grant_risk_audit/train.py.
Run and inspect the audit
Before you run the complete audit, which metric do you expect to expose the clearest difference between the majority baseline and the fitted model?
The first full run downloads the workbook before training the model. A quiet terminal during that work does not mean the process has stalled.
- Run the complete audit from the activated environment by entering:
python -m grant_risk_audit.trainWhat does this run prove?
- The terminal names the detected text and outcome columns.
- The terminal reports baseline accuracy and baseline macro-F1.
- The terminal reports model accuracy and model macro-F1.
- The final line confirms that the audit wrote its artifacts to artifacts.
That is the full evaluation loop working. One command now moves from the public workbook to model metrics, slice evidence, and responsible-use documentation.
- Expand the artifacts folder in your editor sidebar.
- Open metrics.json from the artifacts folder.
- Open slice_metrics.json from the artifacts folder.
- Open MODEL_CARD.md from the artifacts folder.
You'll see baseline_metrics.json, metrics.json, slice_metrics.json, and MODEL_CARD.md in the folder.
What should you inspect?
In metrics.json, compare the baseline and model accuracy values with their macro-F1 values. Inspect precision, recall, F1, support, and the confusion matrix for both outcomes.
In slice_metrics.json, compare the short and long slices around the held-out median character length. A gap shows that aggregate metrics can hide performance differences linked to summary length.
In MODEL_CARD.md, confirm the forbidden uses, privacy context, human-oversight requirements, and no-deployment recommendation. Those limits are part of the product's output.
Audit command did not finish?
Confirm that the activated environment from the earlier setup step is still active. Missing package messages usually mean the command is using a different Python environment.
Confirm that your Linux machine can reach the internet because the default source is the official workbook URL. If the command reaches training but does not write every artifact, compare grant_risk_audit/train.py with the full-file reference above.
Help me diagnose my NLP audit run.
Your audit now trains a reproducible NLP model while keeping its risks visible. Next up, you'll prove the product behaves safely with automated tests and operating documentation.
Prove Product Quality
Your NLP audit now produces baseline metrics, model metrics, slice analysis, and a responsible-use model card. Strong metrics still cannot prove that the product handles broken data safely.
Automated tests turn expected behavior into repeatable checks. A complete README.md gives another developer the commands and operating context needed to reproduce your results.
In this step, get ready to:
- Build synthetic tests for defensive data handling.
- Verify end-to-end artifact generation.
- Document the product's operation in README.md.
Test defensive data handling
A synthetic fixture gives every test known inputs without downloading the public workbook. It also lets you break the schema deliberately without changing production data.
- Create a tests folder inside your project folder using your code editor's file sidebar.
- Create tests/__init__.py inside the new folder using the file sidebar.
- Identify the test package by adding this code to tests/__init__.py:
"""Tests for the responsible grant-text audit."""What Does This File Do?
The module docstring identifies tests as the test package for this audit.
- Save tests/__init__.py.
- Confirm that tests/__init__.py appears beneath the tests folder in your editor's file sidebar.
✔️ Awesome, I've got everything!
Great. Your test package is ready for its test module.
ⓧ I'd like to double check the full code
"""Tests for the responsible grant-text audit."""- Create tests/test_train.py inside the tests folder using your editor's file sidebar.
- Add the test dependencies and production imports by pasting this code into tests/test_train.py:
import json
import tempfile
import unittest
from pathlib import Path
import pandas as pd
from grant_risk_audit.train import (
detect_schema,
prepare_dataset,
run_audit,
)What Do These Imports Provide?
- The standard-library imports support JSON inspection, temporary output folders, test assertions, and file paths.
- Pandas builds the synthetic in-memory workbook fixture.
- The production imports expose the exact schema, cleaning, and audit functions under test.
- Save tests/test_train.py.
- Confirm that the editor shows the three imported audit functions at the bottom of the import group.
- Add the synthetic Norwegian fixture beneath the imports by pasting this code:
def fixture_frame(rows_per_class: int = 30) -> pd.DataFrame:
rows = []
for index in range(rows_per_class):
rows.append(
{
"Søker": f"Grønn Bedrift {index}",
"Resultat": "Godkjent",
"Prosjekttittel": f"Effektiv energistyring {index}",
"Populærvitenskapelig sammendrag": (
"Prosjektet forsker på energieffektiv styring, måling og "
f"systematisk validering i norske industribygg nummer {index}."
),
}
)
rows.append(
{
"Søker": f"Uklar Bedrift {index}",
"Resultat": "Avslått",
"Prosjekttittel": f"Generell markedsføring {index}",
"Populærvitenskapelig sammendrag": (
"Prosjektet beskriver ordinær markedsføring, salg og daglig "
f"drift uten en avgrenset forskningsmetode nummer {index}."
),
}
)
return pd.DataFrame(rows)How Does the Fixture Work?
- Each loop adds one approved row and one rejected row with the documented Norwegian outcome values.
- The summary field contains enough text to pass the production length filter.
- The two text patterns give the integration test a learnable signal without using the public workbook.
- Save tests/test_train.py.
- Check that the test module imports cleanly by running this command in the activated terminal:
python -m unittest discover -vWhat Does This Command Check?
Python discovers test modules beneath the current project folder. Verbose mode prints the name and result of every discovered test.
The command should complete without an import failure. This proves that the synthetic fixture can be loaded by the test runner.
Does Test Discovery Fail?
Confirm that your activated terminal is in the project folder containing grant_risk_audit and tests.
Check that the file is named test_train.py so discovery can identify it.
Help me fix test discovery for this project.
- Add the schema and cleaning test class beneath fixture_frame() by pasting this code:
class GrantRiskAuditTests(unittest.TestCase):
def test_detects_schema_without_fixed_headers(self) -> None:
frame = fixture_frame()
schema = detect_schema(frame)
self.assertEqual(schema["outcome"], "Resultat")
self.assertEqual(
schema["text"], "Populærvitenskapelig sammendrag"
)
def test_prepare_dataset_maps_both_labels(self) -> None:
frame = fixture_frame()
dataset = prepare_dataset(frame, detect_schema(frame))
self.assertEqual(set(dataset["label"]), {0, 1})
self.assertEqual(len(dataset), 60)
def test_missing_outcome_fails_safely(self) -> None:
frame = fixture_frame().drop(columns=["Resultat"])
with self.assertRaises(ValueError):
detect_schema(frame)What Do These Tests Prove?
- The first test proves that schema detection uses values instead of fixed workbook headers.
- The second test proves that both Norwegian labels become numeric classes without losing rows.
- The third test proves that a missing outcome field causes a deliberate failure.
- Save tests/test_train.py.
- Run the three defensive data tests with this command:
python -m unittest discover -vWhat Is Being Verified?
The runner discovers each method whose name begins with test_. Every assertion checks the observable contract of the production functions.
You'll see the three test names in the terminal. Each test line ends with ok.
Is One of the Data Tests Failing?
Compare the Norwegian fixture keys and labels with the code block above. A spelling change can alter schema detection.
Confirm that each class still contains 30 unique rows.
Help me diagnose a failing schema or cleaning test.
Test end-to-end artifact generation
Unit tests isolate individual contracts. An integration test runs the complete audit against the synthetic fixture and inspects the files that a real run depends on.
- Append the integration test and module entry point beneath the existing methods in tests/test_train.py by pasting this code:
def test_end_to_end_audit_writes_documented_outputs(self) -> None:
frame = fixture_frame()
with tempfile.TemporaryDirectory() as directory:
output_dir = Path(directory)
model, metrics = run_audit(frame, output_dir)
self.assertIsNotNone(model)
self.assertIn("model", metrics)
self.assertTrue((output_dir / "metrics.json").exists())
self.assertTrue((output_dir / "slice_metrics.json").exists())
self.assertTrue((output_dir / "MODEL_CARD.md").exists())
saved = json.loads(
(output_dir / "metrics.json").read_text(encoding="utf-8")
)
self.assertGreaterEqual(saved["model"]["macro_f1"], 0.5)
if __name__ == "__main__":
unittest.main()What Does the Integration Test Cover?
- A temporary directory keeps generated files isolated from your normal artifacts folder.
- The test confirms that the trained model and its metrics are returned.
- The file assertions prove that metrics, slice analysis, and the model card are generated together.
- The macro-F1 assertion checks that the synthetic patterns produce a meaningful learned result.
- Save tests/test_train.py.
Before you run the complete suite, do you expect the synthetic text patterns to satisfy every assertion?
- Run the complete test suite by entering this command:
python -m unittest discover -vWhat Does a Passing Suite Establish?
The four tests exercise schema detection, label cleaning, safe failure, and artifact generation. The final result covers both isolated behavior and the complete local workflow.
You'll see all four test methods listed in the terminal. Every test line ends with ok.
Does the Integration Test Fail?
Confirm that the integration method remains indented inside GrantRiskAuditTests.
Check the temporary output paths against the exact artifact names generated by run_audit().
Help me diagnose the failing end-to-end audit test.
✔️ Awesome, I've got everything!
Excellent. Save tests/test_train.py before moving to the project documentation.
ⓧ I'd like to double check the full code
import json
import tempfile
import unittest
from pathlib import Path
import pandas as pd
from grant_risk_audit.train import (
detect_schema,
prepare_dataset,
run_audit,
)
def fixture_frame(rows_per_class: int = 30) -> pd.DataFrame:
rows = []
for index in range(rows_per_class):
rows.append(
{
"Søker": f"Grønn Bedrift {index}",
"Resultat": "Godkjent",
"Prosjekttittel": f"Effektiv energistyring {index}",
"Populærvitenskapelig sammendrag": (
"Prosjektet forsker på energieffektiv styring, måling og "
f"systematisk validering i norske industribygg nummer {index}."
),
}
)
rows.append(
{
"Søker": f"Uklar Bedrift {index}",
"Resultat": "Avslått",
"Prosjekttittel": f"Generell markedsføring {index}",
"Populærvitenskapelig sammendrag": (
"Prosjektet beskriver ordinær markedsføring, salg og daglig "
f"drift uten en avgrenset forskningsmetode nummer {index}."
),
}
)
return pd.DataFrame(rows)
class GrantRiskAuditTests(unittest.TestCase):
def test_detects_schema_without_fixed_headers(self) -> None:
frame = fixture_frame()
schema = detect_schema(frame)
self.assertEqual(schema["outcome"], "Resultat")
self.assertEqual(
schema["text"], "Populærvitenskapelig sammendrag"
)
def test_prepare_dataset_maps_both_labels(self) -> None:
frame = fixture_frame()
dataset = prepare_dataset(frame, detect_schema(frame))
self.assertEqual(set(dataset["label"]), {0, 1})
self.assertEqual(len(dataset), 60)
def test_missing_outcome_fails_safely(self) -> None:
frame = fixture_frame().drop(columns=["Resultat"])
with self.assertRaises(ValueError):
detect_schema(frame)
def test_end_to_end_audit_writes_documented_outputs(self) -> None:
frame = fixture_frame()
with tempfile.TemporaryDirectory() as directory:
output_dir = Path(directory)
model, metrics = run_audit(frame, output_dir)
self.assertIsNotNone(model)
self.assertIn("model", metrics)
self.assertTrue((output_dir / "metrics.json").exists())
self.assertTrue((output_dir / "slice_metrics.json").exists())
self.assertTrue((output_dir / "MODEL_CARD.md").exists())
saved = json.loads(
(output_dir / "metrics.json").read_text(encoding="utf-8")
)
self.assertGreaterEqual(saved["model"]["macro_f1"], 0.5)
if __name__ == "__main__":
unittest.main()Document product operation
Tests prove the code behaves as expected. The repository also needs operating instructions that explain reproduction, architecture, responsible use, and the planned Azure Machine Learning workflow.
- Create README.md inside your project folder using your editor's file sidebar.
- Add the project position and reproducible local setup by pasting this first section:
# Responsible Grant-Text Risk Audit
This repository is a production-minded NLP evaluation exercise for a deliberately sensitive public-sector use case. It predicts historical SkatteFUNN application outcomes from public text so that its limitations, class imbalance, wording sensitivity, and governance risks can be audited.
It is not a funding recommendation system. Do not use it to approve, reject, rank, prioritize, or score real applications.
## What it demonstrates
- Reproducible Python and dependency setup
- Defensive workbook schema detection
- A majority-class baseline before model training
- TF-IDF and class-balanced logistic regression in one pipeline
- Accuracy, macro-F1, per-class metrics, confusion matrix, and text-length slices
- Automated unit and integration tests
- A generated model card with a no-deployment recommendation
- A tracked Azure Machine Learning serverless command job
## Local setup
```bash
curl -LsSf https://astral.sh/uv/0.13.7/install.sh | sh
source "$HOME/.local/bin/env"
uv python install 3.12.16
uv venv --python 3.12.16
source .venv/bin/activate
uv pip install -r requirements.txt
```What Does This Section Establish?
The opening defines the repository as an evaluation exercise. The warning states the forbidden funding uses before readers reach any model commands.
The local setup records the pinned environment commands that make the project reproducible on another Linux machine.
- Save README.md.
- Select your editor's Markdown preview control from the editor toolbar.
- Confirm that the preview shows the project title, the capability list, and the local setup code block.
Does the Markdown Preview Look Broken?
Check that the opening and closing code fences each contain three backticks. An incomplete fence can absorb every section beneath it.
Help me repair the first README section.
- Append the local commands and architecture section beneath the existing content by pasting this code:
## Run
```bash
python -m grant_risk_audit.train --baseline-only
python -m grant_risk_audit.train
python -m unittest discover -v
```
Generated files appear in `artifacts/`:
- `baseline_metrics.json`
- `metrics.json`
- `slice_metrics.json`
- `MODEL_CARD.md`
## Architecture
The loader reads an official Excel workbook. Schema detection searches for one column dominated by the documented outcomes `godkjent` and `avslått`, then selects the longest substantial text field. Cleaning removes missing, short, and duplicate summaries. A fixed stratified holdout is shared by a majority baseline and a TF-IDF logistic regression pipeline. The evaluator writes machine-readable metrics and a decision-focused model card.Why Document Commands and Architecture?
The run section gives one path from the baseline through the complete test suite. The artifact list makes the result of each run inspectable.
The architecture section explains how data moves from schema detection to evaluation. A reader can understand the design without opening the implementation first.
- Save README.md.
- Confirm that the Markdown preview shows separate Run and Architecture sections.
Are the Run Commands Missing From the Preview?
Confirm that the bash code fence closes before the generated artifact list begins.
Help me fix the README run section.
- Append the serverless cloud workflow beneath the architecture section by pasting this code:
## Azure run
Install Azure CLI and the Machine Learning extension, sign in, and create isolated project resources:
```bash
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az extension add -n ml
az login
GROUP="rg-grant-risk-audit"
LOCATION="eastus"
WORKSPACE="mlw-grant-risk-audit"
az group create -n "$GROUP" -l "$LOCATION"
az ml workspace create -n "$WORKSPACE" -g "$GROUP" -l "$LOCATION"
JOB_NAME=$(az ml job create --file azure-job.yml --resource-group "$GROUP" --workspace-name "$WORKSPACE" --query name -o tsv)
az ml job stream --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE"
az ml job download --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE" --download-path azure-download --output-name audit
```Why Include the Azure Workflow Now?
The README records the exact serverless job workflow that the repository supports. The next step applies these documented commands to real Azure resources.
The dedicated names keep the future workspace and resource group tied to this project.
- Save README.md.
- Confirm that the preview shows the Azure workflow as one fenced command block.
Does the Azure Block Spill Into the Next Section?
Check for three closing backticks immediately after the output download command.
Help me fix the Azure workflow formatting.
- Complete the README with cleanup, responsible-use limits, and interview discussion points by pasting this final section:
## Cleanup
```bash
az group delete --name rg-grant-risk-audit --yes --no-wait
```
Deleting the resource group is irreversible and removes the resources inside it.
## Responsible-use position
Historical approval is not objective ground truth for proposal quality. Text can encode organization, language, sector, geography, and writing-style proxies. Random holdout performance does not establish fairness, temporal validity, legal permissibility, or operational safety. Any real decision-support proposal would require a legal basis, data-protection review, temporal and subgroup validation, stakeholder consultation, monitoring, appeal mechanisms, and accountable human decision makers.
## Interview discussion points
- Why macro-F1 and class-specific recall matter when approvals dominate
- Why the preprocessing and classifier belong in one fitted pipeline
- Why schema detection should fail closed instead of guessing silently
- Why serverless execution proves portability without creating a permanent cluster
- Why technically predictive text can still be unsuitable for public decision automationWhat Does the Final Section Add?
The cleanup section records the irreversible resource-group deletion command before any cloud resources exist. The responsible-use section states why predictive performance cannot justify automated funding decisions.
The interview prompts connect technical choices to class imbalance, leakage prevention, safe failure, portability, and governance.
- Save README.md.
- Confirm that the preview ends with five interview discussion points.
Is the README Missing a Final Section?
Compare the heading order with the full-file reference below. Each section should appear once.
Help me compare my README with the required structure.
✔️ Awesome, I've got everything!
Your repository now explains how to reproduce, operate, evaluate, and clean up the audit.
ⓧ I'd like to double check the full code
# Responsible Grant-Text Risk Audit
This repository is a production-minded NLP evaluation exercise for a deliberately sensitive public-sector use case. It predicts historical SkatteFUNN application outcomes from public text so that its limitations, class imbalance, wording sensitivity, and governance risks can be audited.
It is not a funding recommendation system. Do not use it to approve, reject, rank, prioritize, or score real applications.
## What it demonstrates
- Reproducible Python and dependency setup
- Defensive workbook schema detection
- A majority-class baseline before model training
- TF-IDF and class-balanced logistic regression in one pipeline
- Accuracy, macro-F1, per-class metrics, confusion matrix, and text-length slices
- Automated unit and integration tests
- A generated model card with a no-deployment recommendation
- A tracked Azure Machine Learning serverless command job
## Local setup
```bash
curl -LsSf https://astral.sh/uv/0.13.7/install.sh | sh
source "$HOME/.local/bin/env"
uv python install 3.12.16
uv venv --python 3.12.16
source .venv/bin/activate
uv pip install -r requirements.txt
```
## Run
```bash
python -m grant_risk_audit.train --baseline-only
python -m grant_risk_audit.train
python -m unittest discover -v
```
Generated files appear in `artifacts/`:
- `baseline_metrics.json`
- `metrics.json`
- `slice_metrics.json`
- `MODEL_CARD.md`
## Architecture
The loader reads an official Excel workbook. Schema detection searches for one column dominated by the documented outcomes `godkjent` and `avslått`, then selects the longest substantial text field. Cleaning removes missing, short, and duplicate summaries. A fixed stratified holdout is shared by a majority baseline and a TF-IDF logistic regression pipeline. The evaluator writes machine-readable metrics and a decision-focused model card.
## Azure run
Install Azure CLI and the Machine Learning extension, sign in, and create isolated project resources:
```bash
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az extension add -n ml
az login
GROUP="rg-grant-risk-audit"
LOCATION="eastus"
WORKSPACE="mlw-grant-risk-audit"
az group create -n "$GROUP" -l "$LOCATION"
az ml workspace create -n "$WORKSPACE" -g "$GROUP" -l "$LOCATION"
JOB_NAME=$(az ml job create --file azure-job.yml --resource-group "$GROUP" --workspace-name "$WORKSPACE" --query name -o tsv)
az ml job stream --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE"
az ml job download --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE" --download-path azure-download --output-name audit
```
## Cleanup
```bash
az group delete --name rg-grant-risk-audit --yes --no-wait
```
Deleting the resource group is irreversible and removes the resources inside it.
## Responsible-use position
Historical approval is not objective ground truth for proposal quality. Text can encode organization, language, sector, geography, and writing-style proxies. Random holdout performance does not establish fairness, temporal validity, legal permissibility, or operational safety. Any real decision-support proposal would require a legal basis, data-protection review, temporal and subgroup validation, stakeholder consultation, monitoring, appeal mechanisms, and accountable human decision makers.
## Interview discussion points
- Why macro-F1 and class-specific recall matter when approvals dominate
- Why the preprocessing and classifier belong in one fitted pipeline
- Why schema detection should fail closed instead of guessing silently
- Why serverless execution proves portability without creating a permanent cluster
- Why technically predictive text can still be unsuitable for public decision automationBefore you run the documented local workflow, which artifacts do you expect the full audit to recreate?
- Reproduce the documented local workflow by running these commands in the activated terminal:
python -m grant_risk_audit.train --baseline-only
python -m grant_risk_audit.train
python -m unittest discover -vWhat Does This Final Check Prove?
The first command reproduces the majority-class baseline. The second command regenerates the complete local audit artifacts.
The final command checks the production functions against synthetic data without relying on notebook state.
You'll see the detected columns and both sets of metrics in the terminal. The test runner reports every test as ok.
That is the product-quality proof complete. Your audit is reproducible, tested, documented, and designed to fail safely on a broken outcome schema.
Does the Final Workflow Fail?
Confirm that the local environment remains activated before running the documented commands.
Use the first failing command to narrow the problem to data access, audit execution, or test discovery.
Help me diagnose the final local workflow.
Your local audit now has repeatable quality checks and operating documentation. Next, you'll run the same tested workload as a tracked cloud job.
Run the Audit in Azure
Your test suite proves that the audit works on your machine. A tracked cloud run now needs to prove that the same product works outside your local environment.
In this step, you will run the audit as an Azure Machine Learning command job. Managed serverless compute keeps the run portable without creating a permanent cluster.
In this step, get ready to:
- Install the required Azure command-line tools.
- Create an Azure Machine Learning workspace for the audit.
- Submit the audit as a tracked serverless job.
Install and connect the Azure tools
The Azure CLI sends commands to your Azure account. Its Machine Learning extension adds the commands that create workspaces and submit jobs.
- Check your installed Azure CLI version by running this command:
az versionWhat does this command do?
This command reports the installed Azure CLI version. The project uses version 2.92.0.
✔️ I see version 2.92.0
Your Azure CLI version matches the project pin. Continue to the extension check below.
ⓧ I see an older version
Your installed Azure CLI needs an update before it can match this project.
- Update Azure CLI with the official Debian or Ubuntu installer by running:
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bashWhat does this command do?
The installer configures Microsoft's package source for Azure CLI. It installs the current release through your Linux package manager.
- Check the updated Azure CLI version by running:
az versionWhat should I see?
The version report should now identify Azure CLI 2.92.0.
ⓧ Command not found
Azure CLI is missing from your Linux machine. The installer uses sudo, so your terminal may ask for your Linux password.
- Install Azure CLI with the official Debian or Ubuntu installer by running:
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bashWhat does this command do?
The installer adds Microsoft's package source to your machine. It installs Azure CLI through the system package manager.
- Confirm the Azure CLI installation by running:
az versionWhat should I see?
The version report should identify Azure CLI 2.92.0.
Azure CLI installation failed?
Confirm that your Linux account can use sudo. Check that your machine can reach Microsoft's package servers.
If the installation completed but the command is still unavailable, start a fresh terminal session before checking again.
Help me diagnose my Azure CLI installation.
Azure Machine Learning commands live in the separate ml extension. Checking it separately catches missing or outdated extension installations before a job is submitted.
- Check your installed Azure CLI extensions by running:
az extension listWhat does this command do?
This command lists every installed Azure CLI extension. Find the entry named ml and read its version.
✔️ I see ml version 2.45.2
The Machine Learning extension matches the project pin. Your command-line tools are ready.
ⓧ I see an older ml version
The extension is installed at an older version. Update it before creating project resources.
- Update the installed Machine Learning extension by running:
az extension update -n mlWhat does this command do?
This command updates the existing ml extension. It leaves the main Azure CLI installation in place.
- Check the updated extension version by running:
az extension listWhat should I see?
The ml entry should report version 2.45.2.
ⓧ The ml extension is missing
The Machine Learning extension is not installed yet. Adding it gives Azure CLI the workspace and job commands used below.
- Install the Machine Learning extension by running:
az extension add -n mlWhat does this command do?
This command downloads the supported ml extension. The extension adds Azure Machine Learning commands to Azure CLI.
- Confirm the extension installation by running:
az extension listWhat should I see?
The extension list should contain ml at version 2.45.2.
Machine Learning extension failed?
Confirm that az version works before installing the extension. An unavailable Azure CLI installation prevents extension management.
Check your internet connection if the extension download stops before completion.
Help me fix the Azure Machine Learning extension.
Interactive sign-in opens a browser for authentication. Azure CLI then lists the subscriptions available to your account so you can choose the one used for this project.
- Start the interactive Azure sign-in flow by running:
az loginWhat does this command do?
This command authenticates Azure CLI through your browser. Your selected subscription becomes the default for the current CLI session.
- Complete the sign-in request in your browser.
- Return to your terminal after authentication.
- Enter the number for a subscription where you can create resources.
Your terminal now uses the selected subscription. The next command confirms whether your account has permission to create the project resource group.
Can't complete Azure sign-in?
Confirm that the browser account matches the Azure account you intend to use. Your account also needs access to an active subscription.
If no usable subscription appears, contact the subscription administrator for resource-creation permission.
Help me troubleshoot Azure CLI sign-in.
Create the Azure project resources
A dedicated resource group keeps the workspace and its dependent services together. That boundary makes the whole cloud environment removable with one cleanup command.
Azure resources can incur costs
Azure Machine Learning has no additional service charge. The job's compute and dependent services can incur pay-as-you-go charges.
The cleanup section removes the dedicated resource group. Deleting that group stops persistent project-resource costs.
- Set the project resource names and create the resource group by running:
GROUP="rg-grant-risk-audit"
LOCATION="eastus"
WORKSPACE="mlw-grant-risk-audit"
az group create -n "$GROUP" -l "$LOCATION"What does this command do?
- The shell variables keep the resource names consistent across later commands.
- The final command creates rg-grant-risk-audit in eastus.
Azure returns a resource description for rg-grant-risk-audit. This confirms that the selected subscription accepted the resource-group request.
Resource group creation failed?
A permissions failure means the selected subscription does not allow your account to create resource groups. Sign in with an authorized account or ask the subscription administrator for access.
Help me diagnose Azure resource-group creation.
Workspace provisioning can take a few minutes because Azure creates dependent services behind the workspace. A quiet terminal during this period does not mean the command has stopped.
- Create the Machine Learning workspace by running:
az ml workspace create -n "$WORKSPACE" -g "$GROUP" -l "$LOCATION"What does this command do?
This command creates mlw-grant-risk-audit inside the dedicated resource group. Azure also provisions the dependent resources needed by the workspace.
Azure returns a successful workspace description containing mlw-grant-risk-audit and eastus. The workspace is now ready to receive a job.
Workspace provisioning failed?
Confirm that the resource-group command succeeded before retrying the workspace command. A quota or regional policy can also block provisioning in the selected subscription.
Help me diagnose Azure Machine Learning workspace creation.
Submit and verify the serverless job
A command-job file describes the code upload and runtime command. Leaving out a compute target tells Azure Machine Learning to use fully managed serverless compute.
- Create azure-job.yml at the top level of your project by pasting this configuration:
$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: .
command: >-
python -m pip install --no-cache-dir -r requirements.txt &&
python -m grant_risk_audit.train
--output-dir ${{outputs.audit}}
environment:
image: library/python:3.12.16
outputs:
audit:
display_name: grant-risk-audit
experiment_name: grant-risk-audit
description: Responsible evaluation of a historical grant-text classifier.What does this configuration do?
- The schema URL lets tooling validate the command-job structure.
- The code setting uploads the current project directory.
- The command installs requirements.txt before running the audit.
- The audit output binds generated artifacts to tracked Azure storage.
- The image uses Python 3.12.16 to match the local runtime.
- Save azure-job.yml.
- Confirm that azure-job.yml appears beside requirements.txt in your editor's file sidebar.
Configuration file looks wrong?
Confirm that the filename is exactly azure-job.yml. Keep the indentation under command, environment, and outputs unchanged.
Help me check my Azure job configuration.
✔️ Awesome, I've got everything!
Great. Confirm that azure-job.yml is saved at the top level of the project.
ⓧ I'd like to double check the full code
$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: .
command: >-
python -m pip install --no-cache-dir -r requirements.txt &&
python -m grant_risk_audit.train
--output-dir ${{outputs.audit}}
environment:
image: library/python:3.12.16
outputs:
audit:
display_name: grant-risk-audit
experiment_name: grant-risk-audit
description: Responsible evaluation of a historical grant-text classifier.How should I compare this file?
Compare each line with your saved azure-job.yml. The indentation and output expression must match exactly.
- Return to the terminal panel from earlier.
- Confirm that its current folder contains azure-job.yml.
Before you submit the file, do you expect Azure to create a permanent cluster or use managed serverless compute?
- Submit the job and store its generated name by running:
JOB_NAME=$(az ml job create --file azure-job.yml --resource-group "$GROUP" --workspace-name "$WORKSPACE" --query name -o tsv)What does this command do?
Azure uploads the project code and creates the command job from azure-job.yml. The generated job name is saved in JOB_NAME for the remaining commands.
The job now exists as a tracked run under the grant-risk-audit experiment. Azure uses serverless compute because the job file does not name a compute target.
Job submission failed?
Confirm that the terminal is at the top level of the project. Azure must be able to find both azure-job.yml and requirements.txt.
A permissions failure means your account cannot create jobs in mlw-grant-risk-audit.
Help me diagnose my Azure Machine Learning job submission.
The first serverless run can take several minutes while Azure prepares compute and installs the pinned packages. The logs keep the wait visible by showing each stage as it runs.
Before you stream the job, which local audit messages do you expect to reappear in the cloud logs?
- Stream the tracked job logs by running:
az ml job stream --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE"What does this command do?
This command follows the saved job until it finishes. It displays dependency installation and the audit's own terminal output.
You will see the detected text and outcome columns in the logs. You will also see baseline metrics and model metrics before Azure reports that the job completed.
Cloud job did not complete?
Read upward from the final log lines to find the first failed command. Dependency failures point to requirements.txt, while import failures usually mean the project code was not uploaded from the correct folder.
Help me interpret my Azure job logs.
Before you download the named output, which three audit files should the cloud run have produced?
- Download the tracked audit output by running:
az ml job download --name "$JOB_NAME" --resource-group "$GROUP" --workspace-name "$WORKSPACE" --download-path azure-download --output-name auditWhat does this command do?
This command downloads the named audit output from the tracked job. Azure writes the files into the local azure-download/ folder.
- Expand azure-download/ in your editor's file sidebar.
- Confirm that the folder contains metrics.json, slice_metrics.json, and MODEL_CARD.md.
That completes the portability proof. Your tested local workload now has a completed Azure job record and a matching set of downloaded audit artifacts.
Downloaded output is missing?
Confirm that the streamed job reached completion before downloading its output. The output name must remain audit because it matches the key in azure-job.yml.
Help me recover my Azure job output.
Secret mission
Test Wording Sensitivity
Aggregate metrics cannot show whether persuasive wording changes an individual prediction. This challenge adds paired neutral and promotional summaries to measure how strongly wording alone moves the model's approval probability.
Clean Up Your Resources
Clean Up Your Resources
Choose whether to keep the audit available, pause your work, or delete its generated resources entirely. Local files are free to retain.
Cost warning
Azure Machine Learning has no additional service charge. Azure compute, storage, networking, and dependent services can incur charges.
Workspace-related resources can continue accruing costs until deleted. Deletion is the cost-control path when you no longer need the cloud job record.
Resources you used:
- Local: The .venv/ environment containing the pinned Python runtime and project dependencies.
- Local: The artifacts/ folder containing metrics, slice analysis, the model card, and the counterfactual audit.
- Local: The azure-download/ folder containing outputs downloaded from the tracked cloud job.
- Azure: The rg-grant-risk-audit resource group containing the project cloud resources.
- Azure: The mlw-grant-risk-audit workspace inside the resource group.
- Azure: The dependent storage and monitoring resources inside the resource group.
- Azure: The completed serverless job record inside the resource group.
Keep everything running
No action is needed. Choose this option if you are still demonstrating the audit or extending its responsible evaluation workflow.
- Retain .venv/ for reproducible local runs.
- Retain artifacts/ for local evidence of the baseline, model evaluation, slice analysis, model card, and wording-sensitivity audit.
- Retain azure-download/ for the outputs downloaded from Azure.
- Keep rg-grant-risk-audit for access to the workspace and tracked job record.
- Review the cost attributed to rg-grant-risk-audit in Azure Cost Management.
Pause - I'll come back to this later
The audit commands have finished, so there is no local process to stop. Keep the local outputs while you take a break.
- Leave .venv/ in the project folder for your next local run.
- Leave artifacts/ in the project folder for later analysis.
- Leave azure-download/ in the project folder as local evidence of cloud execution.
- Retain rg-grant-risk-audit only when you need the Azure-hosted job history.
- Review the resource group's charges in Azure Cost Management during the pause.
- Choose the Delete option when you no longer need the cloud job record.
Delete - I don't want to use this again
Deleting the Azure resource group is irreversible, so use this option only when you have finished with the cloud demonstration. Your project source files remain in place because the local cleanup targets generated folders only.
- Return to the Linux terminal in the project folder from earlier.
- Remove the generated local resources by running this command:
rm -rf .venv artifacts azure-downloadWhat does this cleanup remove?
This removes the local virtual environment and generated audit outputs. It also removes the outputs downloaded from Azure.
Your source code, tests, configuration, and documentation remain available in the project folder.
- Delete the Azure resource group by running this command:
az group delete --name rg-grant-risk-audit --yes --no-waitWhat does this command remove?
The command starts deleting rg-grant-risk-audit. The --no-wait flag returns control while Azure continues the deletion.
The deletion removes mlw-grant-risk-audit plus the dependent resources and serverless job record inside the group. This action cannot be undone.
Nice Work!
Nice Work!
You did it! You built a tested Python NLP risk audit that evaluates historical grant outcomes without presenting its predictions as funding advice. The same workload now runs as a tracked serverless job in Azure Machine Learning.
You've learned how to:
- Exposed misleading accuracy by comparing the audit with a majority-class baseline that recorded zero recall for one class.
- Turned a TF-IDF logistic regression pipeline into a tested Python product with defensive schema detection, automated tests, reproducible metrics, text-length slices, and responsible-use documentation.
- Proved cloud portability by running the tested audit as a tracked Azure Machine Learning serverless command job with downloadable outputs.
- Secret Mission: Measured wording sensitivity across three paired Norwegian summaries. You interpreted the probability changes as a robustness risk without treating them as causal fairness evidence.
Ready to quiz yourself?