Build an ICC Difficulty Reviewer

Build a Streamlit app for transparent, human-reviewed ICC difficulty decisions.

Introduction

30 Second Summary

Experts can give the same test question different difficulty labels. Without a clear record of what shaped each choice, the final label becomes hard to explain.

In this project, you will build a local Streamlit reviewer that shows a subject matter expert an item characteristic curve before revealing distance evidence, item history, and an adjacent-class review flag. The app retains both judgments in a downloadable CSV audit log.

What You'll Build

Your finished reviewer keeps the evidence hidden until an expert locks in a first judgment, then reveals a clear path to a final decision and downloadable review record.

By the end of this project, you'll have:

  • A blind-first review flow where you select a demo item, inspect its curve, and record an independent difficulty label before seeing the model evidence.
  • A transparent evidence panel where you compare the closest centroid, all six functional distances, bounded historical evidence, and combined review scores.
  • An auditable review log that preserves the initial judgment, final decision, confidence, reason, distances, and evidence scores in downloadable reviews.csv.
  • Secret Mission: a blind-first consistency audit across all three demo items that identifies the case most in need of a second SME opinion.

Are there any prerequisites?

You need basic familiarity with Python functions and dictionaries, plus Python 3.12.8 and Visual Studio Code on Windows.

The guide supplies setup for every additional package.

Before We Start

Before any hands-on work begins, this step helps you commit to building a decision-support aid that keeps subject matter experts in control. Retaining the independent initial judgment alongside the assisted final judgment makes every reviewed decision auditable.

Set Up the Windows Project

A reproducible review system needs its own isolated environment. This protects the packages used by your other Python work.

In this step, you will prepare the project in Visual Studio Code with a PowerShell terminal. You will use a virtual environment to install Streamlit 1.66.0, NumPy 2.5.4, and pandas 3.0.6 for this project.

In this step, get ready to:
  • Create the local icc-review-system project folder.
  • Prepare an isolated environment with the pinned project packages.
  • Launch Streamlit's example app in your browser.
Create the project folder

Keeping the project on your Desktop gives it a predictable location. The integrated terminal keeps every setup command beside the files it affects.

  • Press the Windows key to open the search bar.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.

You will see the Visual Studio Code window. The editor now provides one place for your project files and terminal.

  • Press Ctrl+` to open the integrated terminal.
  • Select the terminal profile dropdown if the terminal tab does not say PowerShell.
  • Select PowerShell from the available profiles.
  • Move to your Desktop by running this command:
Set-Location -Path "$HOME\Desktop"

What does this command do?

  • PowerShell resolves $HOME to your Windows user folder.
  • The command changes the terminal location to its Desktop folder.
  • Create the project folder and enter it by running these commands:
New-Item -ItemType Directory -Path "icc-review-system"
Set-Location -Path "icc-review-system"

What do these commands do?

  • The first command creates the icc-review-system directory on your Desktop.
  • The second command makes that directory the terminal's current location.

You should see directory details after the folder is created. Your terminal prompt should now end with icc-review-system.

Folder already exists?

PowerShell may report that the folder already exists if you previously created it. Run only the location command from the second line to enter the existing folder.

Ask for help if the terminal still points somewhere else: help me enter my icc-review-system folder in PowerShell

  • Click File in the Visual Studio Code menu bar.
  • Click Open Folder.
  • Select Desktop in the folder picker.
  • Select the icc-review-system folder.
  • Confirm the folder selection.

Visual Studio Code reloads with icc-review-system in the Explorer sidebar. This confirms that future files land inside the correct folder.

Isolate and install the dependencies

A virtual environment gives this project its own package installation. The pinned dependency file makes the same versions available whenever the project is rebuilt.

  • Press Ctrl+` to return to the integrated PowerShell terminal.
  • Create the .venv virtual environment by running this command:
python -m venv .venv

What does this command do?

  • The venv module creates an isolated Python installation inside .venv.
  • The environment keeps this project's package versions separate from your other Python projects.

The command may finish without a success message. You should see a new .venv folder in the Explorer sidebar.

Missing the .venv folder?

Confirm that the terminal prompt ends with icc-review-system. A different location creates the environment in the wrong folder.

Ask for help if the command does not create the folder: help me create a Python virtual environment in PowerShell

  • Activate the virtual environment by running this command:
.venv\Scripts\Activate.ps1

What does activation change?

  • Activation points Python package commands at the environment inside .venv.
  • The terminal keeps using that environment until you deactivate it or close the terminal.

You should see (.venv) at the start of the PowerShell prompt. That prefix confirms the isolated environment is active.

Activation blocked?

Confirm that you are using a PowerShell terminal inside icc-review-system. A different shell uses a different activation command.

Ask for help if PowerShell prevents the activation script from running: help me activate .venv in Windows PowerShell

The package list records every dependency required by the reviewer. Exact pins prevent a future installation from silently choosing different package versions.

  • Select the Explorer view in the Visual Studio Code Activity Bar.
  • Click the New File... button.
  • Enter requirements.txt as the file name.
  • Press Enter to create the file.
  • Add the pinned dependencies by copying this content into requirements.txt:
streamlit==1.66.0
numpy==2.5.4
pandas==3.0.6

What does this file control?

  • Each line names one package required by the review app.
  • The double equals signs pin each package to the tested version.
  • These versions support the project's Python 3.12.8 environment.
  • Save requirements.txt by pressing Ctrl+S.
  • Confirm that requirements.txt appears inside icc-review-system in the Explorer sidebar.

File in the wrong folder?

The Explorer sidebar should show requirements.txt directly beneath icc-review-system. Move the file into that project folder before installing anything.

Ask for help if the file appears elsewhere: help me place requirements.txt in my project folder

✔️ Awesome, I've got everything!

Great. Your dependency file is saved with all three pinned packages.

ⓧ I'd like to double check the full code

Compare your complete requirements.txt file with this reference.

streamlit==1.66.0
numpy==2.5.4
pandas==3.0.6

What should match?

Your file should contain these three lines in this order. Each package name and version must match exactly.

The first installation can take a few minutes while Python downloads the packages. A short pause in the terminal is expected.

  • Install the pinned packages from requirements.txt by running this command:
python -m pip install -r requirements.txt

What does this command do?

  • Python runs the package installer from the active .venv environment.
  • The installer reads each exact package version from requirements.txt.
  • The downloaded packages remain isolated inside this project.

You should see package installation activity followed by a successful completion. The prompt should still begin with (.venv).

Package installation failed?

Confirm that the terminal shows the active (.venv) prefix. Also confirm that requirements.txt contains the three exact pinned lines.

Ask for help with the terminal output if the installation still stops: help me troubleshoot the pinned package installation

Verify Streamlit in the browser

Streamlit includes an example app that checks the installation through a real browser session. The command starts a local process that may open your browser automatically.

Before you run it, what do you expect a working Streamlit installation to open?

  • Launch Streamlit's example app by running this command:
python -m streamlit hello

What does this command do?

  • Python starts Streamlit from the active virtual environment.
  • The hello command launches Streamlit's included example app.
  • The local server sends the example interface to your browser.

Your browser should open Streamlit's example app. Seeing the interactive example confirms that the pinned installation can start a local web app.

Browser did not open?

Check the terminal for the local address printed by Streamlit. Copy that address into your browser if the page does not open automatically.

Confirm that the PowerShell prompt showed (.venv) before you started the app.

Ask for help if the terminal stops before the app loads: help me launch the Streamlit Hello app

That is your environment working end to end. Streamlit can now serve a local app from the isolated project environment.

  • Return to the Visual Studio Code PowerShell terminal.
  • Press Ctrl+C to stop the example app.

Your Windows project is isolated and ready. Next, you will turn this verified setup into the first ICC reviewer.

Build the First ICC Reviewer

Your isolated project environment is ready. The next challenge is turning difficulty evidence into something an SME can inspect.

In this step, you will build the first Streamlit reviewer around synthetic item characteristic curves. It captures an independent label before revealing one nearest-centroid comparison.

In this step, get ready to:
  • Build six ordered difficulty labels with synthetic item curves.
  • Measure the functional distance between each item curve and centroid.
  • Launch a reviewer that reveals the nearest-centroid comparison after an independent judgment.
Build the ICC foundation

An item characteristic curve maps ability to the probability of answering an item correctly. NumPy supplies the shared ability grid. It also calculates each synthetic 3PL curve.

  • Create logic.py inside the icc-review-system folder by using the file sidebar in Visual Studio Code.
  • Paste the ICC foundation into logic.py using this code:
import numpy as np

LABELS = [
    "Very easy",
    "Easy",
    "Moderately easy",
    "Moderately hard",
    "Hard",
    "Very hard",
]

THETA = np.linspace(-3.0, 3.0, num=121)
CENTROID_DIFFICULTIES = dict(zip(LABELS, [-2.5, -1.5, -0.5, 0.5, 1.5, 2.5]))


def make_icc(difficulty, discrimination=1.35, guessing=0.12):
    return guessing + (1.0 - guessing) / (
        1.0 + np.exp(-discrimination * (THETA - difficulty))
    )


CENTROIDS = {
    label: make_icc(difficulty)
    for label, difficulty in CENTROID_DIFFICULTIES.items()
}

What Does This Foundation Define?

  • The LABELS list preserves the six difficulty classes in their intended order.
  • THETA creates 121 ability points from -3.0 to 3.0.
  • make_icc() generates a synthetic curve from its difficulty, discrimination, and guessing values.
  • CENTROIDS creates one preserved reference curve for each ordered label.
  • Save logic.py.
  • Confirm the file sidebar lists logic.py inside icc-review-system.

Seeing a Syntax Marker?

Check that every label has matching quotation marks. Confirm that the closing brackets for LABELS and CENTROIDS are present.

Help me compare my ICC foundation with the target code.

The reviewer needs demo items with different curve shapes. Each item also carries a small classification history that stays attached to its curve.

  • Leave one blank line after the CENTROIDS dictionary.
  • Add the three demo items at the bottom of logic.py using this code:
ITEMS = {
    "ITEM-101": {
        "curve": make_icc(0.05, discrimination=1.20, guessing=0.14),
        "history": ["Moderately easy", "Moderately easy", "Moderately hard"],
    },
    "ITEM-102": {
        "curve": make_icc(1.30, discrimination=1.50, guessing=0.10),
        "history": ["Hard", "Hard", "Moderately hard", "Hard"],
    },
    "ITEM-103": {
        "curve": make_icc(-1.65, discrimination=0.95, guessing=0.18),
        "history": ["Easy", "Easy", "Very easy"],
    },
}

What Do the Demo Items Contain?

Each curve value stores an ICC generated with distinct parameters. Those differences give the reviewer three curve shapes to inspect.

Each history value preserves earlier classifications for that item. The current reviewer keeps this context attached to the item without displaying it.

  • Save logic.py.
  • Confirm the ITEMS dictionary contains ITEM-101, ITEM-102, and ITEM-103.

Are the Item Records Misaligned?

Check the indentation beneath each item ID. Every curve and history pair must remain inside its item dictionary.

Help me fix the ITEMS dictionary in logic.py.

Measure functional distance

An integrated functional distance compares two complete curves across the shared ability domain. A smaller result means the item curve stays closer to that centroid overall.

  • Leave two blank lines after the ITEMS dictionary.
  • Add the distance function at the bottom of logic.py using this code:
def functional_l2_distance(curve, centroid):
    squared_difference = (curve - centroid) ** 2
    return float(np.sqrt(np.trapezoid(squared_difference, x=THETA)))

How Does the Distance Work?

  • squared_difference measures the squared gap between the two curves at every theta point.
  • np.trapezoid integrates those gaps across the complete theta domain.
  • np.sqrt converts the integrated squared gap into the functional L2 distance.
  • Save logic.py.
  • Confirm the entire functional_l2_distance() function is aligned at the left edge of the editor.

Is the Function Nested Inside ITEMS?

Move functional_l2_distance() back to the left edge if it is indented. The function must sit outside the ITEMS dictionary.

Help me fix the placement of functional_l2_distance().

Create the visible reviewer

The first screen keeps the SME focused on the item curve before showing a comparison. pandas places the curve on a labeled theta index that Streamlit can chart.

  • Create app.py inside the icc-review-system folder by using the file sidebar.
  • Paste the first reviewer screen into app.py using this code:
import pandas as pd
import streamlit as st

from logic import CENTROIDS, ITEMS, LABELS, THETA, functional_l2_distance

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write("Record an independent eye-test judgment before viewing the nearest-centroid comparison.")

item_id = st.selectbox("Item to review", list(ITEMS.keys()))
item = ITEMS[item_id]

st.write("## Item characteristic curve")
item_curve_frame = pd.DataFrame({"Item ICC": item["curve"]}, index=THETA)
st.line_chart(
    item_curve_frame,
    x_label="Ability (theta)",
    y_label="Probability correct",
)

initial_label = st.selectbox("Difficulty from the curve alone", LABELS)

What Does the First Screen Show?

  • item_id tracks the item selected by the SME.
  • item_curve_frame pairs the selected curve with the theta index.
  • st.line_chart displays the item ICC before any centroid comparison appears.
  • initial_label captures the SME's classification from the curve alone.
  • Save app.py.

Before you run the reviewer, which curve controls do you expect to see on its first screen?

  • Launch the local reviewer from the existing PowerShell terminal by running this command:
python -m streamlit run app.py

What Does This Command Do?

The command starts app.py through the Streamlit module installed in the active virtual environment. Streamlit serves the reviewer locally in your browser.

Your first review screen is live. You should see the page title, an item selector, the selected ICC, and an independent difficulty selector.

Does the Reviewer Fail to Load?

Confirm the terminal path ends with icc-review-system. Check that app.py and logic.py appear in that folder.

Help me debug the first Streamlit reviewer screen.

The visible ICC now supports an independent eye-test classification. The next block calculates every centroid distance only when the SME reveals the comparison.

  • Return to app.py in the editor.
  • Add the comparison block at the bottom of app.py using this code:
if st.button("Record judgment and reveal comparison"):
    distances = {
        label: functional_l2_distance(item["curve"], centroid)
        for label, centroid in CENTROIDS.items()
    }
    nearest_centroid = min(distances, key=distances.get)
    st.write("Independent judgment recorded:", initial_label)
    st.write("Nearest centroid:", nearest_centroid)
    comparison_frame = pd.DataFrame(
        {
            "Item ICC": item["curve"],
            f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid],
        },
        index=THETA,
    )
    st.line_chart(
        comparison_frame,
        x_label="Ability (theta)",
        y_label="Probability correct",
    )

How Does the Comparison Work?

  • distances applies the functional distance calculation to all six centroids.
  • nearest_centroid selects the label with the smallest distance.
  • comparison_frame places the item curve beside the selected centroid curve.
  • st.line_chart overlays both curves so the SME can compare their shapes.
  • Save app.py.
  • Return to the browser running the reviewer.
  • Select ITEM-101 from the item selector.

You should see the ICC for ITEM-101 before any comparison chart is visible.

  • Select Moderately easy as the independent difficulty label.

Before you reveal the comparison, do you think one nearest-centroid label can show how close the next class is?

  • Click Record judgment and reveal comparison.

You should see the independent judgment text plus a nearest-centroid label. A second line chart overlays the item ICC with that centroid.

That is the first ICC reviewer working. The SME can now move from an isolated curve to one visible reference comparison.

The result also exposes the intended shortfall in this baseline.

Only one nearest class is visible. The runner-up distance remains hidden.

Does the Comparison Stay Hidden?

Confirm the new if st.button block starts at the left edge. Check that every line beneath it keeps four spaces of indentation.

Help me debug the hidden nearest-centroid comparison.

✔️ Awesome, I've got everything!

Your two Python files are saved. Keep the Streamlit process running while you compare your files with the complete references if needed.

ⓧ I'd like to double check the full code

Your complete logic.py should match this reference.

import numpy as np

LABELS = [
    "Very easy",
    "Easy",
    "Moderately easy",
    "Moderately hard",
    "Hard",
    "Very hard",
]

THETA = np.linspace(-3.0, 3.0, num=121)
CENTROID_DIFFICULTIES = dict(zip(LABELS, [-2.5, -1.5, -0.5, 0.5, 1.5, 2.5]))


def make_icc(difficulty, discrimination=1.35, guessing=0.12):
    return guessing + (1.0 - guessing) / (
        1.0 + np.exp(-discrimination * (THETA - difficulty))
    )


CENTROIDS = {
    label: make_icc(difficulty)
    for label, difficulty in CENTROID_DIFFICULTIES.items()
}

ITEMS = {
    "ITEM-101": {
        "curve": make_icc(0.05, discrimination=1.20, guessing=0.14),
        "history": ["Moderately easy", "Moderately easy", "Moderately hard"],
    },
    "ITEM-102": {
        "curve": make_icc(1.30, discrimination=1.50, guessing=0.10),
        "history": ["Hard", "Hard", "Moderately hard", "Hard"],
    },
    "ITEM-103": {
        "curve": make_icc(-1.65, discrimination=0.95, guessing=0.18),
        "history": ["Easy", "Easy", "Very easy"],
    },
}


def functional_l2_distance(curve, centroid):
    squared_difference = (curve - centroid) ** 2
    return float(np.sqrt(np.trapezoid(squared_difference, x=THETA)))

Your complete app.py should match this reference.

import pandas as pd
import streamlit as st

from logic import CENTROIDS, ITEMS, LABELS, THETA, functional_l2_distance

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write("Record an independent eye-test judgment before viewing the nearest-centroid comparison.")

item_id = st.selectbox("Item to review", list(ITEMS.keys()))
item = ITEMS[item_id]

st.write("## Item characteristic curve")
item_curve_frame = pd.DataFrame({"Item ICC": item["curve"]}, index=THETA)
st.line_chart(
    item_curve_frame,
    x_label="Ability (theta)",
    y_label="Probability correct",
)

initial_label = st.selectbox("Difficulty from the curve alone", LABELS)

if st.button("Record judgment and reveal comparison"):
    distances = {
        label: functional_l2_distance(item["curve"], centroid)
        for label, centroid in CENTROIDS.items()
    }
    nearest_centroid = min(distances, key=distances.get)
    st.write("Independent judgment recorded:", initial_label)
    st.write("Nearest centroid:", nearest_centroid)
    comparison_frame = pd.DataFrame(
        {
            "Item ICC": item["curve"],
            f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid],
        },
        index=THETA,
    )
    st.line_chart(
        comparison_frame,
        x_label="Ability (theta)",
        y_label="Probability correct",
    )

Your existing requirements.txt should remain unchanged.

streamlit==1.66.0
numpy==2.5.4
pandas==3.0.6

Your first ICC reviewer now turns a synthetic item curve into a visible nearest-centroid comparison. Next, you will expose the wider evidence that this single-label result leaves hidden.

Expose Adjacent-Class Ambiguity

Your Streamlit reviewer already compares an item characteristic curve with its nearest centroid. That single result hides how close the next difficulty class may be.

This step surfaces all six functional distances. It also adds bounded historical evidence plus a measured flag for close adjacent classes.

In this step, get ready to:
  • Convert the six distances into normalized decision evidence.
  • Blend a Laplace-smoothed historical prior with the distance evidence.
  • Flag close adjacent classes for SME confirmation.
Build normalized distance evidence

A nearest-centroid label discards the size of every other distance. Normalized distance evidence preserves the relative strength of all six labels without claiming statistical calibration.

  • Switch back to logic.py in the editor.
  • Find the existing import at the top of the file:
import numpy as np

What Are You Replacing?

This import gives the existing ICC functions access to NumPy. The new evidence helpers also need exponential decay plus a robust distance scale.

  • Replace that import with the following import group:
from math import exp
from statistics import median

import numpy as np

Why Add These Imports?

  • The median distance supplies a scale based on the middle of all six distances.
  • The exp function converts smaller distances into stronger raw evidence.
  • NumPy continues to generate curves plus integrated functional distances.
  • Add the new evidence helper below functional_l2_distance() by copying this code:
def distance_evidence(curve):
    distances = {
        label: functional_l2_distance(curve, centroid)
        for label, centroid in CENTROIDS.items()
    }
    scale = max(median(distances.values()), 1e-9)
    raw_scores = {
        label: exp(-distance / scale)
        for label, distance in distances.items()
    }
    total = sum(raw_scores.values())
    normalized = {
        label: score / total
        for label, score in raw_scores.items()
    }
    return distances, normalized

What Does This Code Do?

  • The first dictionary calculates a functional L2 distance for every preserved centroid.
  • The median distance creates a stable scale for the exponential transformation.
  • The final dictionary normalizes the raw scores so they sum to one.
  • These values remain decision evidence. They are not calibrated class probabilities.
  • Save logic.py.
  • Return to the running reviewer in your browser.
  • Confirm the item curve still loads after Streamlit reruns the app.

The curve loading successfully confirms that the new helper imports without interrupting the existing reviewer.

Did the App Stop Loading?

Check that the three imports remain at the top of logic.py. Make sure distance_evidence() starts after the complete functional_l2_distance() function.

Compare the indentation inside each dictionary comprehension. Python requires every nested line to stay aligned.

Help me fix the new distance evidence helper in logic.py.

Blend bounded historical evidence

A historical prior captures the labels previously assigned to an item. Laplace smoothing gives every difficulty label an initial count before the recorded history is added.

The fixed history weight of 0.25 keeps that evidence bounded. This value is a demonstration parameter that needs validation before operational use.

  • Add the historical prior helper below distance_evidence() in logic.py by copying this code:
def historical_prior(history):
    counts = {label: 1.0 for label in LABELS}
    for label in history:
        counts[label] += 1.0
    total = sum(counts.values())
    return {label: count / total for label, count in counts.items()}

How Does Laplace Smoothing Help?

  • Every label starts with a count of 1.0.
  • Each historical classification increments its matching label.
  • Dividing by the total converts the counts into a complete prior distribution.
  • Save logic.py.
  • Return to the browser after Streamlit reruns the reviewer.
  • Confirm the item selector still lists all three demo items.

The three item IDs confirm that the existing ITEMS data still loads beside the new historical helper.

Are the Demo Items Missing?

Check that historical_prior() sits below the closing line of distance_evidence(). A nested function would prevent the app from importing it later.

Help me restore the item selector after adding historical_prior.

  • Add the blending helper below historical_prior() by copying this code:
def blend_evidence(distance_scores, history_scores, history_weight=0.25):
    return {
        label: (1.0 - history_weight) * distance_scores[label]
        + history_weight * history_scores[label]
        for label in LABELS
    }

What Does the Blend Represent?

The blend assigns 0.75 of each combined score to the curve-based evidence. It assigns the remaining 0.25 to item history.

The result is a transparent review score for each ordered label. It does not become a calibrated probability through this calculation.

  • Save logic.py.
  • Return to the browser after the reviewer reruns.
  • Confirm the selected item still displays its ICC.

Your logic layer can now calculate curve-based evidence plus bounded historical evidence without breaking the visible ICC.

Is the ICC No Longer Visible?

Check the closing brace in blend_evidence(). Make sure the continuation line beginning with + history_weight stays inside the dictionary value expression.

Help me correct the blend_evidence function in logic.py.

Flag close adjacent classes

The combined scores need a decision rule that respects the order of the six classes. The review flag activates only when the two leading labels are adjacent.

The top-two margin must also fall below 0.08. This threshold is another demonstration parameter for routing ambiguous items into human review.

  • Add the review summary helper below blend_evidence() in logic.py by copying this code:
def review_summary(scores, margin_threshold=0.08):
    ranked = sorted(LABELS, key=scores.get, reverse=True)
    top_label, second_label = ranked[:2]
    margin = scores[top_label] - scores[second_label]
    adjacent = abs(LABELS.index(top_label) - LABELS.index(second_label)) == 1
    needs_review = adjacent and margin < margin_threshold
    return needs_review, margin, top_label, second_label

How Is Ambiguity Detected?

  • The sorted label list identifies the recommendation plus its runner-up.
  • The margin measures the score gap between those two labels.
  • Their positions in LABELS determine whether the classes are adjacent.
  • Both conditions must be true before needs_review becomes active.
  • Save logic.py.
  • Return to the browser after Streamlit reruns the reviewer.
  • Confirm the current ICC reviewer still loads without a Python error.

Your four evidence helpers now load together. The next edits connect them to the visible reviewer.

Seeing an Error After the Review Helper?

Check that ranked[:2] uses a colon inside the slice. Confirm that LABELS.index appears on both sides of the subtraction.

Help me debug review_summary in logic.py.

The interface needs a small staged rewrite so each saved version stays runnable. The first version loads every helper before the evidence panel is attached.

  • Switch back to app.py in the editor.
  • Select its contents with Ctrl+A.
  • Replace the selected contents with this first runnable section:
import pandas as pd
import streamlit as st

from logic import (
    CENTROIDS,
    ITEMS,
    LABELS,
    THETA,
    blend_evidence,
    distance_evidence,
    historical_prior,
    review_summary,
)

HISTORY_WEIGHT = 0.25

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write("Record an independent eye-test judgment first. The app then reveals functional and historical decision evidence.")

item_id = st.selectbox("Item to review", list(ITEMS.keys()))
item = ITEMS[item_id]

st.write("## Item characteristic curve")
item_curve_frame = pd.DataFrame({"Item ICC": item["curve"]}, index=THETA)
st.line_chart(item_curve_frame, x_label="Ability (theta)", y_label="Probability correct")

initial_label = st.selectbox("Difficulty from the curve alone", LABELS)

What Does This Section Prepare?

  • The grouped import brings all four evidence helpers into the app.
  • The HISTORY_WEIGHT constant makes the bounded influence visible near the top of the file.
  • The item selector plus ICC chart preserve the blind first look at the item.
  • Save app.py.
  • Return to the browser after Streamlit reruns the app.
  • Confirm the updated introduction appears above the item selector.

You should see the ICC plus the independent difficulty selector. The updated introduction now promises functional plus historical decision evidence.

Does the App Show an Import Error?

Confirm that all four helper names in app.py match their definitions in logic.py. Check that the import block closes with a parenthesis.

Help me resolve the evidence helper imports in app.py.

  • Add the comparison block below the initial_label line by copying this code:
if st.button("Record judgment and reveal evidence"):
    distances, distance_scores = distance_evidence(item["curve"])
    history_scores = historical_prior(item["history"])
    combined_scores = blend_evidence(distance_scores, history_scores, history_weight=HISTORY_WEIGHT)
    needs_review, margin, recommendation, runner_up = review_summary(combined_scores)
    nearest_centroid = min(distances, key=distances.get)
    st.write("Independent judgment recorded:", initial_label)
    comparison_frame = pd.DataFrame({"Item ICC": item["curve"], f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid]}, index=THETA)
    st.line_chart(comparison_frame, x_label="Ability (theta)", y_label="Probability correct")

What Happens After the Judgment?

  • The four helpers calculate every evidence layer only after the SME records an independent label.
  • The nearest centroid remains available as a visual comparison.
  • The recommendation plus runner-up remain available for the evidence panel.
  • Save app.py.
  • Return to the reviewer in your browser.
  • Click Record judgment and reveal evidence.

You should see the independent label you selected plus the nearest-centroid comparison chart. This restores the original visible comparison on top of the new evidence calculations.

Does the Button Show an Error?

Check that each line inside the if block has four spaces of indentation. Confirm that the four values returned by review_summary() use the same order shown above.

Help me debug the evidence calculation block in app.py.

  • Add the complete decision evidence panel directly below the comparison chart code:
    st.write("## Decision evidence")
    st.write("These are normalized review scores, not statistically calibrated class probabilities.")
    evidence_frame = pd.DataFrame({"Distance evidence": [distance_scores[label] for label in LABELS], "Historical prior": [history_scores[label] for label in LABELS], "Combined review score": [combined_scores[label] for label in LABELS]}, index=LABELS)
    st.bar_chart(evidence_frame)
    distance_frame = pd.DataFrame({"Functional L2 distance": [distances[label] for label in LABELS]}, index=LABELS)
    st.write(distance_frame.round(4))
    st.write("Recommended class:", recommendation)
    st.write("Runner-up class:", runner_up)
    st.write("Top-two score margin:", round(margin, 4))
    if needs_review:
        st.write("Review flag: close adjacent classes require SME confirmation.")
    else:
        st.write("Review flag: no close adjacent-class warning at the demonstration threshold.")

How Does the Evidence Panel Help?

  • The bar chart compares distance evidence with the historical prior plus the combined score.
  • The distance table preserves the underlying integrated measurements for all six labels.
  • The recommendation plus runner-up expose the two strongest classes.
  • The margin plus review flag show whether close adjacent leaders require SME confirmation.

Before you test the completed reviewer, do you think ITEM-101 will produce a clear winner or a close adjacent-class decision?

  • Save app.py.
  • Return to the reviewer after Streamlit reruns the app.
  • Select ITEM-101 from the item selector.
  • Choose an independent difficulty label from the curve alone.
  • Click Record judgment and reveal evidence.

You should see the nearest-centroid comparison plus a bar chart with three evidence series. Below it, you should see all six functional L2 distances.

You should also see the recommended class, the runner-up class, the top-two margin, plus a review flag or a message confirming that the demonstration threshold did not trigger.

Why These Scores Need Careful Labels

The distance normalization plus weighted blend are transparent teaching constructions. The evidence panel explicitly describes their output as review scores.

Operational probability claims would require a larger reviewed dataset plus a separate calibration process.

Is the Evidence Panel Missing?

Make sure every evidence-panel line stays inside the outer button block. The nested review flag needs eight spaces before each st.write call.

Confirm that you clicked the updated button after the latest Streamlit rerun. The panel appears only after that click.

Help me fix the missing evidence panel in app.py.

✔️ Awesome, I've got everything!

Great work. Your reviewer now exposes the full evidence behind an adjacent difficulty decision.

Double check that you saved both logic.py plus app.py before moving on.

ⓧ I'd like to double check the full code

Compare your complete logic.py with this reference:

from math import exp
from statistics import median

import numpy as np

LABELS = [
    "Very easy",
    "Easy",
    "Moderately easy",
    "Moderately hard",
    "Hard",
    "Very hard",
]

THETA = np.linspace(-3.0, 3.0, num=121)
CENTROID_DIFFICULTIES = dict(zip(LABELS, [-2.5, -1.5, -0.5, 0.5, 1.5, 2.5]))


def make_icc(difficulty, discrimination=1.35, guessing=0.12):
    return guessing + (1.0 - guessing) / (
        1.0 + np.exp(-discrimination * (THETA - difficulty))
    )


CENTROIDS = {
    label: make_icc(difficulty)
    for label, difficulty in CENTROID_DIFFICULTIES.items()
}

ITEMS = {
    "ITEM-101": {
        "curve": make_icc(0.05, discrimination=1.20, guessing=0.14),
        "history": ["Moderately easy", "Moderately easy", "Moderately hard"],
    },
    "ITEM-102": {
        "curve": make_icc(1.30, discrimination=1.50, guessing=0.10),
        "history": ["Hard", "Hard", "Moderately hard", "Hard"],
    },
    "ITEM-103": {
        "curve": make_icc(-1.65, discrimination=0.95, guessing=0.18),
        "history": ["Easy", "Easy", "Very easy"],
    },
}


def functional_l2_distance(curve, centroid):
    squared_difference = (curve - centroid) ** 2
    return float(np.sqrt(np.trapezoid(squared_difference, x=THETA)))


def distance_evidence(curve):
    distances = {
        label: functional_l2_distance(curve, centroid)
        for label, centroid in CENTROIDS.items()
    }
    scale = max(median(distances.values()), 1e-9)
    raw_scores = {
        label: exp(-distance / scale)
        for label, distance in distances.items()
    }
    total = sum(raw_scores.values())
    normalized = {
        label: score / total
        for label, score in raw_scores.items()
    }
    return distances, normalized


def historical_prior(history):
    counts = {label: 1.0 for label in LABELS}
    for label in history:
        counts[label] += 1.0
    total = sum(counts.values())
    return {label: count / total for label, count in counts.items()}


def blend_evidence(distance_scores, history_scores, history_weight=0.25):
    return {
        label: (1.0 - history_weight) * distance_scores[label]
        + history_weight * history_scores[label]
        for label in LABELS
    }


def review_summary(scores, margin_threshold=0.08):
    ranked = sorted(LABELS, key=scores.get, reverse=True)
    top_label, second_label = ranked[:2]
    margin = scores[top_label] - scores[second_label]
    adjacent = abs(LABELS.index(top_label) - LABELS.index(second_label)) == 1
    needs_review = adjacent and margin < margin_threshold
    return needs_review, margin, top_label, second_label

This file contains the six-label ICC data plus every evidence helper used by the reviewer.

Compare your complete app.py with this reference:

import pandas as pd
import streamlit as st

from logic import (
    CENTROIDS,
    ITEMS,
    LABELS,
    THETA,
    blend_evidence,
    distance_evidence,
    historical_prior,
    review_summary,
)

HISTORY_WEIGHT = 0.25

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write("Record an independent eye-test judgment first. The app then reveals functional and historical decision evidence.")

item_id = st.selectbox("Item to review", list(ITEMS.keys()))
item = ITEMS[item_id]

st.write("## Item characteristic curve")
item_curve_frame = pd.DataFrame({"Item ICC": item["curve"]}, index=THETA)
st.line_chart(item_curve_frame, x_label="Ability (theta)", y_label="Probability correct")

initial_label = st.selectbox("Difficulty from the curve alone", LABELS)

if st.button("Record judgment and reveal evidence"):
    distances, distance_scores = distance_evidence(item["curve"])
    history_scores = historical_prior(item["history"])
    combined_scores = blend_evidence(distance_scores, history_scores, history_weight=HISTORY_WEIGHT)
    needs_review, margin, recommendation, runner_up = review_summary(combined_scores)
    nearest_centroid = min(distances, key=distances.get)
    st.write("Independent judgment recorded:", initial_label)
    comparison_frame = pd.DataFrame({"Item ICC": item["curve"], f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid]}, index=THETA)
    st.line_chart(comparison_frame, x_label="Ability (theta)", y_label="Probability correct")
    st.write("## Decision evidence")
    st.write("These are normalized review scores, not statistically calibrated class probabilities.")
    evidence_frame = pd.DataFrame({"Distance evidence": [distance_scores[label] for label in LABELS], "Historical prior": [history_scores[label] for label in LABELS], "Combined review score": [combined_scores[label] for label in LABELS]}, index=LABELS)
    st.bar_chart(evidence_frame)
    distance_frame = pd.DataFrame({"Functional L2 distance": [distances[label] for label in LABELS]}, index=LABELS)
    st.write(distance_frame.round(4))
    st.write("Recommended class:", recommendation)
    st.write("Runner-up class:", runner_up)
    st.write("Top-two score margin:", round(margin, 4))
    if needs_review:
        st.write("Review flag: close adjacent classes require SME confirmation.")
    else:
        st.write("Review flag: no close adjacent-class warning at the demonstration threshold.")

This file keeps the curve visible before the click. It reveals the comparison plus the complete evidence panel after the independent judgment.

Your unchanged requirements.txt should still contain these pinned packages:

streamlit==1.66.0
numpy==2.5.4
pandas==3.0.6

These pins preserve the package versions installed in your active virtual environment.

That is the hidden ambiguity exposed. Next, you will preserve the independent judgment before capturing a separate final reviewed decision in the audit log.

Add the Blind SME Review Loop

Your evidence panels now expose every distance score. They also show how history affects the recommendation.

Quantitative evidence is most useful when its influence on an SME is observable. This step records the independent judgment before revealing that evidence. It then preserves the assisted final decision in an auditable log.

In this step, get ready to:
  • Keep the independent label and confidence in Streamlit Session State.
  • Append each final reviewed decision to a CSV audit log.
  • Download the completed review log from the app.
Lock the independent judgment

Streamlit reruns your script after each widget interaction. Streamlit Session State preserves a judgment across those reruns. The item ID in each state key keeps the three demo items independent.

  • In app.py from earlier, replace the first two import lines with this import group:
from datetime import datetime, timezone
from pathlib import Path

import pandas as pd
import streamlit as st

Why add these imports?

  • The datetime imports provide a UTC timestamp for every saved review.
  • Path gives the audit log one reusable file location.
  • Find the block that starts with HISTORY_WEIGHT = 0.25.
  • Replace that block through the current one-line reviewer description with this updated configuration:
REVIEW_PATH = Path("reviews.csv")
HISTORY_WEIGHT = 0.25

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write(
    "Record an independent eye-test judgment first. The app then reveals "
    "functional and historical evidence for a final reviewed decision."
)

What changes here?

  • REVIEW_PATH identifies the local file that receives each audit row.
  • The page description now states the complete independent-to-assisted workflow.
  • Save app.py.
  • Switch back to the browser page from earlier.

You should see the updated description mention a final reviewed decision. The ICC remains visible below it.

Still seeing the old description?

Confirm that app.py is saved. Refresh the existing browser page if the running app has not rerun.

Help me check why my saved Streamlit page text is not updating.

The current button reveals evidence for one rerun only. The next block gives each item a persistent state key. It also captures confidence alongside the independent label.

  • Find the existing initial_label = st.selectbox("Difficulty from the curve alone", LABELS) line.
  • Replace that line with this state-backed control block:
state_key = f"independent_judgment_{item_id}"
locked_judgment = st.session_state.get(state_key)

if locked_judgment is None:
    st.write("## Independent SME judgment")
    initial_label = st.selectbox(
        "Difficulty from the curve alone",
        LABELS,
        key=f"initial_label_{item_id}",
    )
    initial_confidence = st.slider(
        "Confidence in the independent judgment",
        min_value=1,
        max_value=5,
        value=3,
        key=f"initial_confidence_{item_id}",
    )
    if st.button("Record judgment and reveal evidence", key=f"reveal_{item_id}"):
        locked_judgment = {
            "label": initial_label,
            "confidence": initial_confidence,
        }
        st.session_state[state_key] = locked_judgment

How does the judgment lock work?

  • state_key includes the selected item ID. Each item therefore receives a separate independent judgment.
  • locked_judgment reads any existing label and confidence from Session State.
  • The controls appear only while the selected item has no stored independent judgment.
  • Find the old unkeyed button block below the new state controls.
  • Select from the old button line through the old independent-judgment message.
  • Replace that selection with this evidence gate:
if locked_judgment is None:
    st.write("The model evidence remains hidden until the independent judgment is recorded.")
else:
    st.write(
        "Independent judgment recorded:",
        locked_judgment["label"],
        f"with confidence {locked_judgment['confidence']}/5.",
    )

    distances, distance_scores = distance_evidence(item["curve"])
    history_scores = historical_prior(item["history"])
    combined_scores = blend_evidence(
        distance_scores,
        history_scores,
        history_weight=HISTORY_WEIGHT,
    )
    needs_review, margin, recommendation, runner_up = review_summary(combined_scores)
    nearest_centroid = min(distances, key=distances.get)

    st.write("## Curve comparison")

What does the evidence gate protect?

  • The first branch displays a clear message while the evidence is hidden.
  • The second branch calculates the recommendation only after an independent judgment exists.
  • The recorded confidence remains attached to the independent label throughout the review.
  • Find the one-line comparison_frame definition.
  • Replace that definition through its following one-line chart call with this formatted comparison block:
    comparison_frame = pd.DataFrame(
        {
            "Item ICC": item["curve"],
            f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid],
        },
        index=THETA,
    )
    st.line_chart(
        comparison_frame,
        x_label="Ability (theta)",
        y_label="Probability correct",
    )

What stays inside the gate?

The comparison chart remains indented inside the evidence branch. The item curve stays visible before judgment while the centroid remains hidden.

  • Find the one-line evidence_frame definition.
  • Replace that definition through the distance-table output with this evidence block:
    evidence_frame = pd.DataFrame(
        {
            "Distance evidence": [distance_scores[label] for label in LABELS],
            "Historical prior": [history_scores[label] for label in LABELS],
            "Combined review score": [combined_scores[label] for label in LABELS],
        },
        index=LABELS,
    )
    st.bar_chart(evidence_frame)

    distance_frame = pd.DataFrame(
        {"Functional L2 distance": [distances[label] for label in LABELS]},
        index=LABELS,
    )
    st.write(distance_frame.round(4))
    st.write("Historical classifications:", ", ".join(item["history"]))

Why show the history list?

The bar chart shows the historical prior numerically. The added classification list exposes the source labels behind that prior.

  • Save app.py.
  • Switch back to the existing browser page.
  • Select ITEM-101 in the item selector.
  • Choose an independent difficulty label from the visible curve.
  • Set the confidence slider to a value from 1 through 5.
  • Click Record judgment and reveal evidence.

You should see the stored label with its confidence. You should also see the centroid comparison and every evidence panel remain visible after the rerun.

Does the evidence disappear again?

Check that both Session State lines use state_key. Confirm that the second condition reads if locked_judgment is None:.

Help me debug why my Streamlit evidence does not remain visible after recording a judgment.

Append the reviewed decision

An auditable review needs both decisions. The saved row also needs the evidence that informed the final label. A CSV keeps that record readable without adding a database.

  • Locate the adjacent-class review flag at the end of the evidence branch.
  • Add this final-decision interface below the complete review-flag condition:
    st.write("## Final reviewed decision")
    final_label = st.selectbox(
        "Final difficulty classification",
        LABELS,
        index=LABELS.index(recommendation),
        key=f"final_label_{item_id}",
    )
    reason = st.text_area(
        "Reason for confirming or changing the recommendation",
        key=f"reason_{item_id}",
    )

What does the SME decide here?

  • The final-label control starts with the combined-evidence recommendation.
  • The reason field records why the SME confirmed or changed that recommendation.
  • The independent judgment remains locked above these assisted controls.
  • Save app.py.
  • Switch back to the browser page.

You should see a final classification selector below the review flag. You should also see a reason field for the SME's decision.

  • Add this save handler directly below the reason control:
    if st.button("Save reviewed decision", key=f"save_{item_id}"):
        record = {
            "timestamp_utc": datetime.now(timezone.utc).isoformat(),
            "item_id": item_id,
            "independent_label": locked_judgment["label"],
            "independent_confidence": locked_judgment["confidence"],
            "history_weight": HISTORY_WEIGHT,
            "recommended_label": recommendation,
            "runner_up_label": runner_up,
            "top_two_margin": margin,
            "review_flag": needs_review,
            "final_label": final_label,
            "reason": reason,
        }

What enters the base audit row?

  • The UTC timestamp places each decision on a consistent time standard.
  • The row retains the independent label and confidence separately from the final label.
  • The recommendation and review flag capture what the model evidence showed at decision time.
  • Add this evidence expansion and file-writing block directly below the record dictionary:
        record.update({
            f"distance_{label.lower().replace(' ', '_')}": distances[label]
            for label in LABELS
        })
        record.update({
            f"score_{label.lower().replace(' ', '_')}": combined_scores[label]
            for label in LABELS
        })
        pd.DataFrame([record]).to_csv(
            REVIEW_PATH,
            mode="a",
            header=not REVIEW_PATH.exists(),
            index=False,
        )
        st.write("Saved the reviewed decision to reviews.csv.")

How is the audit row completed?

  • The first update adds one functional-distance field for each difficulty label.
  • The second update adds one combined-score field for each difficulty label.
  • Append mode preserves earlier reviews while adding the current row.
  • The header is written only when the audit file does not already exist.
  • Save app.py.
  • Switch back to the browser page.
  • Select a final difficulty classification.
  • Enter The curve and adjacent-class evidence support this decision. in the reason field.
  • Click Save reviewed decision.

You should see confirmation that the reviewed decision was saved to reviews.csv. The file now contains the independent judgment and the assisted final decision.

Was the review not saved?

Check the terminal for an indentation error around the two record.update blocks. Both updates and the CSV write must remain inside the save-button condition.

Help me debug why my Streamlit review is not being appended to reviews.csv.

Download and verify the audit log

The local file provides persistence across app reruns. A browser download makes the same audit trail portable for analysis or second-opinion review.

  • Move to the left margin after the complete evidence branch in app.py.
  • Add this download control at the end of the file:
if REVIEW_PATH.exists():
    st.download_button(
        "Download review log",
        data=REVIEW_PATH.read_bytes(),
        file_name="reviews.csv",
        mime="text/csv",
    )

When does the download appear?

The file-existence check hides the download control until at least one review has been saved. The button then reads the current audit log without exposing any credentials.

  • Save app.py.
  • Switch back to the existing browser page.

Before you complete the final check, do you expect the audit log to contain the eye-test label or both SME decisions?

  • Select ITEM-101 if another item is currently selected.
  • Record an independent label if this item has no locked judgment.
  • Choose the final reviewed label.
  • Enter a reason for the final decision.
  • Click Save reviewed decision.
  • Click Download review log.

You should see the save confirmation and the download control. The downloaded reviews.csv contains the item ID, both judgments, confidence, reason, evidence summary, six distances, and six combined scores.

Is the download control missing?

Save a reviewed decision before checking for the control. Confirm that reviews.csv exists beside app.py.

Help me find why my Streamlit download control does not appear after reviews.csv is created.

✔️ Awesome, I've got everything!

Great work. Your reviewer now preserves an independent judgment, records the assisted decision, and exports the complete audit trail.

ⓧ I'd like to double check the full code

Compare your complete app.py with this reference. The indentation controls which evidence remains hidden before the independent judgment.

from datetime import datetime, timezone
from pathlib import Path

import pandas as pd
import streamlit as st

from logic import (
    CENTROIDS,
    ITEMS,
    LABELS,
    THETA,
    blend_evidence,
    distance_evidence,
    historical_prior,
    review_summary,
)

REVIEW_PATH = Path("reviews.csv")
HISTORY_WEIGHT = 0.25

st.write("# Human-in-the-Loop ICC Difficulty Review")
st.write(
    "Record an independent eye-test judgment first. The app then reveals "
    "functional and historical evidence for a final reviewed decision."
)

item_id = st.selectbox("Item to review", list(ITEMS.keys()))
item = ITEMS[item_id]

st.write("## Item characteristic curve")
item_curve_frame = pd.DataFrame({"Item ICC": item["curve"]}, index=THETA)
st.line_chart(
    item_curve_frame,
    x_label="Ability (theta)",
    y_label="Probability correct",
)

state_key = f"independent_judgment_{item_id}"
locked_judgment = st.session_state.get(state_key)

if locked_judgment is None:
    st.write("## Independent SME judgment")
    initial_label = st.selectbox(
        "Difficulty from the curve alone",
        LABELS,
        key=f"initial_label_{item_id}",
    )
    initial_confidence = st.slider(
        "Confidence in the independent judgment",
        min_value=1,
        max_value=5,
        value=3,
        key=f"initial_confidence_{item_id}",
    )
    if st.button("Record judgment and reveal evidence", key=f"reveal_{item_id}"):
        locked_judgment = {
            "label": initial_label,
            "confidence": initial_confidence,
        }
        st.session_state[state_key] = locked_judgment

if locked_judgment is None:
    st.write("The model evidence remains hidden until the independent judgment is recorded.")
else:
    st.write(
        "Independent judgment recorded:",
        locked_judgment["label"],
        f"with confidence {locked_judgment['confidence']}/5.",
    )

    distances, distance_scores = distance_evidence(item["curve"])
    history_scores = historical_prior(item["history"])
    combined_scores = blend_evidence(
        distance_scores,
        history_scores,
        history_weight=HISTORY_WEIGHT,
    )
    needs_review, margin, recommendation, runner_up = review_summary(combined_scores)
    nearest_centroid = min(distances, key=distances.get)

    st.write("## Curve comparison")
    comparison_frame = pd.DataFrame(
        {
            "Item ICC": item["curve"],
            f"{nearest_centroid} centroid": CENTROIDS[nearest_centroid],
        },
        index=THETA,
    )
    st.line_chart(
        comparison_frame,
        x_label="Ability (theta)",
        y_label="Probability correct",
    )

    st.write("## Decision evidence")
    st.write(
        "These are normalized review scores, not statistically calibrated class probabilities."
    )
    evidence_frame = pd.DataFrame(
        {
            "Distance evidence": [distance_scores[label] for label in LABELS],
            "Historical prior": [history_scores[label] for label in LABELS],
            "Combined review score": [combined_scores[label] for label in LABELS],
        },
        index=LABELS,
    )
    st.bar_chart(evidence_frame)

    distance_frame = pd.DataFrame(
        {"Functional L2 distance": [distances[label] for label in LABELS]},
        index=LABELS,
    )
    st.write(distance_frame.round(4))
    st.write("Historical classifications:", ", ".join(item["history"]))
    st.write("Recommended class:", recommendation)
    st.write("Runner-up class:", runner_up)
    st.write("Top-two score margin:", round(margin, 4))

    if needs_review:
        st.write("Review flag: close adjacent classes require SME confirmation.")
    else:
        st.write("Review flag: no close adjacent-class warning at the demonstration threshold.")

    st.write("## Final reviewed decision")
    final_label = st.selectbox(
        "Final difficulty classification",
        LABELS,
        index=LABELS.index(recommendation),
        key=f"final_label_{item_id}",
    )
    reason = st.text_area(
        "Reason for confirming or changing the recommendation",
        key=f"reason_{item_id}",
    )

    if st.button("Save reviewed decision", key=f"save_{item_id}"):
        record = {
            "timestamp_utc": datetime.now(timezone.utc).isoformat(),
            "item_id": item_id,
            "independent_label": locked_judgment["label"],
            "independent_confidence": locked_judgment["confidence"],
            "history_weight": HISTORY_WEIGHT,
            "recommended_label": recommendation,
            "runner_up_label": runner_up,
            "top_two_margin": margin,
            "review_flag": needs_review,
            "final_label": final_label,
            "reason": reason,
        }
        record.update({
            f"distance_{label.lower().replace(' ', '_')}": distances[label]
            for label in LABELS
        })
        record.update({
            f"score_{label.lower().replace(' ', '_')}": combined_scores[label]
            for label in LABELS
        })
        pd.DataFrame([record]).to_csv(
            REVIEW_PATH,
            mode="a",
            header=not REVIEW_PATH.exists(),
            index=False,
        )
        st.write("Saved the reviewed decision to reviews.csv.")

if REVIEW_PATH.exists():
    st.download_button(
        "Download review log",
        data=REVIEW_PATH.read_bytes(),
        file_name="reviews.csv",
        mime="text/csv",
    )

This reference matches the completed review app. Your existing logic.py and requirements.txt remain unchanged.

That completes the review loop. Your app now measures the shift from an independent eye test to an evidence-assisted final decision.

Secret mission

Run a Blinded Consistency Audit

Complete blind-first reviews for all three demo items. Compare the resulting audit trail to identify the first item that warrants a second SME opinion.

Clean Up Your Resources

Clean Up Your Resources

Everything you built runs on your Windows computer. There are no ongoing cloud or API costs.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The icc-review-system folder containing all local project resources.
  • The local source files app.py, logic.py, and requirements.txt.
  • The .venv local virtual environment.
  • The reviews.csv local audit log.
  • The Streamlit process running app.py.

Keep everything running

No action needed. Choose this if you want to keep reviewing items or inspect your audit log.

  • Leave the Streamlit process running in the PowerShell terminal.
  • Keep the icc-review-system folder unchanged.
  • Reactivate the saved environment later by running .venv\Scripts\Activate.ps1 from the icc-review-system folder.

Pause - I'll come back to this later

Pausing stops the local app. Your project files remain ready for another review session.

  • Switch back to the PowerShell terminal where Streamlit is running.
  • Press Ctrl+C to stop the app.
  • Run deactivate to leave the virtual environment.

Your prototype is safely paused. All files remain inside icc-review-system.

Delete - I don't want to use this again

Deleting the folder is permanent. A copy of reviews.csv preserves your audit trail after cleanup.

  • Switch back to the PowerShell terminal where Streamlit is running.
  • Press Ctrl+C to stop the app.
  • Run deactivate to leave the virtual environment.
  • Copy reviews.csv to a location outside icc-review-system if you want to retain the audit trail.
  • Close Visual Studio Code.
  • Navigate to the Windows folder that contains icc-review-system.
  • Delete the icc-review-system folder.
  • Check the same location to confirm the folder was removed.

Cleanup complete. You should no longer see icc-review-system in its former location.

Nice Work!

Nice Work!

Excellent work! Your local Streamlit reviewer now turns each item characteristic curve into a transparent difficulty review across six ordered labels. Your completed CSV audit log preserves every stage of each decision.

You've learned how to:

  • Build a blind-first review workflow that displays each item curve before revealing its nearest preserved centroid.
  • Calculate integrated functional distance across six difficulty classes. Blend bounded historical evidence into normalized review scores without presenting them as calibrated probabilities.
  • Create an auditable SME workflow that preserves independent judgments before evidence is shown. Save assisted final decisions with their supporting evidence in reviews.csv.
  • Secret Mission: Completed blind-first reviews for ITEM-101, ITEM-102, and ITEM-103. Applied a second-opinion rule to nominate the first case needing another SME review.

Ready to quiz yourself?