Ground Discovery Calls Locally

Build a local Python extractor that grounds discovery findings in call evidence.

Introduction

30 Second Summary

A complaint like "the app is slow" can send a team toward a fix before anyone agrees on the real problem. Important details from customer calls often stay buried in notes.

In this project, you will use Python with Ollama to turn four discovery-call transcripts into grounded scope evidence. Your local extractor will validate six findings before it generates the scope documents.

What You'll Build

You will demo a local site survey that turns four calls into validated findings, a six-section scope document, three flagged risks, and a traceable close-out.

By the end of this project, you'll have:

A grounded extraction you can run across all four calls. Each result shows VALID with ungrounded quotes: 0.
A deterministic flag report that identifies data_not_ready on t1, no_internal_owner on t2, plus success_undefined on t3. The t4 call stays clean.
A must-fail proof where an invented owner triggers two test failures before nine tests pass again. Your scored close-out traces each surviving claim to evidence.
Secret Mission: Check whether every non-empty quote shares at least one normalized word with its answer.

Are there any prerequisites?

You need Python, Git, Ollama, an existing editor, plus an existing AI client. The guided setup keeps every call on your machine.

Before We Start

This short checkpoint keeps the project focused on testing the problem behind "the app is slow" instead of claiming to fix the app. The Delta is: four customer calls become grounded, testable scope evidence before any app change is approved.

Set Up the Local Workspace

Customer calls need a private local workspace before any evidence reaches the model. A fixed design also keeps the project focused on testing the problem behind "the app is slow".

You will prepare a local Git repository with an isolated Python environment. You will also pull the required model through Ollama before copying the design.

In this step, get ready to:

Create the local repository at ~/site-survey.
Prepare the local model with the required Python package families.
Copy the fixed design into the Add the design docs commit.

Create the repository and pull the model

The discovery calls stay on your machine throughout this project. The repository holds the fixed design while the local model supports extraction later.

Removing an old project folder is permanent. The platform-specific command below targets only ~/site-survey.

Move to your home folder by running this command:

cd ~

Why Start Here?

The home folder is the required starting location. Every later terminal can reach the project through the same full path.

Remove only an existing ~/site-survey folder by following the tab for your platform.

macOS

Remove the old folder by running this command:

rm -rf ~/site-survey

What Does This Command Do?

This deletes the old ~/site-survey folder. Other folders in your home directory remain untouched.

Windows

Remove the old folder by running this command:

if (Test-Path ~/site-survey) { Remove-Item ~/site-survey -Recurse -Force }

What Does This Command Do?

PowerShell checks for ~/site-survey before deleting it. Other folders in your home directory remain untouched.

Linux

Remove the old folder by running this command:

rm -rf ~/site-survey

What Does This Command Do?

This deletes the old ~/site-survey folder. Other folders in your home directory remain untouched.

Initialize the replacement repository by running this command:

git init site-survey 2>&1 | tee -a check-out.txt

What Does This Command Do?

The command initializes the site-survey repository. It also appends the terminal output to check-out.txt.

Enter the new repository by running this command:

cd ~/site-survey

Why Use the Full Path?

Every new terminal in this project starts at ~/site-survey. The full home-relative path prevents later commands from running in another folder.

Git needs an author name before it can create the design commit. The next command reveals whether a value is already available.

Check the Git author name by running this command:

git config user.name 2>&1 | tee -a check-out.txt

What Should You See?

A configured author name appears in the terminal. A blank result means you need to set the value for this repository.

Use the next tabs only when the name check is blank.

macOS

Set the local author name by running this interactive command:

read -r -p "Git user.name: " GIT_USER_NAME && git config user.name "$GIT_USER_NAME"

What Happens Next?

The terminal asks for your author name. Git stores your response in this repository.

Windows

Set the local author name by running this interactive command:

git config user.name (Read-Host "Git user.name") 2>&1 | tee -a check-out.txt

What Happens Next?

PowerShell asks for your author name. Git stores your response in this repository.

Linux

Set the local author name by running this interactive command:

read -r -p "Git user.name: " GIT_USER_NAME && git config user.name "$GIT_USER_NAME"

What Happens Next?

The terminal asks for your author name. Git stores your response in this repository.

An author email completes the commit identity. The next check shows whether that value is already available.

Check the Git author email by running this command:

git config user.email 2>&1 | tee -a check-out.txt

What Should You See?

A configured email appears in the terminal. A blank result means you need to set the value for this repository.

Use the next tabs only when the email check is blank.

macOS

Set the local author email by running this interactive command:

read -r -p "Git user.email: " GIT_USER_EMAIL && git config user.email "$GIT_USER_EMAIL"

What Happens Next?

The terminal asks for your author email. Git stores your response in this repository.

Windows

Set the local author email by running this interactive command:

git config user.email (Read-Host "Git user.email") 2>&1 | tee -a check-out.txt

What Happens Next?

PowerShell asks for your author email. Git stores your response in this repository.

Linux

Set the local author email by running this interactive command:

read -r -p "Git user.email: " GIT_USER_EMAIL && git config user.email "$GIT_USER_EMAIL"

What Happens Next?

The terminal asks for your author email. Git stores your response in this repository.

The model download is the slowest part of this setup. Expect it to take several minutes while Ollama stores gemma3:4b locally.

Pull the local model by running this command:

ollama pull gemma3:4b 2>&1 | tee -a check-out.txt

What Does This Command Do?

Ollama downloads the model for local use. The command also records its progress in check-out.txt.

Model Download Stopped?

Confirm that Ollama is running locally. Check your network connection before repeating the pull command.

For an unresolved failure, help me diagnose the local model download.

Prepare the Python environment

A virtual environment keeps this project's dependencies inside .venv. The active terminal also receives cache controls that keep generated folders out of later commits.

Start a new terminal using your usual terminal application.
Enter the full project path by running this command:

cd ~/site-survey

Why Repeat the Path?

A new terminal can start in a different folder. This command places it inside the repository before the environment is created.

Create the environment by following the tab for your platform.

macOS

Create .venv by running this command:

python3 -m venv .venv 2>&1 | tee -a check-out.txt

What Does This Command Do?

Python creates the isolated environment inside .venv. The terminal output is appended to check-out.txt.

Activate the environment by running this command:

source .venv/bin/activate

What Should You See?

Your terminal prompt gains an environment indicator. Later package commands now target the isolated environment.

Disable Python bytecode cache files for this terminal by running this command:

export PYTHONDONTWRITEBYTECODE=1

Why Set This Value?

This prevents Python from writing bytecode cache files during the active terminal session.

Disable the pytest cache provider for this terminal by running this command:

export PYTEST_ADDOPTS="-p no:cacheprovider"

Why Set This Value?

This prevents later test runs from creating the pytest cache directory.

Windows

Create .venv by running this command:

python -m venv .venv 2>&1 | tee -a check-out.txt

What Does This Command Do?

Python creates the isolated environment inside .venv. The terminal output is appended to check-out.txt.

Activate the environment by running this command:

.venv/Scripts/Activate.ps1

What Should You See?

Your terminal prompt gains an environment indicator. Later package commands now target the isolated environment.

Activation Refused?

This policy change applies only to the current PowerShell session. It allows the activation script to run without changing the policy for later sessions.

Allow the activation script in the current session by running this command:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

The command changes the policy for this PowerShell process only.

Repeat the activation by running this command:

.venv/Scripts/Activate.ps1

Your terminal prompt should now show an environment indicator.

If activation remains blocked, help me activate the Windows environment.

Disable Python bytecode cache files for this terminal by running this command:

$env:PYTHONDONTWRITEBYTECODE = "1"

Why Set This Value?

This prevents Python from writing bytecode cache files during the active terminal session.

Disable the pytest cache provider for this terminal by running this command:

$env:PYTEST_ADDOPTS = "-p no:cacheprovider"

Why Set This Value?

This prevents later test runs from creating the pytest cache directory.

Linux

Create .venv by running this command:

python3 -m venv .venv 2>&1 | tee -a check-out.txt

What Does This Command Do?

Python creates the isolated environment inside .venv. The terminal output is appended to check-out.txt.

Virtual Environment Creation Failed?

Debian or Ubuntu may need the operating-system package that supplies virtual environment support.

Install the virtual environment package by running this command:

sudo apt install python3-venv 2>&1 | tee -a check-out.txt

This supplies virtual environment support. Repeat the environment creation command after the installation completes.

For another failure, help me create the local Python environment.

Activate the environment by running this command:

source .venv/bin/activate

What Should You See?

Your terminal prompt gains an environment indicator. Later package commands now target the isolated environment.

Disable Python bytecode cache files for this terminal by running this command:

export PYTHONDONTWRITEBYTECODE=1

Why Set This Value?

This prevents Python from writing bytecode cache files during the active terminal session.

Disable the pytest cache provider for this terminal by running this command:

export PYTEST_ADDOPTS="-p no:cacheprovider"

Why Set This Value?

This prevents later test runs from creating the pytest cache directory.

The active environment is ready for the version-pinned package families. Pydantic provides the schema contract while pytest checks the finished scope later.

Install Pydantic 2.14.* with pytest 9.1.* plus Ollama Python 0.6.* by running this command:

python -m pip install "pydantic==2.14.*" "pytest==9.1.*" "ollama==0.6.*" 2>&1 | tee -a check-out.txt

What Does This Command Install?

The command installs the schema library with the test runner. It also installs the Python client used to call the local model.

Package Installation Failed?

Confirm that your terminal still shows an active environment indicator. Check your network connection before repeating the installation.

For another installer failure, help me diagnose the package installation.

The project separates design documents from customer calls. A third directory holds the retained fixture used by the must-fail test later.

Create the three project directories by running these commands:

mkdir docs
mkdir transcripts
mkdir fixtures

What Do These Commands Create?

The docs directory holds the five design documents.
The transcripts directory holds the four discovery calls.
The fixtures directory holds the retained fabricated input created later.

Before you check the model list, which model name do you expect to see?

Verify the local model by running this command:

ollama list 2>&1 | tee -a check-out.txt

What Should You See?

You should see gemma3:4b in the model list. That entry proves the required model is available locally.

Copy the fixed design and commit it

The five design documents freeze the Delta with the acceptance criteria before coding begins. The fixed inputs give every learner the same evidence base.

Use your existing editor to open ~/site-survey.
Create docs/requirements-brief.md inside the existing docs directory.
Paste the opening section below into docs/requirements-brief.md.

# Requirements Brief

## Title

The problem behind "the app is slow"

## Delta

Four customer calls become grounded, testable scope evidence before any app change is approved.

## Dated reason

Ollama added structured outputs in December 2024, allowing a local model response to be constrained by a JSON schema.

## Core time

- Setup: 5 minutes.
- Design: 7 minutes.
- Build: 21 minutes.
- Validation: 6 minutes.
- Close-out: 6 minutes.
- Total: 45 minutes.

What Does This Section Fix?

This section fixes the project's responsibility. It also protects the 21-minute build time.

Append the acceptance section below to docs/requirements-brief.md.

## Acceptance criteria

- AC-1: Four `VALID` lines, each with `ungrounded quotes: 0`.
- AC-2: Flags match the supplied key. `t4` has none. This is one clean call, not a false-positive rate.
- AC-3: `prd.md` has six sections and `red-flags.md` has exactly three non-empty lines.
- AC-4: The clean test suite reports nine passed tests.
- AC-5: The retained invented-owner fixture fails two tests, is undone, and the restored suite reports nine passed tests.
- AC-6: `readout.html` is self-contained and puts the answer first. It includes the three flags, 900 cases against the warning about more than several hundred, `173,638 CommCare users` and Dimagi, and what this project does not fix.

## Acceptance judgement

For every acceptance item, report `EVIDENCE:` followed by a traceable source. Never report yes. If only a learner run can establish an item and the run has not happened, report `owed`.

The acceptance report must also show that the must-fail caught the invented owner twice and that each decision record was defended aloud.

Why Define Acceptance Now?

These criteria define the observable finish line. The evidence rule prevents an expected result from becoming a completed claim.

Append the close-out section below to docs/requirements-brief.md.

## Close-out

The learner's own AI drafts the first versions of `README.md`, `docs/TEACHBACK.md`, `docs/AAR.md`, and `readout.html` only after the restored test run. The first three close-out documents and the readout are never pre-written here.

`docs/TEACHBACK.md` is a one-page Discovery Checklist titled `The problem behind "the app is slow"` and is defended aloud.

`docs/AAR.md` states what worked and what to change. It records the first-draft score as `untraceable claims / total claims`. Every untraceable claim is struck before acceptance.

`readout.html` is self-contained, uses the same title, and puts the answer first. Never print the finished readout.

Every claim in the readout, teach-back, and AAR traces to these design documents or to a pasted result. The close-out reports time as built against the minute split, confirms whether the build received 21 minutes, checks every required artifact, confirms that no helper AI prompt ran before close-out, lists every term used but not defined, and reports unfinished work with its trade.

Why Delay Helper AI?

The helper AI drafts close-out only after trusted output is restored. This keeps product execution separate from the later reporting task.

Append the limitations below to docs/requirements-brief.md.

## What this does not fix

- It fixes no app. Trade: scope stays honest, but the customer sees no performance improvement.
- It measures no performance. Trade: no unsupported speed claim is made, but no baseline or improvement is established.
- A grounded quote can still sit under the wrong answer. Trade: exact evidence is auditable, but classification judgement remains.
- Three rules cover only three flags. Trade: the checks are deterministic, but narrow.
- A local 4B model costs speed and judgement. Trade: calls stay on the machine.

Why Record the Limits?

The limitations set the boundary for every later claim. Grounded extraction never becomes proof of an app fix or a performance improvement.

Append the glossary below to docs/requirements-brief.md.

## Glossary

- 4B model: the local model class selected for this project.
- After-action review: the record of what worked and what to change.
- Answer first: state the recommendation before supporting detail.
- Claim: one factual assertion that can be checked against a source.
- Cost of inaction: what is lost if nothing changes.
- Day 2 owner: who inside the customer runs the app after go-live.
- Delta: the change this project is responsible for producing.
- Discovery Checklist: the one-page teach-back used to defend the discovery approach aloud.
- Evidence: a design statement or pasted run result that supports a claim.
- Fixture: a retained test input used to exercise a known path.
- Grounded quote: a quote whose normalized text occurs inside the normalized transcript.
- Local Ollama: Ollama running the model on the learner's machine.
- Must-fail: a deliberate test run expected to fail for a named reason.
- Owed: evidence that only a future learner action or run can produce.
- PRD: the generated scope document named `prd.md`.
- Red flag: one of the three deterministic scope-stop labels.
- Scale: how many cases or users one person or device carries.
- Schema contract: the six-field Pydantic structure required for every model response.
- Structured outputs: model output constrained by a JSON schema.
- Success measure: the target that defines success.
- System of record: what the speaker calls the official record.
- Traceable claim: a claim supported by a design document or pasted result.
- Untraceable claim: a claim with no such support.

Why Include a Glossary?

The glossary fixes the meaning of terms used during the build. It also supports the final audit for undefined terminology.

Save docs/requirements-brief.md.
Confirm that its final heading is Glossary.

Requirements Brief Looks Incomplete?

Check that each appended section sits below the previous section. Confirm that no earlier content was replaced.

For help comparing the sections, help me check the requirements brief.

Create docs/decisions.md inside the docs directory.
Paste Decision Record 1 below into docs/decisions.md.

# Decision Records

## Record 1: Keep the calls local

Decision: The calls must stay on the machine, so a 4B model runs on local Ollama.

It beat: a hosted API and the learner's chat client.

Cost: speed and judgement.

Reverse when: the customer approves a hosted model.

Aloud defense: explain why privacy wins here, name both rejected alternatives, state the cost, and state the reversal condition.

What Does Record 1 Decide?

The first record makes local processing the fixed privacy choice. It also records the cost with the reversal condition.

Append Decision Record 2 below to docs/decisions.md.

## Record 2: The model only proposes

Decision: The schema is the contract and the code checks every quote.

It beat: free text and JSON with no schema.

Cost: six fields must be defined.

Reverse when: discovery comes as a form.

Aloud defense: explain why structure and quote checks remain code-owned, state both rejected alternatives, state the cost, and state the reversal condition.

What Does Record 2 Decide?

The second record gives the model a proposal role. The schema contract with the quote checks remains code-owned.

Append Decision Record 3 below to docs/decisions.md.

## Record 3: Rules decide the flags

Decision: Asked to flag, the model missed two of three in testing, so three rules decide the flags.

It beat: letting the model flag.

Coverage: three flags only.

Reverse when: a model flags right on a labelled set.

Aloud defense: explain why the test result justifies deterministic rules, state the rejected approach, state the coverage limit, and state the reversal condition.

What Does Record 3 Decide?

The third record assigns the named red flags to deterministic rules. Its coverage stays limited to three flags.

Save docs/decisions.md.
Confirm that the file contains three decision records.

Create docs/topology.md inside the docs directory.
Paste the topology below into docs/topology.md.

# Discovery Extractor Topology

```mermaid
flowchart LR
    T[transcripts/*.txt] --> E[extract.py]
    R[rules.txt] --> E
    S[schema.py] --> E
    E --> O[local Ollama gemma3:4b]
    O --> E
    E --> J[out/*.json]
    J --> C[scope.py]
    C --> P[prd.md]
    C --> F[red-flags.md]
    K[answer_key.json] --> X[test_scope.py]
    T --> X
    J --> X
    P --> X
    F --> X
    B[fixtures/fabricated.json] -. replaces out/t2.json .-> X
    X --> Z[learner AI close-out after restore]
```

The model proposes. The schema constrains. Python checks quotes and flags. The learner's AI drafts close-out only after the fixture is undone and nine tests are restored.

What Does the Topology Show?

The Mermaid graph separates model proposals from code-owned checks. It also places learner AI close-out after the restored test run.

Save docs/topology.md.
Confirm that its final node names learner AI close-out after restore.

Create docs/build-plan.md inside the docs directory.
Paste the build plan below into docs/build-plan.md.

# Build Plan

## Minute split

| Phase | Minutes | Required exit |
|---|---:|---|
| Setup | 5 | Home-folder repository, local model, environment, packages, and folders are ready. |
| Design | 7 | Five copied design documents, TASKS, fixed inputs, preview, and design commit exist. |
| Build | 21 | Schema 5, extractor plus first run and commit 8, scope 4, tests and commit 4. |
| Validation | 6 | The retained invented-owner fixture is caught by exactly two tests. |
| Close-out | 6 | Output is restored, nine tests are rerun, drafts are scored and corrected, decisions are defended aloud, artifacts are checked, and the tag is listed. |
| Total | 45 | Core project complete. |

## As-built rule

The close-out records actual minutes beside this plan. It does not copy planned minutes into an as-built claim. If no actual time was recorded, the evidence is `owed`.

## Scope rule

Do not cut scope to recover time. Fix failures in place. The optional quote-word check happens only after close-out.

What Does the Plan Protect?

The plan reserves 21 minutes for the core build. It also requires actual timing evidence during close-out.

Save docs/build-plan.md.
Confirm that the total remains 45 minutes.

Create docs/source-map.md inside the docs directory.
Paste the source map below into docs/source-map.md.

# Source Map

## Dated reason

- Ollama, Structured outputs, December 6, 2024: https://ollama.com/blog/structured-outputs

## Readout facts

- Dimagi, 2024 Year-in-Review: `173,638 CommCare users`: https://dimagi.com/blog/2024-year-in-review/
- Dimagi performance guidance: `If users are syncing more than several hundred cases at a time, performance may start to decline.`: https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143956576/Managing+Application+Size+as+it+Relates+to+Performance

## Project evidence

- `answer_key.json` defines the three expected flags and the clean `t4` call.
- `check-out.txt` receives pasted run results.
- The must-fail fixture supplies the invented owner used to prove two checks.
- The performance source requires testing the specific app on its devices at anticipated load. This project performs no such measurement.

What Does the Source Map Protect?

The source map ties dated facts to named sources. It also records the limits of the project's performance evidence.

Save docs/source-map.md.
Confirm that it contains the dated reason with the readout facts.

Create TASKS.md inside ~/site-survey.
Paste the checklist below into TASKS.md.

Delta: four customer calls become grounded, testable scope evidence before any app change is approved.

## Build

- [ ] Build the six-field schema.
- [ ] Build and run the local extractor.
- [ ] Commit four local outputs.
- [ ] Build the scope writer.
- [ ] Build the nine-test suite.

## Acceptance criteria

- [ ] AC-1: Four VALID lines with zero ungrounded quotes.
- [ ] AC-2: Flags match the key, including no flags on t4.
- [ ] AC-3: Six PRD sections and three red-flag lines.
- [ ] AC-4: Nine tests pass.
- [ ] AC-5: The retained fixture fails twice, is undone, and nine tests pass again.
- [ ] AC-6: The scored and corrected self-contained readout contains the required answer-first evidence and limitations.

What Does the Checklist Track?

The checklist separates build work from acceptance evidence. Each box stays empty until its result exists.

Save TASKS.md.
Confirm that all build boxes remain unticked.

Create .gitignore inside ~/site-survey.
Paste the ignore rules below into .gitignore.

.venv/
pycache/

What Do These Rules Exclude?

Git excludes the local environment with the named cache directory. The terminal cache controls prevent additional Python test cache files.

Save .gitignore.
Confirm that the file contains exactly two lines.

Create answer_key.json inside ~/site-survey.
Paste the fixed key below into answer_key.json.

{"t1": ["data_not_ready"], "t2": ["no_internal_owner"], "t3": ["success_undefined"], "t4": []}

What Does the Key Define?

The key assigns one expected flag to each of the first three calls. It defines t4 as one clean call.

Save answer_key.json.
Confirm that t4 has an empty list.

Create rules.txt inside ~/site-survey.
Paste the extraction rules below into rules.txt.

Fill every field from this call. Each answer has a quote copied word for word.

system_of_record: what the speaker calls the official record.

cost_of_inaction: what is lost if nothing changes.

day2_owner: who inside the customer runs the app after go-live.

data_ready: "later" with its quote only if real data will be ready later, else "now" with an empty quote.

success_measure: the target for success, such as a number, a time or a deadline.

scale: how many cases or users one person or device carries.

If nothing fits, the answer is "none named" and the quote is empty.

What Do the Rules Require?

The rules define all six findings. They also require word-for-word quotes when evidence exists.

Save rules.txt.
Confirm that all six field names appear.

Create transcripts/t1.txt inside the transcripts directory.
Paste Call 1 below into transcripts/t1.txt.

Call 1, program director. Honestly, the app is slow, that is the whole problem. Our official record is the district health information system. When sync fails we lose about two days of home visits a week. After go-live Grace, our IT lead, runs it, and success is fewer than one failed sync a month. Oh, and the case export will be ready in two weeks, so no real data yet.

What Evidence Does Call 1 Contain?

Call 1 includes a system of record with a cost of inaction. It also includes an owner with a countable target plus data that arrives later.

Save transcripts/t1.txt.
Confirm that the transcript ends with no real data yet.

Create transcripts/t2.txt inside the transcripts directory.
Paste Call 2 below into transcripts/t2.txt.

Call 2, field supervisor. Each of my health workers carries about 900 open cases on one phone, and the case list takes forever to open. We need it open in under five seconds. The record that counts is the district health information system. If it stays this slow, workers go back to paper. Who looks after it after launch? Nobody here, really. We assumed the vendor would.

What Evidence Does Call 2 Contain?

Call 2 includes 900 open cases with a target below five seconds. It also names the absence of an internal owner.

Save transcripts/t2.txt.
Confirm that the transcript ends with the vendor assumption.

Create transcripts/t3.txt inside the transcripts directory.
Paste Call 3 below into transcripts/t3.txt.

Call 3, data manager. I run the nightly sync into the district health information system, so that is our system of record. A failed sync costs me a full day of cleanup, about four days a month. After go-live I own the app day to day. Success? We will know it is working when people stop complaining.

What Evidence Does Call 3 Contain?

Call 3 names the system of record with an internal owner. Its success statement has no countable target.

Save transcripts/t3.txt.
Confirm that the transcript ends with the complaint-based success statement.

Create transcripts/t4.txt inside the transcripts directory.
Paste Call 4 below into transcripts/t4.txt.

Call 4, finance lead. Our donor reports come from the district health information system, our system of record. Late reports cost us a follow-up audit visit each month. Grace, our IT lead, owns the app after go-live, and success is every report filed by the fifth working day of the month.

What Evidence Does Call 4 Contain?

Call 4 names the required record with a cost. It also includes an owner with a countable success measure.

Save transcripts/t4.txt.
Confirm that the transcript ends with the fifth-working-day target.

Missing a Copied File?

Confirm that the five design documents sit inside docs. Confirm that the four calls sit inside transcripts.

For help checking the file tree, help me compare the local workspace with the required files.

The topology preview is the first visible proof that the copied design connects extraction to scope generation. It also shows that helper AI belongs after restoration.

Use your editor's Markdown preview feature for docs/topology.md.
Confirm that the graph ends with learner AI close-out after restore.

You should see a rendered left-to-right graph. Its final node places learner AI close-out after the restored tests.

Check Before Sharing

A screenshot can expose your local user name through a file path. Crop or hide that detail before sharing the image.

Keep Helper AI for Close-Out

Your helper AI client drafts close-out files only after the must-fail fixture is restored later. Ollama extraction calls count as product execution.

Leave your helper AI client unused during this setup step.
Stage the copied design with the fixed inputs by running this command:

git add -A 2>&1 | tee -a check-out.txt

What Does This Command Stage?

Git stages the copied documents with the fixed inputs. The ignored .venv directory stays outside the commit.

Create the design commit by running this command:

git commit -m "Add the design docs" 2>&1 | tee -a check-out.txt

What Does This Command Create?

Git creates a commit named Add the design docs. Its terminal output is appended to check-out.txt.

Before you inspect the commit, which copied files do you expect its summary to include?

Inspect the design commit by running this command:

git show --stat --oneline HEAD 2>&1 | tee -a check-out.txt

What Should You See?

You should see Add the design docs beside the latest commit. Its summary should include the five design documents with TASKS.md plus the fixed inputs.

That is the setup complete. Your local model with the fixed design now has a traceable starting point.

Commit Missing Files?

Compare the commit summary with your editor's file tree. Save any unsaved file before staging it again.

If Git reports a missing identity, repeat the author checks from this step. For another issue, help me fix the design commit.

✔️ Awesome, I've got everything!

Your local workspace is ready. The rendered topology with the design commit proves the setup is complete.

ⓧ I'd like to double check the full code

Compare your workspace with this complete setup checkpoint. Every item below belongs to this step.

The .gitignore file contains .venv/ with pycache/.
Ollama lists gemma3:4b as a locally available model.
The active .venv contains Pydantic 2.14.* with pytest 9.1.* plus Ollama Python 0.6.*.
The repository exists at ~/site-survey with an Add the design docs commit.
The root contains TASKS.md with .gitignore plus answer_key.json plus rules.txt plus check-out.txt.
The fixed key is {"t1": ["data_not_ready"], "t2": ["no_internal_owner"], "t3": ["success_undefined"], "t4": []}.
The docs directory contains requirements-brief.md with decisions.md plus topology.md plus build-plan.md plus source-map.md.
The transcripts directory contains t1.txt with t2.txt plus t3.txt plus t4.txt.
The empty fixtures directory exists for the retained fixture created later.
The topology preview renders the extraction flow with post-restore close-out.
The helper AI client has not received a close-out request.

Your private workspace now has a fixed design with local dependencies. Next, you will turn the six required findings into a schema contract that Python can inspect.

Build the Six-Field Contract

Your workspace now holds the fixed design documents and four discovery transcripts. The next piece is a schema contract that gives every call the same six finding slots.

Free text can omit a required finding. JSON without a schema can change shape between calls.

In this step, Pydantic makes the contract visible before any model call. Python also takes responsibility for checking quotes and applying the three flag rules.

In this step, get ready to:

Define the six-field extraction contract.
Implement deterministic evidence checks.
Verify the ordered field list from Python.

Define the six-field schema

Each Finding stores an answer beside its supporting quote. Extraction requires one finding for every discovery question.

In your existing editor, use its new-file control to create schema.py directly inside ~/site-survey.
Add the model definitions and ordered field names to schema.py with this code:

import re

from pydantic import BaseModel


class Finding(BaseModel):
    answer: str
    quote: str


class Extraction(BaseModel):
    system_of_record: Finding
    cost_of_inaction: Finding
    day2_owner: Finding
    data_ready: Finding
    success_measure: Finding
    scale: Finding


FINDING_FIELDS = (
    "system_of_record",
    "cost_of_inaction",
    "day2_owner",
    "data_ready",
    "success_measure",
    "scale",
)

What does this code do?

The Finding model keeps each proposed answer attached to the quote that supports it.
The Extraction model makes all six discovery findings required.
The FINDING_FIELDS tuple gives later checks one fixed field order.

Save schema.py.
Return to the active terminal from setup.
Check that the first part of the schema imports by running:

python -c "import schema" 2>&1 | tee -a check-out.txt

What does this check prove?

Python loads schema.py from the current project. A return to the terminal prompt without a traceback proves that the model definitions are valid Python.

Your six required findings now exist as a code-owned structure. The remaining functions turn that structure into an evidence contract.

Does the schema fail to import?

Confirm that schema.py sits directly inside ~/site-survey. Check the class indentation against the code above.

Make sure the active terminal is still inside ~/site-survey.

help me diagnose my schema import

Add grounding and flag rules

Quote grounding compares normalized text so capitalization and punctuation do not hide an exact match. A countable-success expression also lets Python identify success answers that lack a number or time target.

In schema.py, place your cursor after the closing parenthesis of FINDING_FIELDS.
Add the countable-success expression and normalization function with this code:

COUNTABLE = re.compile(
    r"\d|\b(one|two|three|four|five|six|seven|eight|nine|ten|first|fifth|tenth|percent|seconds?|minutes?|hours?|days?|weeks?|months?)\b"
)


def norm(s: str) -> str:
    lowered = s.lower()
    cleaned = re.sub(r"[^a-z0-9 ]", " ", lowered)
    return " ".join(cleaned.split())

How does normalization help?

The COUNTABLE expression recognizes digits plus common words for quantities and time.
The norm() function converts text to lowercase.
The same function removes punctuation and collapses repeated spaces so quote checks compare consistent text.

Save schema.py.
Check that the normalization code imports by running:

python -c "import schema" 2>&1 | tee -a check-out.txt

What did Python check?

Python compiles the expression when it imports the module. Returning to the prompt without a traceback confirms that the expression and norm() are syntactically valid.

Does the expression cause an import problem?

Compare every backslash and parenthesis with the code block. Keep the lowercase r before the expression string.

help me check the normalization code

Normalized text gives the project a consistent comparison surface. The grounding function now uses it to find quotes that are absent from their source transcript.

In schema.py, place your cursor below norm().
Add the quote-grounding function with this code:

def ungrounded(ex: Extraction, text: str) -> list[str]:
    text_norm = norm(text)
    missing = []
    for field_name in FINDING_FIELDS:
        finding = getattr(ex, field_name)
        if finding.answer.lower() == "none named":
            continue
        if norm(finding.quote) not in text_norm:
            missing.append(finding.quote)
    return missing

What does the grounding check do?

The ungrounded() function normalizes the complete transcript once.
The loop checks every field listed in FINDING_FIELDS.
Answers recorded as none named do not require a quote under the supplied extraction rules.
Every other quote is added to missing when its normalized text does not occur in the transcript.

Save schema.py.
Check that the grounding function imports by running:

python -c "import schema" 2>&1 | tee -a check-out.txt

What does this import confirm?

Python resolves Extraction, FINDING_FIELDS, and norm() inside the new function. A clean return to the prompt confirms that the references are connected.

Does the grounding function fail to import?

Check that every line inside ungrounded() uses four spaces of indentation. Confirm that both if blocks remain inside the loop.

help me fix the grounding function

Grounding answers whether a quote exists in the transcript. The final function applies the three ordered scope-stop rules without asking the model to decide.

In schema.py, place your cursor below ungrounded().
Add the deterministic flag function with this code:

def red_flags(ex: Extraction) -> list[str]:
    flags = []
    if ex.data_ready.answer == "later":
        flags.append("data_not_ready")
    if ex.day2_owner.answer == "none named":
        flags.append("no_internal_owner")
    if not COUNTABLE.search(ex.success_measure.answer.lower()):
        flags.append("success_undefined")
    return flags

How are the flags decided?

The data_not_ready flag appears when real data is recorded as available later.
The no_internal_owner flag appears when the customer names no Day 2 owner.
The success_undefined flag appears when the success answer contains no countable target.
The order of these checks fixes the order of any returned flags.

Save schema.py.
Syntax-check the complete module by running:

python -c "import schema" 2>&1 | tee -a check-out.txt

What does the complete import prove?

This command imports every model, constant, and function in schema.py. Returning to the prompt without a traceback confirms that the complete contract is valid Python.

The complete module now owns both evidence checks and flag decisions. That keeps the later model run focused on proposing findings.

Does the complete import fail?

Start with the last function you added. Confirm that each condition ends with a colon and each flags.append() line is indented beneath its condition.

help me debug the complete schema module

✔️ Awesome, I've got everything!

Great. Make sure schema.py is saved before you run the visible field check.

ⓧ I'd like to double check the full code

Compare your saved schema.py with the complete file below:

import re

from pydantic import BaseModel


class Finding(BaseModel):
    answer: str
    quote: str


class Extraction(BaseModel):
    system_of_record: Finding
    cost_of_inaction: Finding
    day2_owner: Finding
    data_ready: Finding
    success_measure: Finding
    scale: Finding


FINDING_FIELDS = (
    "system_of_record",
    "cost_of_inaction",
    "day2_owner",
    "data_ready",
    "success_measure",
    "scale",
)

COUNTABLE = re.compile(
    r"\d|\b(one|two|three|four|five|six|seven|eight|nine|ten|first|fifth|tenth|percent|seconds?|minutes?|hours?|days?|weeks?|months?)\b"
)


def norm(s: str) -> str:
    lowered = s.lower()
    cleaned = re.sub(r"[^a-z0-9 ]", " ", lowered)
    return " ".join(cleaned.split())


def ungrounded(ex: Extraction, text: str) -> list[str]:
    text_norm = norm(text)
    missing = []
    for field_name in FINDING_FIELDS:
        finding = getattr(ex, field_name)
        if finding.answer.lower() == "none named":
            continue
        if norm(finding.quote) not in text_norm:
            missing.append(finding.quote)
    return missing


def red_flags(ex: Extraction) -> list[str]:
    flags = []
    if ex.data_ready.answer == "later":
        flags.append("data_not_ready")
    if ex.day2_owner.answer == "none named":
        flags.append("no_internal_owner")
    if not COUNTABLE.search(ex.success_measure.answer.lower()):
        flags.append("success_undefined")
    return flags

How is the file organized?

The file moves from data shape to ordered fields. It then adds text normalization, quote grounding, and deterministic flag rules.

Verify the visible contract

The import check proves that Python accepts the module. Printing the model fields now reveals whether the contract contains every required discovery finding in the intended order.

Before you run the check, which six field names do you expect Python to list?

Print the schema field names and record the result by running:

python -c "from schema import Extraction; print(list(Extraction.model_fields))" 2>&1 | tee -a check-out.txt

What does this command inspect?

Pydantic exposes the fields declared by Extraction through model_fields. Converting those field names to a list makes the contract visible in the terminal.

You should see system_of_record, cost_of_inaction, day2_owner, data_ready, success_measure, and scale in that order.

Is a field missing or out of order?

Compare the Extraction class with the full-code tab. Each field must use the exact project identifier and the Finding type.

Save the file before rerunning the command if your terminal still shows an earlier field list.

help me fix the six-field contract

That is the contract working: all six discovery findings are visible before the first local extraction. Next, you will let Ollama propose four schema-valid records while Python checks every quote.

Extract the Four Calls

Your six-field Pydantic schema contract now names every discovery finding the project needs. The field-list result proves that the structure exists.

However, the contract contains no customer evidence. The four transcripts need to become schema-valid findings before they can support scope decisions.

In this step, a local Ollama model proposes findings through a JSON Schema. Python validates each response before it checks every grounded quote.

In this step, get ready to:

Build extract.py to process the four local transcripts.
Run the extractor to record its actual status lines in check-out.txt.
Compare the recorded evidence with the supplied key before committing the outputs.

Set up the local extractor

The extractor needs the model name plus the copied rules before it can process a transcript. Its first runnable slice also creates the out folder for the generated files.

Create extract.py inside ~/site-survey from the file tree in your existing editor.
Paste this setup into extract.py.

import json
from pathlib import Path

import ollama

from schema import Extraction, red_flags, ungrounded

MODEL = "gemma3:4b"
RULES = Path("rules.txt").read_text(encoding="utf-8")


def main():
    schema = Extraction.model_json_schema()
    output_dir = Path("out")
    output_dir.mkdir(exist_ok=True)


if __name__ == "__main__":
    main()

What does this code do?

The imports provide local model access plus the validation helpers from schema.py.
MODEL selects the local gemma3:4b model.
RULES reads the fixed extraction instructions from rules.txt.
main() produces the JSON schema before creating the output directory.

Save extract.py.
Create the output folder by running the current extractor slice:

python extract.py 2>&1 | tee -a check-out.txt

What should I see?

The terminal returns to its prompt without a traceback. Your editor file tree now shows an out folder inside ~/site-survey.

That first slice works. The extractor now has a destination for four local responses.

Seeing a traceback?

Confirm that extract.py is inside ~/site-survey. The relative paths depend on that location.

Check that your virtual environment remains active. The environment from earlier contains the required Python packages.

help me diagnose this extractor setup traceback

Extract and inspect the four calls

The processing loop applies the same contract to every transcript. Structured outputs constrain each response before Pydantic accepts it.

Place your cursor on the blank line below output_dir.mkdir(exist_ok=True) in extract.py.
Paste this loop inside main().

    for transcript_path in sorted(Path("transcripts").glob("*.txt")):
        text = transcript_path.read_text(encoding="utf-8")
        reply = ollama.chat(
            model=MODEL,
            messages=[
                {
                    "role": "system",
                    "content": RULES + "\n" + json.dumps(schema),
                },
                {"role": "user", "content": text},
            ],
            format=schema,
            options={"temperature": 0, "seed": 0},
        )
        ex = Extraction.model_validate_json(reply.message.content)
        output_path = output_dir / f"{transcript_path.stem}.json"
        output_path.write_text(ex.model_dump_json(indent=2), encoding="utf-8")

        flags = ", ".join(red_flags(ex)) or "none"
        missing_count = len(ungrounded(ex, text))
        print(
            f"{transcript_path.stem}: VALID, red flags: {flags}, "
            f"ungrounded quotes: {missing_count}"
        )

What does this code do?

The sorted transcript paths keep the run ordered from t1 through t4.
ollama.chat() sends the rules plus the schema to the local model.
format constrains the response shape. The deterministic options set temperature and seed to 0.
Extraction.model_validate_json() validates each response before the extractor writes its local JSON file.
red_flags() applies the three code-owned rules. ungrounded() counts quotes missing from the source transcript.

Save extract.py.

✔️ Awesome, I've got everything!

Your completed extractor is ready to run against all four local transcripts.

ⓧ I'd like to double check the full code

Compare your complete extract.py with this reference.

import json
from pathlib import Path

import ollama

from schema import Extraction, red_flags, ungrounded

MODEL = "gemma3:4b"
RULES = Path("rules.txt").read_text(encoding="utf-8")


def main():
    schema = Extraction.model_json_schema()
    output_dir = Path("out")
    output_dir.mkdir(exist_ok=True)

    for transcript_path in sorted(Path("transcripts").glob("*.txt")):
        text = transcript_path.read_text(encoding="utf-8")
        reply = ollama.chat(
            model=MODEL,
            messages=[
                {
                    "role": "system",
                    "content": RULES + "\n" + json.dumps(schema),
                },
                {"role": "user", "content": text},
            ],
            format=schema,
            options={"temperature": 0, "seed": 0},
        )
        ex = Extraction.model_validate_json(reply.message.content)
        output_path = output_dir / f"{transcript_path.stem}.json"
        output_path.write_text(ex.model_dump_json(indent=2), encoding="utf-8")

        flags = ", ".join(red_flags(ex)) or "none"
        missing_count = len(ungrounded(ex, text))
        print(
            f"{transcript_path.stem}: VALID, red flags: {flags}, "
            f"ungrounded quotes: {missing_count}"
        )


if __name__ == "__main__":
    main()

Before you run the extractor, consider whether every quote will be grounded.

Process the four local calls by running:

python extract.py 2>&1 | tee -a check-out.txt

How to Judge the Run

AC-1 requires four recorded status lines. Each line must contain VALID plus ungrounded quotes: 0.

The supplied key expects:

t1 has data_not_ready.
t2 has no_internal_owner.
t3 has success_undefined.
t4 has none.

Only your recorded output can support these acceptance claims. Any unsupported claim remains owed.

Confirm that the editor file tree lists out/t1.json, out/t2.json, out/t3.json, and out/t4.json.
Compare the actual red flags in the terminal with the supplied key.

Extractor Run Stopped?

Check that Ollama remains available locally. The extractor depends on the gemma3:4b model from earlier.

Compare extract.py with the full-code reference. A misplaced loop can stop the run before all four files are written.

help me debug my local extraction run

Record the evidence and commit the outputs

A Git commit preserves the extractor plus the local evidence from this run. Acceptance boxes only become complete when the recorded output supports them.

Tick the first three Build boxes in TASKS.md only when the recorded output supports them.
Tick AC-1 in TASKS.md only when all four recorded lines show zero ungrounded quotes.

Unsupported items remain unticked. Their evidence remains owed.

Stage the extractor outputs plus the recorded run evidence by running:

git add -A 2>&1 | tee -a check-out.txt

What does this command do?

This stages extract.py plus the four local JSON outputs. It also stages the latest evidence in check-out.txt.

Before you commit, consider which new files Git will include in this snapshot.

Create the extraction commit by running:

git commit -m "Extract four calls" 2>&1 | tee -a check-out.txt

What should I see?

Git prints a commit summary containing Extract four calls. That summary confirms the extractor and its local outputs now have a repository checkpoint.

Commit Did Not Complete?

Read the terminal output for an unstaged-file or local-identity issue. Your repository should still use the identity configured during setup.

Confirm that the extractor run created all four files before staging again.

help me finish the extraction commit

That is a major checkpoint. Your local evidence pipeline now has a committed extractor plus its recorded run. Next, you'll generate the scope documents and test the evidence against deterministic expectations.

Generate Scope and Test the Evidence

Your four local extracts now contain candidate scope findings from the discovery calls. Their recorded status lines show what the extractor produced, but valid extraction alone does not prove that the flags are correct or that bad evidence gets rejected.

This step turns those findings into scope documents. A deterministic pytest suite then checks the evidence before an invented owner is placed on the main path.

In this step, get ready to:

Generate a six-section prd.md file and a three-line red-flags.md file from the local outputs.
Create nine tests for quote grounding, expected flags, and generated scope files.
Retain a fabricated fixture and run the required must-fail suite against a deliberately corrupted out/t2.json file.

Generate the scope documents

The extraction files hold individual findings. The scope writer combines them into a recommendation document while keeping the three flag decisions inside Python.

Create scope.py inside ~/site-survey by pasting both code sections below in order:

from pathlib import Path

from schema import Extraction, red_flags


SECTIONS = (
    ("System of record", "system_of_record"),
    ("Cost of inaction", "cost_of_inaction"),
    ("Day 2 owner", "day2_owner"),
    ("Scale", "scale"),
    ("Success measure", "success_measure"),
)


def load_records() -> list[tuple[str, Extraction]]:
    records = []
    for path in sorted(Path("out").glob("*.json")):
        ex = Extraction.model_validate_json(path.read_text(encoding="utf-8"))
        records.append((path.stem, ex))
    return records


def main():
    records = load_records()
    flag_lines = [
        f"- {flag} ({name})"
        for name, ex in records
        for flag in red_flags(ex)
    ]

What does the first section do?

The SECTIONS tuple fixes the five evidence categories that become PRD sections.
The load_records() function reads every local extraction from out/ and validates it against Extraction.
The first part of main() applies the code-owned flag rules to every validated record.

    prd_lines = [
        "# Discovery Scope PRD",
        "",
        "Recommendation: stop app-fix scope until the listed red flags are resolved and the engineer ratifies the evidence.",
        "",
    ]
    for heading, field_name in SECTIONS:
        prd_lines.append(f"## {heading}")
        prd_lines.extend(
            f"- {name}: {getattr(ex, field_name).answer}"
            for name, ex in records
        )
        prd_lines.append("")

    prd_lines.append("## Out of scope until resolved")
    prd_lines.extend(flag_lines or ["- none"])
    prd_lines.append("")

    Path("prd.md").write_text("\n".join(prd_lines), encoding="utf-8")
    Path("red-flags.md").write_text(
        "\n".join(flag_lines) + "\n",
        encoding="utf-8",
    )

    print(f"prd.md: {len(records)} transcripts")
    print(f"red-flags.md: {len(flag_lines)} red flags")


if __name__ == "__main__":
    main()

What does the second section do?

The PRD starts with the recommendation before listing supporting findings from all four calls.
The final PRD section collects every deterministic flag under the unresolved scope.
The two write operations create prd.md and red-flags.md without printing either file in full.
The two status lines expose only the transcript count and flag count for verification.

Save scope.py.

Before you run the scope writer, predict how many transcripts and red flags its status lines will report.

Generate the two scope files and record the actual status lines by running:

python scope.py 2>&1 | tee -a check-out.txt

What does this command prove?

The command runs the scope writer against your actual local extraction files. It also appends the two status lines to check-out.txt so later acceptance claims can point to recorded evidence.

When the outputs match the supplied key, you will see four transcripts and three red flags reported. Treat the lines in your own terminal as the evidence.

Select prd.md in your existing editor's file sidebar.
Count the headings that begin with ##.
Select red-flags.md in the file sidebar.
Count its non-empty lines.

The supported target is six PRD sections and exactly three non-empty flag lines. If your generated files differ, the actual extraction remains the source to investigate.

Scope counts do not match?

Check that out/ still contains t1.json through t4.json. A missing output reduces the transcript count.

Compare the generated flags with answer_key.json. Rerun the extractor only if the recorded evidence shows that a local extraction needs correction.

Help me diagnose my generated scope counts.

✔️ Awesome, I've got everything!

Your scope writer is saved. Keep the actual counts from check-out.txt as the evidence for this run.

ⓧ I'd like to double check the full code

Compare your saved scope.py with this complete file.

from pathlib import Path

from schema import Extraction, red_flags


SECTIONS = (
    ("System of record", "system_of_record"),
    ("Cost of inaction", "cost_of_inaction"),
    ("Day 2 owner", "day2_owner"),
    ("Scale", "scale"),
    ("Success measure", "success_measure"),
)


def load_records() -> list[tuple[str, Extraction]]:
    records = []
    for path in sorted(Path("out").glob("*.json")):
        ex = Extraction.model_validate_json(path.read_text(encoding="utf-8"))
        records.append((path.stem, ex))
    return records


def main():
    records = load_records()
    flag_lines = [
        f"- {flag} ({name})"
        for name, ex in records
        for flag in red_flags(ex)
    ]

    prd_lines = [
        "# Discovery Scope PRD",
        "",
        "Recommendation: stop app-fix scope until the listed red flags are resolved and the engineer ratifies the evidence.",
        "",
    ]
    for heading, field_name in SECTIONS:
        prd_lines.append(f"## {heading}")
        prd_lines.extend(
            f"- {name}: {getattr(ex, field_name).answer}"
            for name, ex in records
        )
        prd_lines.append("")

    prd_lines.append("## Out of scope until resolved")
    prd_lines.extend(flag_lines or ["- none"])
    prd_lines.append("")

    Path("prd.md").write_text("\n".join(prd_lines), encoding="utf-8")
    Path("red-flags.md").write_text(
        "\n".join(flag_lines) + "\n",
        encoding="utf-8",
    )

    print(f"prd.md: {len(records)} transcripts")
    print(f"red-flags.md: {len(flag_lines)} red flags")


if __name__ == "__main__":
    main()

What should match?

This reference reads the four extraction files, generates both scope documents, and prints only their counts. Your saved file should match it exactly.

Test the generated evidence

A generated document can look plausible while containing a bad quote or an incorrect flag. The test suite checks the source evidence and the generated scope as separate responsibilities.

Create test_scope.py inside ~/site-survey by pasting both code sections below in order:

import json
from pathlib import Path

import pytest

from schema import Extraction, red_flags, ungrounded

KEY = json.load(open("answer_key.json", encoding="utf-8"))


def load(stem: str) -> Extraction:
    return Extraction.model_validate_json(
        Path("out", f"{stem}.json").read_text(encoding="utf-8")
    )


@pytest.mark.parametrize("stem", sorted(KEY))
def test_grounded(stem):
    text = Path("transcripts", f"{stem}.txt").read_text(encoding="utf-8")
    assert ungrounded(load(stem), text) == []

How does the grounding test work?

The supplied answer_key.json determines the four transcript names used by the suite.
The load() helper validates each saved output before a test uses it.
The parametrized grounding test compares every non-empty quote with its source transcript.



@pytest.mark.parametrize("stem", sorted(KEY))
def test_red_flags(stem):
    assert red_flags(load(stem)) == KEY[stem]


def test_scope_files():
    prd = Path("prd.md").read_text(encoding="utf-8")
    flag_lines = [
        line
        for line in Path("red-flags.md").read_text(encoding="utf-8").splitlines()
        if line.strip()
    ]
    assert prd.count("## ") == 6
    assert flag_lines == [
        "- data_not_ready (t1)",
        "- no_internal_owner (t2)",
        "- success_undefined (t3)",
    ]

What do the remaining tests protect?

The flag parametrization compares each call's ordered flags with the supplied key.
The scope-file test requires exactly six PRD sections.
The same test requires exactly the three expected non-empty red-flag lines in their fixed order.

Save test_scope.py.

Before you run the clean suite, predict whether the actual local outputs satisfy all three kinds of evidence check.

Run the clean nine-test suite and record its actual result by running:

python -m pytest -q 2>&1 | tee -a check-out.txt

What does the clean suite check?

The command runs four grounding cases, four expected-flag cases, and one generated-file case. The quiet output is appended to check-out.txt for traceability.

When the actual outputs match the supplied evidence, you will see nine passing tests. That recorded run supports the clean state before the deliberate corruption.

Clean suite did not pass?

Read the failing test name first. A grounding failure points to a quote in an output file, while a flag failure points to a mismatch with answer_key.json.

A scope-file failure points to the section count in prd.md or the exact lines in red-flags.md.

Help me trace my failing clean test to its source.

✔️ Awesome, I've got everything!

Your test file now covers grounding, flags, and generated scope structure. Keep the clean result in check-out.txt.

ⓧ I'd like to double check the full code

Compare your saved test_scope.py with this complete file.

import json
from pathlib import Path

import pytest

from schema import Extraction, red_flags, ungrounded

KEY = json.load(open("answer_key.json", encoding="utf-8"))


def load(stem: str) -> Extraction:
    return Extraction.model_validate_json(
        Path("out", f"{stem}.json").read_text(encoding="utf-8")
    )


@pytest.mark.parametrize("stem", sorted(KEY))
def test_grounded(stem):
    text = Path("transcripts", f"{stem}.txt").read_text(encoding="utf-8")
    assert ungrounded(load(stem), text) == []


@pytest.mark.parametrize("stem", sorted(KEY))
def test_red_flags(stem):
    assert red_flags(load(stem)) == KEY[stem]


def test_scope_files():
    prd = Path("prd.md").read_text(encoding="utf-8")
    flag_lines = [
        line
        for line in Path("red-flags.md").read_text(encoding="utf-8").splitlines()
        if line.strip()
    ]
    assert prd.count("## ") == 6
    assert flag_lines == [
        "- data_not_ready (t1)",
        "- no_internal_owner (t2)",
        "- success_undefined (t3)",
    ]

What should match?

This reference defines exactly nine cases. Four check grounding, four check ordered flags, and one checks both generated files.

Update TASKS.md only for the build items and acceptance criteria supported by your recorded clean results.
Stage the clean scope files and tests by running:

git add -A 2>&1 | tee -a check-out.txt

What does this command record?

The command stages the scope writer, tests, generated documents, task updates, and run log. The staged state is ready for a clean evidence commit.

Create the clean scope-and-tests commit only when the recorded run supports it by running:

git commit -m "Add the scope and tests" 2>&1 | tee -a check-out.txt

Why commit the clean state now?

This commit preserves the trusted outputs before the deliberate corruption. It gives the next step a known version to restore.

Trigger the retained must-fail

A fixture preserves a known test input so the same failure can be repeated. This must-fail checks whether an invented owner can slip through both the grounding rule and the supplied flag key.

Retain a copy of the original out/t2.json by running:

cp out/t2.json fixtures/fabricated.json 2>&1 | tee -a check-out.txt

Why retain this copy?

The copied file keeps every original t2 finding together. You will change only its owner field, which isolates the exact evidence defect under test.

Select fixtures/fabricated.json in your existing editor's file sidebar.
Set the day2_owner answer to Grace.
Set the same object's quote to Grace will run it after launch..
Save fixtures/fabricated.json.

The retained fixture should differ from the original extraction only in day2_owner. The new answer and quote are invented evidence.

Place the fabricated fixture on the main path by running:

cp fixtures/fabricated.json out/t2.json 2>&1 | tee -a check-out.txt

What changed on the main path?

The command replaces only the active t2 output with the retained fixture. The tests now receive the invented owner as if it came from the extractor.

Before you run the suite, predict whether both evidence checks will trust the invented owner.

Run the required must-fail suite and record what the tests actually catch by running:

python -m pytest -q 2>&1 | tee -a check-out.txt

This failure is intentional

The intended result is exactly two failures. test_grounded[t2] rejects the invented quote because it does not occur in the transcript.

test_red_flags[t2] rejects the invented owner because it removes the expected no_internal_owner flag. The actual terminal result remains the evidence for whether both checks fired.

When those two failures appear, the suite has done its job. It has caught the same invented owner through two independent checks.

Did the must-fail produce a different result?

Check that out/t2.json now contains the same invented day2_owner values as fixtures/fabricated.json.

Check that every other field in the fixture still matches the original local t2 extraction. An extra edit can create unrelated failures.

Help me diagnose my must-fail result.

Your clean scope is committed, and the retained fixture now proves whether invented evidence is rejected on the main path. Next, you will restore the trusted output before drafting and scoring the close-out.

Restore, Close Out, and Tag

The required must-fail run left out/t2.json deliberately corrupted. Its recorded output shows whether your grounding check and expected-flag check caught the invented owner.

Your trusted evidence needs to return before a helper AI drafts any close-out claim.

After restoration, you will score every draft for traceability. You will finish by creating a Git tag for the supported close-out.

In this step, get ready to:

Restore the trusted t2 extraction. Rerun the nine-test suite before using your helper AI client.
Draft the four close-out files with your helper AI client. Score every claim for traceability.
Commit the accepted close-out. Create the v1.0-survey tag.

Restore the trusted output

The fabricated fixture proved that your checks can reject invented evidence. The tracked out/t2.json now provides the trusted version that existed before the obstacle.

Restore the tracked out/t2.json file by running this command:

git checkout -- out/t2.json 2>&1 | tee -a check-out.txt

What does this command do?

Git replaces only out/t2.json with its committed version. The retained fixtures/fabricated.json file stays unchanged for future inspection.

The command also appends its terminal output to check-out.txt. That log preserves what actually happened.

Before you rerun the suite, do you expect the two must-fail checks to recover now that the trusted extraction is back?

Test the restored output by running this command:

python -m pytest -q 2>&1 | tee -a check-out.txt

What does this command prove?

The suite checks four quote-grounding cases. It also checks four expected-flag cases.

The final test checks the six PRD sections and the three red-flag lines. A clean run makes the restored result usable as evidence.

You should see all nine tests pass when the trusted extraction is restored. Treat the result as evidence only because the actual output is now recorded in check-out.txt.

Are tests still failing?

Check that out/t2.json no longer contains the invented Grace quote.
Confirm that fixtures/fabricated.json still contains the invented owner.
Leave the restored-test acceptance entry as owed until all nine tests pass in an actual run.

help me diagnose the restored test failures

Draft and score the close-out

Your helper AI needs the fixed design and actual run log before it can draft grounded close-out files. A detailed prompt keeps every completion claim tied to those sources.

Switch back to your existing helper AI client.
Attach docs/requirements-brief.md, docs/decisions.md, docs/topology.md, docs/build-plan.md, and docs/source-map.md.
Attach TASKS.md, prd.md, red-flags.md, answer_key.json, and check-out.txt.
Paste the following request without submitting it yet:

Draft the first versions of README.md, docs/TEACHBACK.md, docs/AAR.md, and a self-contained readout.html. Do not write or modify Python.

Use the title The problem behind "the app is slow" in the teach-back and readout. Make the teach-back a one-page Discovery Checklist that I can defend aloud. Make the AAR say what worked and what to change. Put the answer first in the readout.

Use only the attached design documents and pasted run results. Never turn an expected result into a completed result. For every acceptance criterion write EVIDENCE: followed by its source, never yes. Write owed for anything only a run or an aloud defense can show and has not shown.

The readout must cover the three flags; no flags on t4 as one clean call, not a false-positive rate; 900 cases against more than several hundred; 173,638 CommCare users and Dimagi; and the fact that this fixes no app, measures no performance, and cannot prevent a grounded quote from sitting under the wrong answer.

Show that the must-fail caught the invented owner twice. Include an evidence-or-owed line for each of the three decision records being defended aloud. Confirm whether any helper AI prompt ran before close-out. Report time AS BUILT against the 5, 7, 21, 6, 6 minute split, including whether build received all 21 minutes. Check that every required artifact exists. List every term used but not defined. Report every item the build leaves undone with its trade.

Treat each factual assertion as one claim. Before correcting the drafts, count claims in docs/TEACHBACK.md, docs/AAR.md, and readout.html. In docs/AAR.md record the first-draft score as untraceable claims / total claims. Strike every untraceable claim using Markdown strikethrough or an HTML s element. Never print the finished readout.

README.md must say what this project is and contain only the three supplied run commands in its Run section. Return the four file drafts and nothing else.

Why is the request this specific?

The request separates expected results from completed results. It also forces every acceptance entry to cite evidence or remain owed.

The scoring instructions expose unsupported claims before the project is tagged. The close-out becomes an audit of evidence instead of a polished guess.

Append the extractor command to the same AI request as its own fenced block:

python extract.py 2>&1 | tee -a check-out.txt

Why include the extractor command?

This gives the drafting client the exact extractor command allowed in the README.md Run section. It prevents an invented command from entering the close-out.

Append the scope command to the same AI request as its own fenced block:

python scope.py 2>&1 | tee -a check-out.txt

Why include the scope command?

This preserves the exact command that generates prd.md and red-flags.md. The README stays aligned with the project the learner actually ran.

Append the test command to the same AI request as its own fenced block:

python -m pytest -q 2>&1 | tee -a check-out.txt

Why include the test command?

This supplies the exact verification command for the README. It also keeps test evidence in check-out.txt whenever the learner runs it.

Submit the complete request after all three fenced commands are present.

Once the response arrives, your four first drafts are ready for the traceability audit. That completes the first and only helper AI request in the core project.

Save the four returned drafts as README.md, docs/TEACHBACK.md, docs/AAR.md, and readout.html.
Confirm that all four paths appear in your existing editor's file tree.
Count every factual assertion in docs/TEACHBACK.md, docs/AAR.md, and readout.html before changing their wording.
Record the first-draft score as untraceable claims / total claims in docs/AAR.md.
Strike every untraceable claim with Markdown strikethrough or an HTML s element.
Preview the corrected formatting in your existing editor.

You should see the first-draft ratio in the after-action review. You should also see every unsupported claim visibly struck.

What does the score measure?

Each factual assertion counts as one claim. A claim is traceable when a design document or pasted run result supports it.

The ratio records the first draft before correction. Striking unsupported claims preserves the audit trail.

Check every remaining claim against a copied design document or a pasted run result.
Verify that the README.md Run section contains only the three supplied commands.
Replace any unsupported completion statement with owed.

When should evidence stay owed?

Use EVIDENCE: only when a named source proves the acceptance item. Use owed when the required run or spoken defense has not happened.

This keeps silence from turning into a completion claim. Every accepted item retains a path back to its source.

Record actual minutes in docs/AAR.md against the 5/7/21/6/6 minute split. Include whether the build received all 21 minutes.
Preview docs/AAR.md after adding the actual-time entries.

You should see an actual duration or an owed entry for every phase. Planned minutes never become as-built evidence on their own.

Record an evidence-or-owed entry for every acceptance criterion.
List every term used by the close-out that its sources do not define.
Preview the acceptance and undefined-term sections in docs/AAR.md.

You should see a named source or owed beside each acceptance item. The undefined-term list makes hidden assumptions visible.

Report every unfinished item with its trade in the close-out.
Return to docs/decisions.md in your existing editor.
Defend Record 1 aloud using its decision, rejected alternatives, cost, and reversal condition.
Defend Record 2 aloud using its decision, rejected alternatives, cost, and reversal condition.
Defend Record 3 aloud using its decision, rejected approach, coverage limit, and reversal condition.
Replace a defense's owed entry with evidence only after speaking that defense.
Save all four close-out files in your existing editor.

Do the drafts contain unsupported claims?

Trace numerical claims to docs/source-map.md or an actual result in check-out.txt.
Trace acceptance claims to the matching run result before marking them complete.
Strike claims that have no supporting attachment.

help me audit the close-out drafts for traceability

Audit, commit, and tag the close-out

The close-out can be committed only after every required artifact has a local path. A final audit checks existence without printing any learner file.

Before you run the audit, do you expect the missing-files list to be empty?

Audit the required artifact paths by running this command:

python -c "from pathlib import Path; required=['README.md','docs/TEACHBACK.md','docs/AAR.md','readout.html','prd.md','red-flags.md','fixtures/fabricated.json','check-out.txt']; print('missing:', [name for name in required if not Path(name).exists()])" 2>&1 | tee -a check-out.txt

What does this audit check?

The command checks the eight required paths without reading their contents into the terminal. It appends the result to check-out.txt for the acceptance record.

An empty missing-files list confirms that every named artifact exists. It does not replace the traceability review you completed inside the files.

You should see an empty missing-files list when all required artifacts are present. Create any named missing file from the saved AI drafts before continuing.

Stage the completed close-out by running this command:

git add -A 2>&1 | tee -a check-out.txt

What does this staging command do?

Git updates its staging area to match the current project files. This includes the restored output, retained fixture, close-out drafts, and updated run log.

Create the close-out commit by running this command:

git commit -m "Add the close-out readout" 2>&1 | tee -a check-out.txt

What does this commit record?

The commit records the staged close-out with the required message. Its terminal output becomes the evidence that the commit command actually succeeded.

Create the project tag by running this command:

git tag v1.0-survey 2>&1 | tee -a check-out.txt

What does this tag identify?

The lightweight tag names the current close-out commit v1.0-survey. The next command checks whether that name is visible in the repository.

Before the final check, do you expect the new tag to appear in the repository's tag list?

List the repository tags by running this command:

git tag --list 2>&1 | tee -a check-out.txt

What does the final command prove?

The command lists the tags that Git can see in this repository. It also appends that list to check-out.txt.

You should see v1.0-survey in the output. That visible line is the evidence for tag visibility.

Is the close-out commit or tag missing?

Check the artifact audit for a missing file before retrying the commit.
Confirm that the close-out commit succeeded before creating the tag.
Leave the related acceptance entry as owed when the terminal does not show successful evidence.

help me diagnose the missing close-out commit or tag

Strong finish. Your trusted extraction is restored, your close-out claims have an audit trail, and the visible tag marks the accepted core project.

Secret mission

Check Answer-Quote Word Overlap

Exact grounding can still pair evidence with the wrong answer. Add a focused checker that reports whether each non-empty quote shares at least one normalized word with its answer.

Clean Up Your Resources

Choose whether to preserve your local project, pause your tools, or remove the project entirely. Everything runs locally, so there are no ongoing costs.

Resources you used:

Local ~/site-survey folder containing the complete Git repository plus every project artifact. This includes the accepted core evidence plus quote_overlap.py and any optional output appended to check-out.txt.

Keep everything running

No action needed. Choose this if you want to retain the accepted core for another teach-back or review.

Keep ~/site-survey unchanged.
Retain the local Ollama model gemma3:4b for future extraction runs.
Use the v1.0-survey tag when you need to return to the accepted core state.
Review check-out.txt when you need the recorded core evidence or optional overlap results.

Your repository remains ready for another demonstration. The restored trusted outputs stay separate from the retained must-fail fixture.

Pause - I'll come back to this later

Shut down active tools to free memory. Every project file remains available on disk.

Close the terminal after its current command finishes.
Close your existing editor.
Close your existing AI client.
Close Ollama after its local model work finishes.
Keep ~/site-survey on disk.

Your close-out commit remains intact. The v1.0-survey tag remains ready for your next teach-back.

Delete - I don't want to use this again

Remove the local project when you no longer need its evidence. The Ollama model remains outside the project folder.

Before You Delete

Deleting the folder is permanent. A final evidence check gives you a safe stopping point before removal.

Confirm that you no longer need the repository or its accepted evidence.

Delete on macOS and Linux

Move to your home folder by running this command:

cd ~

Why Leave the Project Folder?

This command moves your terminal outside ~/site-survey. Your terminal can then remove the project folder cleanly.

Delete only the ~/site-survey folder by running this command:

rm -rf ~/site-survey

What Does This Command Delete?

This command permanently removes the repository plus every file stored inside ~/site-survey.

Your local model remains available because it is stored outside the project folder.

Check your home folder in your file browser.

You should no longer see site-survey in your home folder. That confirms the local repository plus its evidence are gone.

Folder Still Present?

A program may still be using a file inside the project folder. Close your terminal or editor before retrying the removal command.

Check that the command targets ~/site-survey exactly.

Help me diagnose why the project folder remains.

Delete on Windows

Move to your home folder in PowerShell by running this command:

cd ~

Why Leave the Project Folder?

This command moves PowerShell outside ~/site-survey. PowerShell can then remove the project folder cleanly.

Delete only the ~/site-survey folder by running this command:

Remove-Item ~/site-survey -Recurse -Force

What Does This Command Delete?

This command permanently removes the repository plus every file stored inside ~/site-survey.

Your local model remains available because it is stored outside the project folder.

Check your home folder in File Explorer.

You should no longer see site-survey in your home folder. That confirms the local repository plus its evidence are gone.

Folder Still Present?

Windows may keep a project file in use while another program has it open. Close your terminal or editor before retrying the removal command.

Check that the command targets ~/site-survey exactly.

Help me diagnose why the project folder remains.

Nice Work!

You made it! You built a local discovery extractor that turns four customer calls into grounded scope evidence.

You've learned how to:

Keep four customer calls on your machine with a local Ollama model. Use Pydantic to enforce a six-field schema contract. Check each grounded quote against its source transcript.
Generate a six-section prd.md plus a three-line red-flags.md using exactly three deterministic flag rules. Preserve t4 as one clean call. Prove the checks with a must-fail fixture. The fixture catches the invented owner twice. You then restore all nine tests.
Create an answer-first close-out with claims tied to design documents or recorded runs. Apply a traceability score to the first draft. Strike every unsupported claim. Defend all three decision records aloud. Tag the accepted core as v1.0-survey.
Secret Mission: Add an answer-quote overlap check for every non-empty quote. Treat NO OVERLAP as a warning. Keep the core acceptance rules unchanged.

Ready to quiz yourself?