Govern Parallel AI Agents for SCADA

Direct 5 parallel AI agents through a full INCOSE Vee lifecycle for ICS.

Introduction

30 Second Summary

When a city upgrades the software controlling its water treatment plant, people's safety depends on every requirement being traced, every test passing, and every risk being closed. Traditionally this takes months of sequential engineering work across multiple teams.

In this project, you will execute a full INCOSE Vee lifecycle for a municipal water-treatment SCADA upgrade in a single 90-minute sprint by directing five parallel AI agent teams through formal systems engineering gate reviews.

What You'll Build

A director-ready systems engineering program package with a running containerized ICS digital twin, automated KPI tracking, and quantified AI agent performance benchmarks.

By the end of this project, you'll have:

  • A complete INCOSE Vee artifact package with 100% bidirectional requirements traceability, ISO 15288 compliance mapping, and formal gate review records (SRR, PDR, CDR, TRR, FCA).
  • A running containerized digital twin using OpenPLC v4 and FUXA with verified Modbus/TCP communication captured in Wireshark.
  • A reusable AI Agent Governance Framework with agent SOWs, first-pass yield metrics, a performance radar chart, and risk burndown visualization.
  • Secret Mission: Implement a Suricata-based IDS that detects unauthorized Modbus writes to critical coils and integrate security KPIs into your TPM dashboard.

Are there any prerequisites?

This project assumes familiarity with systems engineering fundamentals (requirements decomposition, V&V concepts), basic Git workflows, and introductory Docker knowledge. You also need a ChatGPT Pro subscription ($100/mo) with Codex Desktop already installed.

Program Standup: Environment Setup and Agent Launch

A 90-minute systems engineering sprint lives or dies in the first 15 minutes. If your tools are not installed, your agents are not authorized, and your repository is not initialized, you will spend the entire session fighting infrastructure instead of governing engineering decisions.

In this step, you will stand up your entire program environment in parallel. You will verify every tool, pull Docker images in the background, initialize your monorepo, and launch all five Codex Desktop agent teams simultaneously.

In this step, get ready to:
  • Install and verify all tools (Docker Desktop, Python 3.12+, Git, Capella, OpenPLC Editor, Wireshark) with Docker image pulls running in the background.
  • Configure Codex Desktop with AGENTS.md governance and five TOML role configs, then launch all agent teams simultaneously.
  • Create the program directory structure, gate review templates, and baseline metrics while agents begin parallel execution.
Install and verify all tools in parallel

You need six tools installed and working before any agent can produce useful output. The strategy is to verify each one quickly and kick off the slow Docker image pulls as early as possible so they complete in the background.

  • Open three separate terminal windows.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open your first terminal.
  • Repeat to open two more terminal windows (three total).

In Terminal 1, verify Docker Desktop is running and check your version by running this command:

docker --version

✔️ I see a version number

Docker is ready. Keep this terminal open for the image pull.

ⓧ Command not found

  • Download Docker Desktop from docker.com/products/docker-desktop.
  • Run the installer and follow the prompts.
  • On Windows, ensure the WSL2 backend is selected during installation.
  • Launch Docker Desktop after installation and wait for the engine to start.
  • Run docker --version again to confirm.
  • In Terminal 1, start pulling the OpenPLC Runtime and FUXA images in the background by running these commands:
# Pull both container images - this takes several minutes
docker pull ghcr.io/autonomy-logic/openplc-runtime:latest &
docker pull frangoteam/fuxa:latest &

What does this command do?

The & at the end of each line sends the pull to the background.

These images contain the OpenPLC Runtime v4 (your virtual PLC) and FUXA (your HMI dashboard). They will continue downloading while you set up everything else.

In Terminal 2, verify Python and Git by running these commands:

python3 --version
git --version

✔️ I see Python 3.12+ and Git version numbers

Both tools are ready. You need Python 3.12 or higher for this project.

ⓧ One or both commands failed

  • For Python: download from python.org/downloads and install version 3.12 or higher.
  • On Windows, check Add Python to PATH during installation.
  • For Git: download from git-scm.com/downloads and install with default settings.
  • In Terminal 2, move to your Desktop and initialize the monorepo by running these commands:
cd ~/Desktop
git init wtp-scada-program && cd wtp-scada-program

What does this command do?

This creates a new folder called wtp-scada-program on your Desktop with a Git repository initialized inside it. All program artifacts will live here.

  • Confirm the repository was created by running:
git status

You should see a message like On branch main or On branch master with nothing to commit.

  • Now create your Python virtual environment and install all dependencies by running these commands:
# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install SE toolchain dependencies
pip install doorstop==3.1 pytest pymodbus==3.7.4 robotframework pytest-json-report

What does this install?

  • doorstop==3.1 manages requirements with Git-based bidirectional traceability.
  • pytest runs your V&V test suite against the digital twin.
  • pymodbus==3.7.4 provides the Modbus/TCP client for PLC communication.
  • robotframework handles higher-level acceptance tests.
  • pytest-json-report exports test results as JSON for the TPM dashboard.

Windows activation command

On Windows, use .venv\Scripts\activate instead of source .venv/bin/activate.

Seeing installation errors?

  • If doorstop fails, verify your Python is 3.10+ (required for Doorstop 3.1).
  • If pip is not found, run python3 -m pip install instead.

Help me troubleshoot my pip install errors.

In Terminal 3, verify your remaining desktop tools.

  • Verify Eclipse Capella 7.0.1 is installed by opening it.
  • Press Cmd+Space (macOS) or the Windows key (Windows), type Capella, and press Enter.
  • Capella takes 30-60 seconds on first launch because it is a Java/Eclipse application requiring significant memory.
  • Once the workspace selector appears, click Launch to confirm it loads. Then close it.

Why Capella?

Capella implements the Arcadia method (registered as AFNOR Z67-140). Its phases map directly to the left side of the INCOSE Vee decomposition: Operational Analysis, System Analysis, Logical Architecture, and Physical Architecture.

  • Verify OpenPLC Editor v4 is installed by opening it.
  • Press Cmd+Space (macOS) or the Windows key (Windows), type OpenPLC Editor, and press Enter.
  • Confirm the editor window opens, then close it.
  • Verify Wireshark is installed by running this in Terminal 3:
wireshark --version

✔️ I see a version number

All desktop tools verified. You are ready to proceed.

ⓧ Command not found or tools missing

Help me install these SE tools.

Configure Codex Desktop and launch all 5 agent teams

Your tools are verified and images are pulling in the background. Now you will create the governance framework that authorizes and constrains your five parallel agent teams.

Why governance before agents?

In systems engineering, no work begins without a work authorization. AGENTS.md is your program's Work Authorization Document. It defines what each agent team is allowed to produce, what files they can touch, and what quality standards they must meet.

Without this, agents will produce overlapping artifacts, overwrite each other's work, and violate your configuration management plan.

  • In Terminal 2 (which should be in the wtp-scada-program folder), create the governance directory by running:
mkdir -p governance
  • Create the AGENTS.md work authorization document. Open a new file at governance/AGENTS.md in your text editor and paste the following:
# Agent Governance Framework - WTP SCADA Program

## Program Authority
Principal Systems Engineer (YOU) holds final approval on all artifacts.
No agent output is baselined without gate review approval.

## Work Authorization

| Team | Agent ID | Scope | Output Directory |
|------|----------|-------|------------------|
| Requirements | Agent 1 | ConOps, Stakeholder Needs, System/Subsystem Reqs, Quality Scorecard | /docs, /reqs |
| Architecture | Agent 2 | Capella models, ICD, Digital Thread Visualization | /architecture |
| Implementation | Agent 3 | docker-compose.yml, PLC Structured Text, FUXA config | /implementation |
| V&V | Agent 4 | VCRM, pytest suite, Robot Framework tests | /tests |
| Governance | Agent 5 | PHA, Security Mapping, Trade Studies, Migration Plan, ISO 15288 Matrix | /governance, /reports |

## Permission Boundaries
- Agents MUST NOT modify files outside their assigned output directory.
- Agents MUST NOT modify another team's artifacts without Principal SE approval.
- Agents MUST include Doorstop UIDs in all requirement references.
- Agents MUST use PlantUML for all diagram generation.

## Output Standards
- All documents: Markdown with YAML frontmatter (title, date, author, status).
- All requirements: Doorstop YAML format with links field populated.
- All code: Inline comments explaining purpose, not restating logic.
- All diagrams: PlantUML source committed alongside rendered SVG.
- Naming convention: lowercase-hyphenated for files, UPPERCASE-NNN for requirement UIDs.

## Quality Gate Criteria
- SRR: 100% STK-to-SYS trace, quality score avg >= 6/8.
- PDR: Architecture-to-requirements trace complete, ICD consistent.
- CDR: Implementation matches ICD register map, Modbus verified.
- TRR: 100% requirements-to-test coverage, all tests passing.
- FCA: Full bidirectional trace validated, all artifacts published.

What does this document do?

This is the single authority document that Codex agents reference when deciding what they can and cannot do. The permission boundaries prevent cross-contamination between teams. The output standards ensure all artifacts are mergeable at gate reviews.

  • Save governance/AGENTS.md.

Now create the five TOML role configuration files. Each one defines a Codex agent team's identity, reasoning effort, and deliverables.

  • Create governance/agent1_requirements.toml and paste the following:
[agent]
name = "Requirements Team"
role = "Requirements Engineer"
reasoning_effort = "high"
sandbox = "network-off"

[scope]
output_dirs = ["docs", "reqs"]
deliverables = [
    "ConOps with 3 operational scenarios (normal, degraded, emergency)",
    "Stakeholder Needs Document with regulatory trace (EPA AWIA, AWWA G430)",
    "System Requirements in Doorstop (prefix SYS)",
    "Subsystem Requirements in Doorstop (prefix SUB)",
    "Requirements Quality Scorecard (8 INCOSE attributes per requirement)"
]

[instructions]
prompt = """
You are the Requirements Engineering team for a municipal water treatment plant SCADA modernization.
Produce all deliverables in Doorstop YAML format under /reqs.
Every requirement must be scored against: atomic, complete, consistent, correct, feasible, necessary, unambiguous, verifiable.
Link every STK item to at least one SYS item. Link every SYS item to at least one SUB item.
Reference AGENTS.md for output standards.
"""
  • Create governance/agent2_architecture.toml and paste the following:
[agent]
name = "Architecture Team"
role = "Systems Architect"
reasoning_effort = "high"
sandbox = "network-off"

[scope]
output_dirs = ["architecture"]
deliverables = [
    "Capella Operational Analysis model",
    "Capella System Analysis model",
    "Capella Logical Architecture model",
    "Capella Physical Architecture model",
    "Digital Thread Visualization (PlantUML showing full trace chain)"
]

[instructions]
prompt = """
You are the Architecture team using Arcadia/Capella methodology.
Produce Operational Analysis first, then System Analysis.
All architecture elements must trace to at least one SYS requirement.
The Digital Thread diagram shows one complete chain: STK -> SYS -> SUB -> Capella function -> PLC variable -> test.
Export all diagrams as PlantUML source plus rendered SVG.
Reference AGENTS.md for output standards.
"""
  • Create governance/agent3_implementation.toml and paste the following:
[agent]
name = "Implementation Team"
role = "Control Systems Engineer"
reasoning_effort = "medium"
sandbox = "network-on"

[scope]
output_dirs = ["implementation"]
deliverables = [
    "docker-compose.yml with OpenPLC Runtime + FUXA + network segmentation",
    "Structured Text PLC program for water treatment process",
    "FUXA HMI configuration",
    "Modbus register map implementation"
]

[instructions]
prompt = """
You are the Implementation team building the ICS digital twin.
Produce a docker-compose.yml with Purdue Model network segmentation (Level 1 OT, Level 2 OT).
The PLC program must implement: pump control (coils 0-3), tank level monitoring (holding registers 0-3), emergency shutdown (coil 7).
All Modbus addresses must match the ICD register map produced by Agent 5.
Wait for PDR approval before finalizing implementation.
Reference AGENTS.md for output standards.
"""
  • Create governance/agent4_vv.toml and paste the following:
[agent]
name = "V&V Team"
role = "Verification and Validation Engineer"
reasoning_effort = "high"
sandbox = "network-on"

[scope]
output_dirs = ["tests"]
deliverables = [
    "Verification Cross-Reference Matrix (VCRM) with IADTM classification",
    "pytest test suite with Doorstop UID in every docstring",
    "Robot Framework acceptance tests",
    "Independent requirements quality peer review (25% sample)"
]

[instructions]
prompt = """
You are the V&V team. Every requirement must have at least one test.
Classify each verification method as: Inspection, Analysis, Demonstration, Test, or Modelling.
All pytest tests connect to OpenPLC via pymodbus at 172.25.1.10:502.
Include conftest.py with session-scoped Modbus client fixture.
Wait for CDR approval before executing tests against the live twin.
Reference AGENTS.md for output standards.
"""
  • Create governance/agent5_governance.toml and paste the following:
[agent]
name = "Governance Team"
role = "Program Governance Engineer"
reasoning_effort = "high"
sandbox = "network-off"

[scope]
output_dirs = ["governance", "reports", "docs"]
deliverables = [
    "Preliminary Hazard Analysis (8+ hazards with risk matrix)",
    "Security Control Mapping (NIST SP 800-82 Rev 3 to architecture)",
    "Trade Study Reports (2 key decisions with alternatives-evaluation-decision)",
    "Migration/Cutover Plan with phased approach and fallback criteria",
    "ISO/IEC/IEEE 15288 Compliance Matrix (process-to-artifact mapping)",
    "Risk Burndown data (JSON format for TPM dashboard)",
    "Interface Control Document with Modbus register map and Purdue zones",
    "Program Execution Playbook"
]

[instructions]
prompt = """
You are the Governance team responsible for program-level artifacts.
The PHA must identify 8+ hazards for a water treatment SCADA system with severity and likelihood ratings.
Security controls map NIST SP 800-82 Rev 3 recommendations to specific architecture elements.
The ISO 15288 matrix must map every technical process to a specific artifact in this program.
Risk burndown tracks open risks across gates: start at 8, target 0 by FCA.
Reference AGENTS.md for output standards.
"""

What are the TOML configs for?

Each TOML file defines a Codex custom role. When you open a Codex thread with a specific role, it inherits the reasoning effort level, sandbox permissions, and developer instructions from that file.

The sandbox = "network-off" setting prevents document-only agents from making network calls. The sandbox = "network-on" setting allows the Implementation and V&V teams to interact with Docker containers.

  • Save all five TOML files.
  • Confirm the files exist by running:
ls governance/

You should see AGENTS.md and all five .toml files listed.

Now launch all five agent teams in Codex Desktop simultaneously.

  • Press Cmd+Space (macOS) or the Windows key (Windows), type Codex, and press Enter to open Codex Desktop.
  • Click New Thread to start your first agent.
  • Set the working directory to your wtp-scada-program folder on your Desktop.
  • Paste the contents of governance/agent1_requirements.toml's [instructions].prompt field as the opening message.
  • Add this line at the start of the message: Read governance/AGENTS.md for your work authorization and output standards..
  • Send the message to start Agent 1.
  • Immediately click New Thread again (do not wait for Agent 1 to finish).
  • Repeat the same process for Agent 2 through Agent 5, each with their respective TOML prompt content.
  • You should now have 5 parallel threads running in Codex Desktop.

Why launch all five at the same time?

Codex Desktop supports worktree isolation. Each thread operates independently without file conflicts. By launching all five simultaneously, you compress what would be sequential 4-hour work sessions into parallel 25-minute sprints.

This is the core execution model. Your serial path is governance decisions and gate reviews. Everything else runs concurrently.

Agent thread not starting?

  • Confirm Codex Desktop is connected to your ChatGPT Pro account (check the account icon in the top right).
  • Ensure the working directory is set to wtp-scada-program (not your home folder).
  • If you hit a rate limit, wait 60 seconds and retry. Pro tier includes substantial usage credits.

Help me troubleshoot my Codex Desktop agent launch.

Create the governance scaffolding while agents work

Your agents are now executing in parallel. While they produce artifacts, you will set up the remaining program infrastructure: directory structure, gate review templates, and the metrics baseline.

  • In Terminal 2, create the full program directory structure by running this command:
# Create all program directories
mkdir -p docs reqs architecture implementation tests governance/gate_review_templates reports

What does each directory hold?

  • docs/ holds ConOps, Stakeholder Needs, and the Executive Summary.
  • reqs/ holds Doorstop requirement YAML files with trace links.
  • architecture/ holds Capella model exports and PlantUML diagrams.
  • implementation/ holds the Docker Compose stack and PLC program.
  • tests/ holds pytest and Robot Framework suites.
  • governance/ holds AGENTS.md, role configs, gate records, PHA, and compliance artifacts.
  • reports/ holds the TPM dashboard and published outputs.
  • Confirm the directories exist by running:
ls

You should see architecture, docs, governance, implementation, reqs, reports, and tests listed.

  • Create the gate review record templates. Open a new file at governance/gate_review_templates/srr_template.md and paste the following:
# Gate Review Record: [GATE NAME]

## Review Metadata
- **Gate**: [SRR | PDR | CDR | TRR | FCA]
- **Date**: YYYY-MM-DD HH:MM
- **Reviewer**: Principal Systems Engineer
- **Duration**: X minutes

## Entry Criteria
- [ ] All required artifacts delivered
- [ ] Doorstop validation passes
- [ ] Trace links complete

## Review Items
| Item | Status | Finding | Action |
|------|--------|---------|--------|
| Artifact completeness | PASS/FAIL | | |
| Trace coverage | PASS/FAIL | | |
| Quality score threshold | PASS/FAIL | | |
| Risk posture acceptable | PASS/FAIL | | |

## Risk Burndown Status
- Open risks at this gate: X
- Risks closed since last gate: X
- Residual risks accepted: X

## Decision
- [ ] **PASS** - Proceed to next phase
- [ ] **CONDITIONAL PASS** - Proceed with actions
- [ ] **FAIL** - Rework required

## Actions
| # | Action | Owner | Due |
|---|--------|-------|-----|
| 1 | | | |

## Git Tag
`gate/[GATE]-passed` or `gate/[GATE]-failed`

What does this template do?

This is the formal record you will fill out at each gate review (SRR, PDR, CDR, TRR, FCA). It captures your engineering judgment: what passed, what failed, and what actions are required before proceeding.

The risk burndown section tracks hazard closure across gates. You will start with 8 open hazards at SRR and target 0 by FCA.

  • Save the template file.
  • Create copies for each gate by running these commands:
cd governance/gate_review_templates
cp srr_template.md pdr_template.md
cp srr_template.md cdr_template.md
cp srr_template.md trr_template.md
cp srr_template.md fca_template.md
cd ../..
  • Confirm the templates exist by running:
ls governance/gate_review_templates/

You should see five template files: srr_template.md, pdr_template.md, cdr_template.md, trr_template.md, and fca_template.md.

  • Create the agent metrics baseline. Open a new file at governance/agent_metrics.json and paste the following:
{
  "first_pass_yield_pct": 0,
  "rework_count": 0,
  "total_deliverables": 0,
  "teams": {
    "Agent 1 - Requirements": {
      "first_pass_yield": 0,
      "delivery_speed_score": 0,
      "trace_completeness": 0,
      "rework_cycles": 0,
      "output_volume_score": 0,
      "deliverables_accepted": 0,
      "deliverables_total": 5
    },
    "Agent 2 - Architecture": {
      "first_pass_yield": 0,
      "delivery_speed_score": 0,
      "trace_completeness": 0,
      "rework_cycles": 0,
      "output_volume_score": 0,
      "deliverables_accepted": 0,
      "deliverables_total": 5
    },
    "Agent 3 - Implementation": {
      "first_pass_yield": 0,
      "delivery_speed_score": 0,
      "trace_completeness": 0,
      "rework_cycles": 0,
      "output_volume_score": 0,
      "deliverables_accepted": 0,
      "deliverables_total": 4
    },
    "Agent 4 - V&V": {
      "first_pass_yield": 0,
      "delivery_speed_score": 0,
      "trace_completeness": 0,
      "rework_cycles": 0,
      "output_volume_score": 0,
      "deliverables_accepted": 0,
      "deliverables_total": 4
    },
    "Agent 5 - Governance": {
      "first_pass_yield": 0,
      "delivery_speed_score": 0,
      "trace_completeness": 0,
      "rework_cycles": 0,
      "output_volume_score": 0,
      "deliverables_accepted": 0,
      "deliverables_total": 8
    }
  }
}

What does this file track?

This is your quantified agent performance ledger. As you conduct gate reviews and accept or reject deliverables, you will update these metrics. The TPM dashboard reads this file to generate the Agent Performance Radar Chart.

The five axes of the radar chart (yield, speed, trace, rework, volume) let you compare team effectiveness at a glance.

  • Save governance/agent_metrics.json.
  • Verify all governance files are in place by running:
find governance/ -type f | sort

You should see AGENTS.md, all five .toml files, agent_metrics.json, and all five gate review template files.

  • Check that your Docker image pulls are progressing. In Terminal 1, run:
docker images | grep -E "openplc|fuxa"

If the pulls are complete, you will see both ghcr.io/autonomy-logic/openplc-runtime and frangoteam/fuxa listed. If still downloading, that is fine. They will be ready by Step 3.

Docker pull stuck or failing?

  • Confirm Docker Desktop is running (check the system tray icon).
  • If you see a network timeout, check your internet connection and retry with docker pull ghcr.io/autonomy-logic/openplc-runtime:latest.
  • On Windows with WSL2, ensure the Docker engine has started (the whale icon should be steady, not animating).

Help me fix my Docker pull issues.

Your program is standing. Five agent teams are executing concurrently, Docker images are pulling, and your governance scaffolding is ready to receive artifacts. Next up, you will monitor their output and conduct your first gate review: SRR.

Wave 1 Parallel Execution and SRR/PDR Gates

Your 5 agent teams are now executing in parallel. As Principal Systems Engineer, your job shifts from doing the work to governing it.

In this step, you will monitor all 5 agent deliverables as they arrive, then conduct two formal gate reviews (SRR and PDR) to baseline the program. These gates are the quality checkpoints that separate rigorous systems engineering from ad-hoc development.

In this step, get ready to:
  • Monitor and triage parallel agent deliverables from all 5 teams.
  • Conduct the SRR gate review on Agent 1's requirements baseline.
  • Conduct the PDR gate review on Agent 2's architecture output.
Monitor parallel agent output

All 5 Codex agent threads are working concurrently. Each team produces a distinct set of artifacts as defined in their SOW prompts.

Your role during this phase is to observe progress in the Codex Desktop review queue and triage any early issues before the formal gate reviews.

  • Switch between your 5 Codex agent threads in the Codex Desktop sidebar to monitor progress.
  • Watch for Agent 1 (Requirements) to deliver: docs/conops.md, docs/stakeholder_needs.md, Doorstop STK and SYS document sets in reqs/, and a Requirements Quality Scorecard.
  • Watch for Agent 2 (Architecture) to deliver: Capella Operational Analysis and System Analysis diagrams in architecture/capella/.
  • Watch for Agent 3 (Implementation) to deliver: docker-compose.yml and a Structured Text PLC program in implementation/.
  • Watch for Agent 4 (V&V) to deliver: VCRM template and pytest/Robot Framework scaffolding in tests/.
  • Watch for Agent 5 (Governance) to deliver: PHA, Security Control Mapping, Trade Studies, Migration Plan, ISO 15288 Compliance Matrix, and risk_burndown.json baseline in governance/.

What is the INCOSE Vee Model?

The INCOSE Vee Model is a systems engineering lifecycle where the left side decomposes needs into requirements, architecture, and design, while the right side integrates and verifies back up to stakeholder validation.

Gates (SRR, PDR, CDR, TRR, FCA) are formal decision points between phases. Each gate has entry/exit criteria that artifacts must pass before the program advances.

What are the 8 INCOSE quality attributes?

Agent 1 self-scores every requirement against: atomic, complete, consistent, correct, feasible, necessary, unambiguous, and verifiable. Each attribute gets a binary pass/fail, producing a score out of 8.

A project average of 6+/8 indicates the requirements baseline is mature enough to pass SRR. Below that threshold, Agent 1 gets a rework directive.

While agents work, keep your terminal visible. Docker images should still be pulling in the background from the previous step.

Conduct the SRR gate review

The System Requirements Review (SRR) is your first formal gate. It confirms that stakeholder needs have been properly decomposed into verifiable system requirements with complete traceability.

You have 5 minutes for this gate. The criteria are: Doorstop validation passes, every STK item links to at least one SYS item, and the average quality score is 6+/8.

  • Once Agent 1's output appears in the Codex review queue, activate Plan Mode on the Agent 1 thread to review trace completeness in read-only mode.
  • Verify the Doorstop documents are valid by running this command in your terminal:
cd ~/Desktop/wtp-scada-program
doorstop

What does doorstop validation check?

Running doorstop with no arguments validates the entire requirements tree. It checks that all links resolve, no orphan items exist, and document hierarchies are consistent.

  • Verify the output shows no warnings or errors. Every STK item should link to at least one SYS item.
  • Open governance/requirements_quality_scorecard.md and confirm the average quality score across all requirements is 6 or above out of 8.
  • Open your SRR gate review template in governance/gate_reviews/SRR_record.md and record your pass/fail decision with a timestamp.
  • Record the risk burndown baseline: 8 open hazards at SRR.

What if Agent 1's output fails the gate?

If Doorstop validation fails or the quality score average is below 6/8, send a rework directive to Agent 1's Codex thread. Describe the specific gaps (e.g., "STK-003 has no SYS link" or "SUB-007 scored 4/8, fails unambiguous and verifiable").

Track this rework cycle in agent_metrics.json as a failed first-pass yield for Agent 1.

  • Once the gate passes, commit the SRR baseline and tag it by running these commands:
git add .
git commit -m "SRR baseline"
git tag gate/SRR-passed

Why tag gate decisions in Git?

Git tags create immutable reference points in your program history. An auditor (or CSEP assessor) can run git diff gate/SRR-passed..gate/PDR-passed to see exactly what changed between gates. This is your configuration management evidence.

  • Confirm the tag was created by running:
git tag -l "gate/*"

You should see gate/SRR-passed in the output.

Doorstop validation showing errors?

If you see "unresolved link" errors, it means Agent 1 created link references to UIDs that don't exist yet. Check that both the reqs/STK/ and reqs/SYS/ directories contain .doorstop.yml files and that the UID prefixes match what Agent 1 referenced in its links.

If you see "document not found" errors, Agent 1 may not have run doorstop create to initialize the document sets. Send a rework directive with the exact error message.

Why is my Doorstop validation failing after Agent 1 delivered its output?

Conduct the PDR gate review

The Preliminary Design Review (PDR) confirms that the system architecture addresses all baselined requirements and that interfaces between components are well-defined.

This gate focuses on Agent 2's Arcadia architecture output and Agent 5's Interface Control Document consistency.

  • Once Agent 2's output appears, activate Plan Mode on the Agent 2 thread to review the architecture-to-requirements trace.
  • Verify that every system function in the Capella System Analysis traces back to at least one SYS requirement in Doorstop.
  • Cross-check Agent 5's Interface Control Document against Agent 2's logical architecture interfaces. The Modbus register map in the ICD must align with the physical interfaces defined in Agent 2's architecture.
  • Open governance/gate_reviews/PDR_record.md and record your pass/fail decision with a timestamp.
  • Update the risk burndown: record 2 hazards closed by architecture decisions (6 remaining).

What does architecture-to-requirements trace mean?

Every function or component in the architecture must exist because a requirement demands it. If you find architecture elements with no requirement parent, it is scope creep. If you find requirements with no architecture element addressing them, it is a coverage gap.

Both directions matter. This is bidirectional traceability.

  • Update governance/risk_burndown.json to reflect the PDR status. The file should show 8 open at SRR and 6 open at PDR.
  • Commit and tag the PDR gate by running these commands:
git add .
git commit -m "PDR baseline - architecture approved"
git tag gate/PDR-passed
  • Confirm both tags exist by running:
git tag -l "gate/*"

You should see both gate/SRR-passed and gate/PDR-passed in the output.

  • Immediately direct Agent 3 to align its implementation with the approved architecture. In the Agent 3 Codex thread, send a message referencing the PDR-approved ICD and Purdue Model network segmentation.
  • Update governance/agent_metrics.json with first-pass yield data for all 5 agent teams based on whether they required rework directives during Wave 1.

Architecture doesn't trace cleanly to requirements?

If Agent 2's architecture has functions without requirement parents, send a rework directive asking Agent 2 to either justify the function (it may trace to a non-functional requirement Agent 1 missed) or remove it as scope creep.

If the ICD and architecture interfaces conflict, direct Agent 5 to re-baseline the ICD against Agent 2's approved output.

How do I resolve conflicts between my architecture agent's output and the interface control document?

Why this approach over sequential gate reviews?

Traditional SE programs run gates weeks apart because human teams need time between phases. With parallel AI agents, all left-side Vee artifacts arrive within minutes of each other. This lets you conduct SRR and PDR back-to-back while context is fresh.

The engineering rigor is identical. The calendar time collapses because execution parallelism eliminates idle wait between teams.

Your program now has a baselined requirements set and an approved architecture. Next up, you will launch the digital twin and conduct the CDR gate as implementation and V&V agents converge on a running system.

Wave 2 Integration and CDR Gate

Your left-side Vee artifacts are baselined. SRR and PDR gates are passed. Now you bring the system to life.

In this wave, you launch the containerized ICS digital twin, verify Modbus/TCP communication on the wire, and conduct your Critical Design Review. This is where the architecture meets reality. If the register map in your ICD does not match what the PLC actually exposes, you catch it here before V&V begins.

In this step, get ready to:
  • Launch the digital twin and verify live SCADA data on the HMI.
  • Conduct the CDR gate with Wireshark-verified Modbus communication.
  • Review and accept Agent 5 governance artifacts, then direct Agent 4 to begin V&V.
Launch the digital twin

The Docker images were pulling in the background during Wave 1. Both OpenPLC Runtime v4 and FUXA should be ready. Time to bring your water treatment plant online.

  • Navigate to your implementation directory and start both containers by running this command:
cd implementation
docker compose up -d

What does this command do?

The -d flag runs containers in detached mode so you get your terminal back. Docker Compose reads docker-compose.yml and launches both the PLC runtime (on the ot-level1 network at 172.25.1.10) and the FUXA HMI (bridging both ot-level1 and ot-level2 networks).

  • Verify both containers are healthy by running:
docker ps
  • You should see two containers: wtp-plc-01 and wtp-hmi-01, both with status Up.

Containers not starting?

If you see port already allocated, another process is using port 502 or 8443. Stop it with docker stop or change the host port in your compose file.

If Docker Desktop is not running, launch it first. On Windows, ensure the WSL2 backend is active.

Help me troubleshoot Docker containers that won't start for my SCADA lab

Now connect the OpenPLC Editor to the running runtime and upload the PLC program.

  • Open OpenPLC Editor v4 on your desktop.
  • Click Connect in the toolbar and enter localhost:8443 as the runtime address.
  • Open the PLC program file at implementation/plc_program.st (produced by Agent 3).
  • Click Upload to transfer the program to the runtime container.
  • Click Start to begin PLC execution.

Do not open localhost:8443 in a browser

OpenPLC Runtime v4 is headless. There is no web interface on port 8443 like v3 had. You must use the OpenPLC Editor desktop application to connect, upload, and control the runtime.

Next, configure FUXA to read live data from the PLC via Modbus.

  • Open your browser and navigate to http://localhost:1881 to access the FUXA HMI builder.
  • Click Settings (gear icon in the left sidebar).
  • Select DAQ/Data Sources and add a new data source.
  • Choose Modbus RTU/TCP as the type.
  • Enter 172.25.1.10 as the host and 502 as the port.
  • Click Save and verify the connection status shows Connected.

Why connect via IP instead of localhost?

FUXA runs inside a Docker container on the ot-level1 bridge network. From FUXA's perspective, the PLC is a separate host at 172.25.1.10. This mirrors a real Purdue Model Level 1 network where HMI and PLC communicate over a dedicated OT segment.

  • Navigate to the FUXA Editor view and add a tag mapped to Holding Register 0 (tank level).
  • Verify the tag shows a live value updating in real time.

You should see numeric values changing on the HMI display. This confirms your digital twin is running with end-to-end Modbus communication.

FUXA shows no data or connection refused?

Confirm the PLC program is actually running in OpenPLC Editor (status should show Running). If the program is not started, FUXA will connect but read all zeros.

Check that FUXA is using the container IP 172.25.1.10, not localhost. From inside FUXA's container, localhost refers to itself.

Help me debug FUXA not connecting to my OpenPLC runtime over Modbus

Conduct the CDR gate with Wireshark verification

A Critical Design Review verifies that the detailed design matches the approved architecture. For an ICS system, this means proving that actual network traffic matches the Interface Control Document.

  • Open Wireshark.
  • In the interface list, select the Docker bridge interface for the ot-level1 network (on Windows this appears as vEthernet or a Hyper-V interface).
  • Apply the display filter tcp.port == 502 to isolate Modbus traffic.
  • Click Start Capturing (the blue shark fin icon in the toolbar).

You should see Modbus/TCP frames appearing as FUXA polls the PLC registers. Look for Read Holding Registers and Read Coils function codes in the packet details.

What am I looking for in the capture?

Each Modbus frame shows a function code (FC 01 = Read Coils, FC 03 = Read Holding Registers, FC 05 = Write Single Coil). Confirm you see traffic from 172.25.1.20 (FUXA) to 172.25.1.10 (PLC) with register addresses matching your ICD.

The key verification: coil address 0 maps to pump 1, coil address 7 maps to EMERGENCY_STOP, and holding register address 0 maps to tank level. If these match the ICD that Agent 5 produced, the implementation is correct.

  • Let the capture run for 30 seconds to collect a representative sample.
  • Stop the capture and save it as reports/modbus_capture.pcapng for your evidence package.

Now formally record the CDR gate decision.

  • Open Agent 5's ICD output at docs/icd.md and verify the Modbus register map matches what you observed in Wireshark (coil 0 = pump1, coil 1 = pump2, coil 7 = EMERGENCY_STOP, HR 0 = tank_level 0-10000).
  • Verify subsystem requirements in reqs/SUB/ trace to the PLC program variables by running:
doorstop
  • Confirm no trace warnings appear. Every SUB item should link to a parent SYS item.
  • Update governance/risk_burndown.json to reflect 2 more hazards closed by implementation controls (4 remaining total).
  • Create the CDR gate review record at governance/gate_reviews/CDR_record.md documenting: gate passed, Wireshark evidence confirmed, register map matches ICD, risk burndown updated.
  • Commit and tag by running these commands:
git add .
git commit -m "CDR baseline - digital twin verified"
git tag gate/CDR-passed

What does the CDR gate prove?

The CDR confirms three things: (1) the physical implementation matches the architecture, (2) actual network traffic matches the interface specification, and (3) subsystem requirements are traceable to implementation details. This is your evidence that the system was built as designed.

  • Direct Agent 4 to begin executing tests against the live digital twin. In Codex Desktop, send Agent 4 this prompt: "CDR passed. Digital twin is live at 172.25.1.10:502. Execute the V&V test suite against the running PLC. Finalize tests/test_modbus_verification.py with tests tracing to SYS and SUB requirement UIDs."

What should Agent 4 produce?

Agent 4 will produce a test file covering pump control (SYS-001, SUB-001), tank level monitoring (SYS-003, SUB-005), and emergency shutdown logic (SYS-007, SUB-012). Each test method includes the requirement UID in its docstring for trace audit.

Review and accept Agent 5 governance artifacts

While Agent 4 builds the V&V suite, you have a window to review the remaining governance deliverables. Agent 5 has produced five artifacts that need your acceptance before the program can proceed to TRR.

  • Open governance/pha.md and verify it contains 8 or more identified hazards, each with a mitigation mapped to a specific requirement UID.
  • Open governance/security_control_mapping.md and verify 10 or more NIST SP 800-82 Rev 3 controls are mapped to architecture elements.
  • Open governance/trade_studies.md and confirm both trade studies follow the alternatives-evaluation-decision structure (criteria weighted, options scored, decision justified).
  • Open docs/migration_plan.md and verify it defines phased cutover stages with explicit fallback criteria for each phase.
  • Open governance/iso15288_compliance_matrix.md and verify every ISO/IEC/IEEE 15288 technical process maps to a specific artifact in your program package.

Use Plan Mode for rapid review

In Codex Desktop, switch Agent 5's thread to Plan Mode before asking it to walk you through each artifact. Plan Mode reads without executing, so you can discuss acceptance criteria before approving any changes.

  • For each artifact, record your accept/rework decision. If any artifact fails review, send a rework directive to Agent 5 with specific deficiencies noted.
  • Update governance/agent_metrics.json with the first-pass yield for this wave. Record how many deliverables passed on first review versus required rework.

What is first-pass yield?

First-pass yield is the percentage of deliverables accepted without rework on their first submission. This is your primary agent performance metric. A yield above 80% means your SOWs and role configs are well-calibrated. Below 60% signals that agent instructions need tightening.

  • Check Agent 4's thread in Codex Desktop. Once Agent 4 delivers the test file, review tests/test_modbus_verification.py to confirm every test method references a Doorstop requirement UID in its docstring.

The test file Agent 4 produces should look like this:

"""
V&V Test Suite - Modbus Register Verification
Each test traces to a specific Doorstop requirement UID.
"""
import pytest


class TestPumpControl:
    """Verify pump control coil operations. Traces: SYS-001, SUB-001."""

    def test_pump1_coil_readable(self, modbus_client):
        """Requirement: SYS-001 - System shall control inlet pump via discrete output."""
        result = modbus_client.read_coils(address=0, count=1)
        assert not result.isError(), f"Failed to read pump 1 coil: {result}"

    def test_pump1_coil_writable(self, modbus_client):
        """Requirement: SUB-001 - PLC shall actuate pump 1 via Modbus coil address 0."""
        result = modbus_client.write_coil(address=0, value=True)
        assert not result.isError(), f"Failed to write pump 1 coil: {result}"
        readback = modbus_client.read_coils(address=0, count=1)
        assert readback.bits[0] is True


class TestTankLevel:
    """Verify tank level register operations. Traces: SYS-003, SUB-005."""

    def test_tank_level_readable(self, modbus_client):
        """Requirement: SYS-003 - System shall monitor treated water tank level."""
        result = modbus_client.read_holding_registers(address=0, count=1)
        assert not result.isError(), f"Failed to read tank level: {result}"

    def test_tank_level_range(self, modbus_client):
        """Requirement: SUB-005 - Tank level register shall report 0-10000 (0.0-100.0%)."""
        result = modbus_client.read_holding_registers(address=0, count=1)
        assert not result.isError()
        value = result.registers[0]
        assert 0 <= value <= 10000, f"Tank level {value} out of range 0-10000"


class TestEmergencyShutdown:
    """Verify emergency shutdown logic. Traces: SYS-007, SUB-012."""

    def test_estop_coil_exists(self, modbus_client):
        """Requirement: SYS-007 - System shall provide emergency shutdown capability."""
        result = modbus_client.read_coils(address=7, count=1)
        assert not result.isError(), f"Failed to read E-STOP coil: {result}"

    def test_estop_halts_pumps(self, modbus_client):
        """Requirement: SUB-012 - E-STOP activation shall de-energize all pump coils within 1 scan cycle."""
        modbus_client.write_coil(address=0, value=True)
        modbus_client.write_coil(address=1, value=True)
        modbus_client.write_coil(address=7, value=True)
        import time
        time.sleep(0.5)
        pumps = modbus_client.read_coils(address=0, count=2)
        assert not pumps.bits[0], "Pump 1 still running after E-STOP"
        assert not pumps.bits[1], "Pump 2 still running after E-STOP"
        modbus_client.write_coil(address=7, value=False)

What does this test suite verify?

  • TestPumpControl verifies that pump 1 coil (address 0) can be read and written via Modbus, tracing to SYS-001 and SUB-001.
  • TestTankLevel verifies that the tank level holding register (address 0) is readable and within the specified 0-10000 range, tracing to SYS-003 and SUB-005.
  • TestEmergencyShutdown verifies the safety-critical E-STOP logic: activating coil 7 must de-energize all pump coils within one scan cycle, tracing to SYS-007 and SUB-012.
  • Verify tests/conftest.py contains the session-scoped Modbus client fixture connecting to 172.25.1.10:502.

✔️ Awesome, I've got everything!

Great. Double check you have both files saved in your tests/ directory.

ⓧ I'd like to double check the full code

Here is the complete tests/conftest.py:

import pytest
from pymodbus.client import ModbusTcpClient


@pytest.fixture(scope="session")
def modbus_client():
    """Create a Modbus TCP client connected to the OpenPLC runtime."""
    client = ModbusTcpClient(host="172.25.1.10", port=502, timeout=5)
    connected = client.connect()
    if not connected:
        pytest.skip("Cannot connect to OpenPLC runtime on 172.25.1.10:502")
    yield client
    client.close()

And the complete tests/test_modbus_verification.py:

"""
V&V Test Suite - Modbus Register Verification
Each test traces to a specific Doorstop requirement UID.
"""
import pytest


class TestPumpControl:
    """Verify pump control coil operations. Traces: SYS-001, SUB-001."""

    def test_pump1_coil_readable(self, modbus_client):
        """Requirement: SYS-001 - System shall control inlet pump via discrete output."""
        result = modbus_client.read_coils(address=0, count=1)
        assert not result.isError(), f"Failed to read pump 1 coil: {result}"

    def test_pump1_coil_writable(self, modbus_client):
        """Requirement: SUB-001 - PLC shall actuate pump 1 via Modbus coil address 0."""
        result = modbus_client.write_coil(address=0, value=True)
        assert not result.isError(), f"Failed to write pump 1 coil: {result}"
        readback = modbus_client.read_coils(address=0, count=1)
        assert readback.bits[0] is True


class TestTankLevel:
    """Verify tank level register operations. Traces: SYS-003, SUB-005."""

    def test_tank_level_readable(self, modbus_client):
        """Requirement: SYS-003 - System shall monitor treated water tank level."""
        result = modbus_client.read_holding_registers(address=0, count=1)
        assert not result.isError(), f"Failed to read tank level: {result}"

    def test_tank_level_range(self, modbus_client):
        """Requirement: SUB-005 - Tank level register shall report 0-10000 (0.0-100.0%)."""
        result = modbus_client.read_holding_registers(address=0, count=1)
        assert not result.isError()
        value = result.registers[0]
        assert 0 <= value <= 10000, f"Tank level {value} out of range 0-10000"


class TestEmergencyShutdown:
    """Verify emergency shutdown logic. Traces: SYS-007, SUB-012."""

    def test_estop_coil_exists(self, modbus_client):
        """Requirement: SYS-007 - System shall provide emergency shutdown capability."""
        result = modbus_client.read_coils(address=7, count=1)
        assert not result.isError(), f"Failed to read E-STOP coil: {result}"

    def test_estop_halts_pumps(self, modbus_client):
        """Requirement: SUB-012 - E-STOP activation shall de-energize all pump coils within 1 scan cycle."""
        modbus_client.write_coil(address=0, value=True)
        modbus_client.write_coil(address=1, value=True)
        modbus_client.write_coil(address=7, value=True)
        import time
        time.sleep(0.5)
        pumps = modbus_client.read_coils(address=0, count=2)
        assert not pumps.bits[0], "Pump 1 still running after E-STOP"
        assert not pumps.bits[1], "Pump 2 still running after E-STOP"
        modbus_client.write_coil(address=7, value=False)
  • Commit the finalized test suite by running:
git add tests/
git commit -m "V&V test suite finalized - traces to SYS and SUB UIDs"

Your digital twin is live, your CDR is passed, and Agent 4 has a test suite ready to execute against real hardware. Next up, you will run those tests, close the final gates, and generate your TPM dashboard with performance metrics.

Wave 3 Convergence: V&V Execution and Final Gates

Your digital twin is running. Your agents have produced requirements, architecture, implementation, and test artifacts across two waves of parallel execution. Now everything converges on a single question: does the system meet its requirements?

In this step, you execute the V&V suite against the live twin, conduct the TRR (Test Readiness Review) and FCA (Functional Configuration Audit) gates, and generate the automated TPM Dashboard that visualizes your entire program's health. This closes the right side of the INCOSE Vee.

In this step, get ready to:
  • Execute the V&V test suite and conduct the TRR gate with 100% trace coverage.
  • Conduct the FCA gate, publish requirements, and produce the Digital Thread Visualization.
  • Generate the TPM Dashboard with Agent Performance Radar Chart and Risk Burndown.
Conduct the TRR gate

The Test Readiness Review confirms that every requirement has a corresponding test and that the test infrastructure is ready to execute. Agent 4 has already produced the V&V suite mapped to Doorstop UIDs. Your job is to verify trace coverage, execute the tests, and make the gate decision.

  • In your Codex Desktop thread for Agent 4, switch to Plan Mode.
  • Direct Agent 4 to independently re-score a 25% random sample of requirements against the 8 INCOSE quality attributes (atomic, complete, consistent, correct, feasible, necessary, unambiguous, verifiable).
  • This serves as a peer review against Agent 1's self-scores from Wave 1.

Why peer review requirements quality?

Agent 1 self-scored its own requirements. A second agent performing an independent assessment creates a cross-check that mirrors how real SE programs use independent verification teams. If scores diverge significantly, it signals a requirements quality issue before testing.

While Agent 4 performs the peer review, verify trace coverage by checking that every requirement UID appears in at least one test docstring.

  • Verify trace completeness by running Doorstop validation:
doorstop
  • You should see all documents (STK, SYS, SUB) pass validation with no warnings about unlinked items.

Seeing unlinked requirements?

If Doorstop reports items without links, use spawn_agents_on_csv to batch-generate missing test cases. Create a CSV with one row per unlinked requirement UID, then spawn sub-agents to produce the tests.

Help me fix Doorstop trace gaps using spawn_agents_on_csv

  • Execute the V&V suite against the live digital twin by running:
pytest tests/ --json-report --json-report-file=tests/test_results.json

What does this command do?

This runs all tests in your tests/ directory against the live OpenPLC runtime via Modbus/TCP. The --json-report flag generates a machine-readable results file that your TPM Dashboard script will consume later.

  • You should see test results in your terminal. Note the pass count, fail count, and total. The file tests/test_results.json is now generated.

Tests failing with connection errors?

Verify your digital twin is still running with docker ps. The wtp-plc-01 container must be healthy and listening on port 502. If the container restarted, the PLC program may need to be re-uploaded and started via OpenPLC Editor.

Help me debug Modbus connection failures in my pytest suite

Now make the TRR gate decision based on the evidence.

  • Confirm 100% trace coverage: every STK, SYS, and SUB requirement has at least one test with its Doorstop UID in the test docstring.
  • Review Agent 4's peer review scores against Agent 1's self-scores. Record both in the Requirements Quality Scorecard.
  • Update the risk burndown: 3 more hazards are now verified closed by testing. 1 residual hazard is accepted by program authority (you).
  • Record the TRR gate decision in a new gate review record.
  • Create the gate review record and commit by running:
git add . && git commit -m "TRR passed - 100% trace coverage, V&V executed" && git tag gate/TRR-passed
  • Verify the tag was created by running git tag. You should see gate/TRR-passed in the list alongside your earlier gate tags.
Conduct the FCA (Functional Configuration Audit)

The FCA is the final configuration gate. It verifies that every artifact in the program is complete, internally consistent, and fully traceable from stakeholder need through to passing test. This is where you prove the bidirectional traceability chain is unbroken.

  • Run full Doorstop validation to confirm bidirectional trace integrity:
doorstop
  • Verify the output shows STK links to SYS, SYS links to SUB, and SUB links to TEST items. No orphaned items should appear.
  • Generate the Doorstop HTML publication by running:
doorstop publish all ./reports/requirements

What does this produce?

This generates a browsable HTML publication of all your requirements documents (STK, SYS, SUB) with clickable bidirectional trace links. It goes into reports/requirements/. This is the artifact a program reviewer would use to navigate your trace chain.

  • Confirm the output directory exists by running ls reports/requirements/. You should see HTML files for each document set.

Now direct Agent 2 to produce the Digital Thread Visualization.

  • In your Codex Desktop thread for Agent 2, send this directive: produce a single PlantUML diagram showing one complete trace chain from STK-001 through the corresponding SYS requirement, SUB requirement, Capella function, PLC variable, to the passing test.
  • Specify the output path as architecture/digital_thread.puml.
  • Direct Agent 2 to also export all Capella diagrams as SVGs (Operational Analysis, System Analysis, Logical Architecture, Physical Architecture).

What is the Digital Thread?

The Digital Thread is a single visualization that proves an unbroken chain exists from a stakeholder need all the way down to evidence that the need is satisfied. It is the most powerful artifact in your CSEP evidence package because it demonstrates traceability in a single glance.

  • Verify all gate review records are complete: SRR, PDR, CDR, and TRR records should each exist with passed status and timestamps.
  • Record the final risk burndown status: 0 open hazards (1 residual accepted by program authority).
  • Create the FCA gate record and commit by running:
git add . && git commit -m "FCA passed - full bidirectional trace verified, all artifacts complete" && git tag gate/FCA-passed
  • Run git tag to confirm you now have all five gate tags: gate/SRR-passed, gate/PDR-passed, gate/CDR-passed, gate/TRR-passed, and gate/FCA-passed.

Doorstop publish failing?

Make sure you are running the command from your wtp-scada-program root directory (not from inside reqs/). Doorstop needs to see the .doorstop.yml root file to resolve all document relationships.

Help me troubleshoot Doorstop publish errors

Generate the TPM Dashboard

The Technical Performance Measures Dashboard pulls data from every source in your program (Doorstop, pytest, Git, agent metrics, risk burndown) and renders a single-page HTML report. This is what the director reads to understand program health at a glance.

  • Before running the dashboard, update governance/agent_metrics.json with final first-pass yield data for all 5 agent teams. Each team needs scores for yield, delivery speed, trace completeness, rework cycles, and output volume.
  • Confirm governance/risk_burndown.json reflects the final state: gates=[SRR,PDR,CDR,TRR,FCA], open_risks=[8,6,4,1,0].
  • Generate the TPM Dashboard by running:
python tpm_dashboard.py

What does the dashboard show?

  • Trace Coverage: percentage of requirements with bidirectional links.
  • V&V Pass Rate: percentage of tests passing from the pytest JSON report.
  • Requirement Volatility: number of commits to reqs/ since the SRR baseline tag.
  • Agent First-Pass Yield: percentage of deliverables accepted without rework.
  • Agent Performance Radar Chart: 5-axis comparison (yield, speed, trace, rework, volume) across all teams.
  • Risk Burndown: visualization showing open risks dropping from 8 at SRR to 0 at FCA.
  • You should see TPM Dashboard generated: reports/tpm_dashboard.html in your terminal.
  • Open reports/tpm_dashboard.html in your browser to verify the KPI cards, radar chart table, and risk burndown bars all render correctly.

Dashboard showing 0% for trace coverage?

The get_trace_coverage() function runs doorstop publish all --format json from the reqs/ subdirectory. Make sure your Doorstop documents are initialized inside reqs/ and that the root .doorstop.yml is accessible from that path.

Help me debug my TPM dashboard showing zero coverage

Finally, set up automated regeneration and direct your last agent deliverable.

  • In Codex Desktop, set up a Codex Automation to re-run python tpm_dashboard.py daily. Results will land in your review queue each morning.
  • Direct Agent 5 to produce the Program Execution Playbook: a transferable guide documenting the parallel sprint methodology, agent SOW patterns, gate sequencing, and timing targets.

What are Codex Automations?

Codex Automations let you schedule agents to work in the background on a recurring basis. The results appear in your review queue, so you can approve or reject without being present when the work runs. This turns your TPM Dashboard into a living program health monitor.

All five gates are passed. Your Vee is closed. The program has full bidirectional traceability, a running digital twin, quantified agent metrics, and automated KPI reporting. Next up, you will assemble the final director-ready delivery package.

Final Assembly and Director Package

Every gate has passed. Every artifact is in its directory. The digital twin is running, tests are green, and your TPM dashboard shows the full story in numbers.

Now you shift from engineer to program authority. Your job in the final 5 minutes is to verify the package is audit-ready, produce the one artifact the director actually reads (the Executive Summary), and seal the program with a release tag. This is the difference between "I did some work" and "here is your evidence package."

In this step, get ready to:
  • Verify the complete delivery package is audit-ready with all artifacts in correct locations.
  • Produce a 1-page Executive Summary with KPIs and go/no-go recommendation.
  • Seal the program with a final Git commit and release tag.
Compile the final delivery package

A CSEP evidence package is only as good as its completeness. Before you sign off, you need to verify every artifact produced across all five agent teams is present, properly located, and internally consistent.

  • Verify the directory structure contains all required artifacts by running this command:
find . -type f \( -name "*.md" -o -name "*.json" -o -name "*.html" -o -name "*.svg" -o -name "*.puml" -o -name "*.st" -o -name "*.yml" \) | sort

What does this command do?

This finds every deliverable file in your monorepo and sorts them alphabetically. You are looking for the complete set of artifacts across all directories.

You should see files spanning all seven directories. Cross-reference against this checklist.

  • Confirm the following artifacts are present in the output:
  • Verify gate review records have timestamps by opening each file in governance/gate_reviews/ and confirming each contains a pass/fail decision with date.
  • Open reports/tpm_dashboard.html in your browser and confirm the Agent Performance Radar Chart and Risk Burndown visualization are both visible.
  • Verify the ISO 15288 Compliance Matrix in governance/iso15288_compliance_matrix.md maps every technical process to a specific artifact.
  • Accept the Program Execution Playbook from Agent 5. Check your Codex Desktop review queue for the completed docs/program_execution_playbook.md deliverable.
  • Review the playbook for completeness. It should describe how to replicate this parallel-agent Vee execution for any critical infrastructure domain.
  • If acceptable, approve the merge in Codex Desktop.

Missing artifacts in the file listing?

Check your Codex Desktop review queue. An agent may have completed work that has not been merged yet. Approve pending merges before continuing.

If an agent failed to deliver, use Plan Mode to review what went wrong, then re-issue the SOW for that specific artifact.

Help me figure out which agent deliverable is missing and how to recover it.

Produce the Executive Summary for the director

The Executive Summary is the single artifact that program authority reads to make a go/no-go decision. Everything else in the package supports this one page.

  • Create docs/executive_summary.md with the following structure. You can write this yourself or direct a Codex agent with access to the governance/ and reports/ directories to compile it:
# Executive Summary

## Program
City of Clearwater WTP SCADA Modernization

## Scope
Full INCOSE Vee lifecycle execution for municipal water-treatment SCADA upgrade.
Modernize Level 1 (PLC) and Level 2 (HMI) per Purdue Model segmentation.

## Timeline
90-minute parallel sprint achieved under AI agent governance.
Equivalent sequential effort: ~20 hours compressed via 5 concurrent agent teams.

## Key Performance Indicators

| KPI | Target | Achieved |
|-----|--------|----------|
| Requirements Trace Coverage | 100% | [value from dashboard]% |
| V&V Pass Rate | 95%+ | [value from dashboard]% |
| Requirements Quality Score (avg) | 6+/8 | [value from scorecard]/8 |
| Agent First-Pass Yield | 80%+ | [value from agent_metrics]% |

## Risk Posture
Burndown: 8 (SRR) > 6 (PDR) > 4 (CDR) > 1 (TRR) > 0 (FCA)
Residual: 1 risk accepted by program authority (documented in PHA).

## ISO/IEC/IEEE 15288 Compliance
All technical processes mapped to specific program artifacts.
See governance/iso15288_compliance_matrix.md for full mapping.

## Gate Review Summary
- SRR: PASSED
- PDR: PASSED
- CDR: PASSED
- TRR: PASSED
- FCA: PASSED

## Recommendation
**GO** - Program artifacts complete, all gates passed, digital twin verified,
residual risk documented and accepted. Package ready for CSEP submission.

What does each section accomplish?

The KPI table pulls real numbers from your TPM dashboard and agent metrics. Replace the bracketed placeholders with actual values from reports/tpm_dashboard.html and governance/agent_metrics.json.

The Risk Posture section shows the burndown from 8 open hazards at SRR to 0 at FCA. This demonstrates that the program actively managed risk at every gate, not just at the end.

The GO recommendation is the Principal SE's formal sign-off. A director reads this page and either approves submission or asks questions. Everything else in the package answers those questions.

  • Open reports/tpm_dashboard.html and copy the actual KPI values into the table placeholders in your executive summary.
  • Open governance/agent_metrics.json and copy the first_pass_yield_pct value into the Agent First-Pass Yield row.
  • Open governance/requirements_quality_scorecard.md and copy the average quality score into the Requirements Quality Score row.
  • Save docs/executive_summary.md.

Not sure what values to pull?

Open the TPM dashboard in a browser. The four KPI cards at the top show Trace Coverage, V&V Pass Rate, Requirement Volatility, and Agent First-Pass Yield. These are the exact numbers for your executive summary.

Help me pull the right KPI values from my TPM dashboard and agent metrics into the executive summary.

Seal the program with a final commit and release tag

In Configuration Management, the release tag is the formal baseline. Once you tag release/v1.0, this package is immutable. Any future changes require a new version.

  • Stage all files and create the final program commit by running these commands:
git add .
git commit -m "Program complete - all gates passed"

What does this commit represent?

This is your final configuration baseline. The commit message is deliberately simple because the gate tags already document the full review history. Anyone auditing the repository can trace from gate/SRR-passed through gate/FCA-passed to see the program's progression.

  • Create the release tag by running:
git tag release/v1.0
  • Push the program package to your remote repository by running:
git push origin main --tags
  • Verify the release tag exists by running:
git tag --list

You should see all six tags listed: gate/SRR-passed, gate/PDR-passed, gate/CDR-passed, gate/TRR-passed, gate/FCA-passed, and release/v1.0.

No remote configured?

If you see a "fatal: No configured push destination" error, you need to add a remote first. Create a new repository on GitHub, then run git remote add origin <your-repo-url> before pushing.

Help me set up a Git remote and push my program package.

The program is closed. Your INCOSE Vee lifecycle package is sealed at release/v1.0 with full bidirectional traceability, five passed gate reviews, a running digital twin, and quantified agent governance metrics. This is CSEP-grade evidence, produced in 90 minutes.

Secret mission

IDS Integration and Lessons Learned

Your digital twin is running and all gates have passed. But in the real world, a SCADA system without intrusion detection is a liability. In this secret mission, you will add a Suricata-based IDS container to your OT network, write custom rules that detect unauthorized Modbus writes to life-safety coils, verify those rules fire against a simulated attack, and integrate alert counts into your TPM dashboard as a security KPI. Then you will close the loop with a Lessons Learned report and process improvement recommendations.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with Docker containers and local files, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers: wtp-plc-01 (OpenPLC Runtime) and wtp-hmi-01 (FUXA HMI).
  • Docker volumes: plc-data and fuxa-data.
  • Docker networks: ot-level1 and ot-level2.
  • Git repository: wtp-scada-program on your Desktop.
  • Python virtual environment with doorstop, pytest, pymodbus, robotframework, and pytest-json-report.
  • Eclipse Capella workspace and project files.
  • OpenPLC Editor project files.
  • Wireshark capture files.

Keep everything running

No action needed. Choose this if you are still actively building on the program package or plan to extend it with additional Vee lifecycle artifacts.

  • Your Docker containers will keep running and consume some memory and CPU.
  • Your Git repository, Capella workspace, and all generated artifacts remain on disk.
  • The Codex Desktop automations (TPM dashboard generation) will continue running on schedule if configured.

Pause - I'll come back to this later

Shut down the running containers to free up memory and CPU. Your files and volumes stay intact for when you return.

  • Stop the Docker containers by running docker compose down from your wtp-scada-program directory. This stops containers but preserves volumes.
  • Disable any Codex Automations you set up so they do not run in the background.
  • To resume later, run docker compose up -d from the same directory.

Delete - I don't want to use this again

Remove all project containers, volumes, networks, and local files. This is irreversible.

  • Stop and remove containers, volumes, and networks by running docker compose down -v from your wtp-scada-program directory.
  • Remove the Docker images by running docker rmi ghcr.io/autonomy-logic/openplc-runtime:latest frangoteam/fuxa:latest.
  • Delete the entire wtp-scada-program folder from your Desktop.
  • Deactivate and delete the Python virtual environment.
  • Remove the Capella workspace if you no longer need it.
  • Delete any Wireshark capture files you saved outside the project folder.

That's a Wrap!

That's a Wrap!

You just executed a full INCOSE Vee lifecycle for critical infrastructure under parallel AI agent governance in 90 minutes. This is your CSEP evidence package, ready to put in front of the director.

You've learned how to:

  • Produce a complete INCOSE Vee lifecycle artifact package with 100% bidirectional requirements traceability managed in Doorstop and validated through formal gate reviews (SRR, PDR, CDR, TRR, FCA).
  • Deploy a containerized ICS digital twin using OpenPLC and FUXA with verified Modbus/TCP communication and automated V&V test execution.
  • Build a reusable AI Agent Governance Framework that directs 5 parallel Codex agent teams with quantified performance metrics, a TPM Dashboard with radar chart and risk burndown, and full ISO/IEC/IEEE 15288 compliance mapping.
  • Secret Mission: Integrate a Suricata-based IDS detecting unauthorized Modbus writes, feed security KPIs into the TPM dashboard, and produce a Lessons Learned report with process improvement recommendations.

Ready to quiz yourself?