Build an AI Cluster Go/No-Go Validator

Build a PowerShell validator for AI cluster evidence and go/no-go reports.

Introduction

30 Second Summary

A green status can make a newly built system feel ready for handover. Hidden faults can turn that confidence into an expensive surprise.

In this project, you will build a Windows PowerShell validator that assesses a fictional four-node AI cluster against the wider NCP-AII bring-up sequence. You will experience an unsafe GO from Slurm-only evidence before broader gates expose the correct NO-GO.

What You'll Build

Picture yourself handing operations a clear NO-GO where every failed gate traces back to specific node evidence.

By the end of this project, you'll have:

  • A repeatable cluster acceptance workflow that evaluates control-plane signals, server bring-up, physical validation, plus cluster-test evidence.
  • A defensible go/no-go decision that exposes four hidden faults behind an all-IDLE Slurm view.
  • Shareable handover evidence in a filterable CSV findings register plus a readable Markdown acceptance report.
  • Secret Mission: Correct the four faulty evidence fields so the unchanged validator returns GO with zero failed findings.

Are there any prerequisites?

You need a Windows computer. Strong Slurm experience is helpful. No GPU hardware or paid service is required.

Before We Start

Before the hands-on work begins, commit to building a validator for a fictional AI cluster handover. Your validator treats an IDLE Slurm node as one scheduler signal within evidence covering GPU inventory, BMC access, firmware alignment, topology, and validation tests.

Set Up the Windows Validation Workspace

An IDLE node is only one input to an NCP-AII acceptance decision. The same evidence pack must also carry the infrastructure facts that Slurm cannot prove.

Windows PowerShell ISE gives you a zero-cost place to run every later gate. The fictional CSV snapshot keeps this lab independent of cluster hardware.

In this step, get ready to:
  • Verify that the 64-bit Windows PowerShell ISE session uses version 5.1 with the Desktop edition.
  • Create the workspace folder structure with its evidence snapshot.
  • Import the four-node snapshot for inspection.
Verify Windows PowerShell ISE 5.1

The version table proves which PowerShell host is executing your commands. This project uses the built-in 5.1 Desktop session so every later result comes from the same environment.

  • Press the Windows key to open the search bar.
  • Type Windows PowerShell ISE into the search bar.
  • Select the 64-bit Windows PowerShell ISE result.
  • Check the current session by running this command in the Console Pane:
$PSVersionTable

What Does This Check Show?

  • The PSVersion entry identifies the version running in the current session.
  • The PSEdition entry identifies the PowerShell edition.
  • This project expects PSVersion to begin with 5.1.
  • This project expects PSEdition to show Desktop.

✔️ I see the expected Windows PowerShell session

Your output shows PSVersion beginning with 5.1.

That check is complete. Your 64-bit Desktop session is ready for every later acceptance gate.

ⓧ I see a different PowerShell session

The project needs the 64-bit Windows PowerShell ISE host because its exercises are pinned to 5.1 Desktop.

  • Close the current PowerShell window.
  • Press the Windows key to open the search bar.
  • Type Windows PowerShell ISE into the search bar.
  • Select the 64-bit Windows PowerShell ISE result.
  • Confirm the corrected session by running this command in the Console Pane:
$PSVersionTable

What Confirms the Correct Session?

Look for PSVersion beginning with 5.1.

Look for PSEdition set to Desktop.

ⓧ I cannot find Windows PowerShell ISE

Windows PowerShell 5.1 is preinstalled on currently supported Windows versions. A missing ISE shortcut means the computer needs an environment check before you continue.

  • Confirm with your device administrator that the computer runs a currently supported Windows version.
  • Ask your device administrator to restore the built-in Windows PowerShell 5.1 features.

Still Unable to Find ISE?

Use Help me check why Windows PowerShell ISE is missing to work through the environment details safely.

Create the workspace and evidence snapshot

The project root keeps the source evidence separate from generated outputs. Its data folder holds the fictional snapshot. Its out folder is reserved for later handover artifacts.

  • Create the three-folder workspace structure by running these commands in the Console Pane:
$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
New-Item -Path $ProjectRoot -ItemType Directory -Force
New-Item -Path "$ProjectRoot\data" -ItemType Directory -Force
New-Item -Path "$ProjectRoot\out" -ItemType Directory -Force

What Does This Code Do?

  • The $ProjectRoot variable holds the full path to your lab workspace.
  • New-Item creates the project root plus its two child folders.
  • $HOME resolves to your Windows home directory.
  • -Force lets the command keep an existing matching folder.
  • Confirm that the command output lists NCP-AII-GoNoGo.
  • Confirm that the command output lists data.
  • Confirm that the command output lists out.

Missing a Workspace Folder?

  • Rerun the folder command after correcting any mistyped folder name.
  • Use Help me fix my PowerShell workspace folders for help with the command output.

The evidence snapshot represents four fictional GPU nodes at the moment of handover. Each column records one signal that a later acceptance gate can evaluate.

  • Select the new-file control in the ISE toolbar to create a blank file.
  • Paste this evidence snapshot into the Script Pane:
Node,SlurmState,ExpectedGpuCount,DetectedGpuCount,ExpectedFirmware,ObservedFirmware,BmcReachable,TopologyMatch,SingleNodeStress,NcclTest,HplTest,StorageTest
gpu-a01,IDLE,8,8,baseline-3,baseline-3,True,True,PASS,PASS,PASS,PASS
gpu-a02,IDLE,8,7,baseline-3,baseline-3,True,True,PASS,PASS,PASS,PASS
gpu-a03,IDLE,8,8,baseline-3,baseline-2,True,True,PASS,PASS,PASS,PASS
gpu-a04,IDLE,8,8,baseline-3,baseline-3,True,False,PASS,FAIL,PASS,PASS

What Does This Evidence Contain?

  • Each row represents one fictional cluster node.
  • The scheduler column records the Slurm signal.
  • The server columns record GPU inventory plus management access plus firmware evidence.
  • The remaining columns record topology plus stress plus communication plus compute plus storage evidence.
  • Save the file as $HOME\NCP-AII-GoNoGo\data\cluster-snapshot.csv using the ISE save control.
  • Confirm that the Script Pane tab shows cluster-snapshot.csv.

Snapshot Saved Somewhere Else?

  • Save the file again with the full path shown above.
  • Use Help me locate my cluster snapshot if the ISE tab shows a different location.

✔️ Awesome, I've got everything!

Your snapshot matches the four-node evidence pack used by the remaining steps.

ⓧ I'd like to double check the full code

Node,SlurmState,ExpectedGpuCount,DetectedGpuCount,ExpectedFirmware,ObservedFirmware,BmcReachable,TopologyMatch,SingleNodeStress,NcclTest,HplTest,StorageTest
gpu-a01,IDLE,8,8,baseline-3,baseline-3,True,True,PASS,PASS,PASS,PASS
gpu-a02,IDLE,8,7,baseline-3,baseline-3,True,True,PASS,PASS,PASS,PASS
gpu-a03,IDLE,8,8,baseline-3,baseline-2,True,True,PASS,PASS,PASS,PASS
gpu-a04,IDLE,8,8,baseline-3,baseline-3,True,False,PASS,FAIL,PASS,PASS
Import and inspect the snapshot

Importing the snapshot turns each evidence row into an object with named properties. This gives the later validators a consistent structure to filter plus compare.

Before you run this, make one prediction: how many node rows the import will show.

  • Verify the session plus import the snapshot by running these commands in the Console Pane:
$PSVersionTable
Import-Csv -Path "$HOME\NCP-AII-GoNoGo\data\cluster-snapshot.csv"

What Do These Commands Prove?

  • $PSVersionTable repeats the runtime proof for the current ISE session.
  • Import-Csv turns each snapshot row into a table-like object.
  • The CSV headings become properties that later gates can evaluate.

You should see PSVersion beginning with 5.1 plus PSEdition set to Desktop.

Below the version table, you should see four rows named gpu-a01 through gpu-a04.

  • Locate SlurmState to identify the scheduler evidence.
  • Locate ExpectedGpuCount plus DetectedGpuCount to identify the GPU inventory evidence.
  • Locate BmcReachable plus the firmware columns to identify the server bring-up evidence.
  • Locate TopologyMatch to identify the physical validation evidence.
  • Locate the stress plus NCCL plus HPL plus storage columns to identify the cluster-test evidence.

That is your lab ready: the same four-node evidence pack can now drive every acceptance gate.

Your evidence workspace is ready. Next, you will run a Slurm-only checker that makes a convincing false-positive decision.

Experience the Slurm-Only False Positive

Your fictional evidence snapshot is loaded in Windows PowerShell ISE. You can now test how far a familiar Slurm signal carries an acceptance decision.

A scheduler can show every node as available while evidence beyond the control plane remains unresolved. This step puts that shortcut on trial before you build a broader NCP-AII validator.

In this step, get ready to:
  • Save a checker that evaluates Slurm state alone.
  • Run the selected checker with F8.
  • Identify the acceptance evidence missing from its GO decision.
Create the Slurm-only checker

The checker imports your CSV snapshot. It searches for nodes whose scheduler state differs from the expected value.

  • Click File in the ISE menu bar.
  • Click New.
  • Paste the following checker into the blank Script Pane.
$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

$nodes = Import-Csv -Path $DataPath
$notIdle = @($nodes | Where-Object -FilterScript { $_.SlurmState -ne "IDLE" })

if ($notIdle.Count -eq 0) {
    "Decision: GO"
    "Reason: Every node is IDLE in Slurm."
}
else {
    "Decision: NO-GO"
    "Reason: One or more nodes are not IDLE in Slurm."
}

What does this code do?

  • The $ProjectRoot variable points to the workspace you created earlier.
  • The $DataPath variable points to data\cluster-snapshot.csv.
  • Import-Csv loads each fictional node as an object in $nodes.
  • Where-Object places every node outside the IDLE state into $notIdle.
  • The final condition returns GO when $notIdle contains zero nodes.
  • Click File in the ISE menu bar.
  • Click Save As.
  • Select the NCP-AII-GoNoGo folder under your user home folder.
  • Enter naive-slurm-check.ps1 in the File name box.
  • Click Save.
  • Confirm the script tab shows naive-slurm-check.ps1 without an unsaved-change asterisk.

Cannot save the checker?

  • Confirm the Save As dialog is inside the existing NCP-AII-GoNoGo folder.
  • Check that the file name ends with the .ps1 extension.

Help me save my PowerShell script in the correct project folder.

✔️ Awesome, I've got everything!

The saved naive-slurm-check.ps1 file now matches the Slurm-only checker.

ⓧ I'd like to double check the full code

$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

$nodes = Import-Csv -Path $DataPath
$notIdle = @($nodes | Where-Object -FilterScript { $_.SlurmState -ne "IDLE" })

if ($notIdle.Count -eq 0) {
    "Decision: GO"
    "Reason: Every node is IDLE in Slurm."
}
else {
    "Decision: NO-GO"
    "Reason: One or more nodes are not IDLE in Slurm."
}
Run the tempting shortcut

PowerShell applies its execution policy to script files. Selected commands can run interactively through the ISE without changing that policy.

  • Click inside the Script Pane.
  • Press Ctrl+A to select every line.

Before you press F8, hold your prediction in mind: GO or NO-GO.

  • Press F8 to run the selected text.

Your checker ran exactly as written. The Console Pane shows Decision: GO followed by Reason: Every node is IDLE in Slurm.

This is the intended shortfall. The checker approved the cluster from scheduler state alone.

No decision in the Console Pane?

  • Confirm every line in the Script Pane is highlighted before you press F8.
  • Check that data\cluster-snapshot.csv still exists inside NCP-AII-GoNoGo.
  • Select the complete script again if only part of the checker ran.

Help me diagnose why my selected PowerShell checker did not print a decision.

Find the missing evidence

The visible GO comes from one narrow rule. Every mandatory acceptance signal outside SlurmState remains outside the decision.

  • Find the $notIdle assignment in naive-slurm-check.ps1.
  • Compare SlurmState with ExpectedGpuCount, DetectedGpuCount, BmcReachable, ExpectedFirmware, and ObservedFirmware in the snapshot.
  • Compare SlurmState with TopologyMatch, SingleNodeStress, NcclTest, HplTest, and StorageTest in the snapshot.
  • Classify the visible GO as a false-positive acceptance decision.

What makes this a false positive?

  • GPU inventory fails on gpu-a02 because the snapshot reports 7 detected GPUs against 8 expected GPUs.
  • Firmware alignment fails on gpu-a03 because baseline-2 differs from baseline-3.
  • Topology validation fails on gpu-a04 because TopologyMatch is False.
  • NCCL evidence fails on gpu-a04 because NcclTest is FAIL.
  • BMC reachability, single-node stress, HPL evidence, and storage evidence also remain unevaluated by the checker.
  • Select every line in naive-slurm-check.ps1 again.

Before the final run, predict whether the unchanged checker can notice any of the four faulty fields.

  • Press F8 to run the selected checker again.

You will still see Decision: GO with the reason that every node is IDLE in Slurm. That repeatable false positive proves the scheduler signal cannot carry the acceptance decision by itself.

You have exposed the gap in a Slurm-only handover. Next, you will turn GPU inventory, BMC reachability, and firmware alignment into mandatory findings.

Add Server Bring-Up Gates

Your Slurm-only check returned GO because every fictional node was IDLE. That scheduler view left the cluster's server evidence untested.

You will build the first NCP-AII acceptance gates in Windows PowerShell ISE. Each node gets a finding for its GPU inventory.

Each node also gets a BMC reachability finding. A firmware alignment finding completes this stage.

In this step, get ready to:
  • Create a validator that keeps Slurm state as one acceptance signal.
  • Generate server bring-up findings for every fictional node.
  • Derive a go or no-go decision from the failed findings.
Build the validator foundation

The new validator stores each check as a structured finding. This keeps the familiar Slurm signal while preparing the decision for additional evidence.

  • Add the validator foundation to a new ISE Script Pane tab by pasting this code:
$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

$nodes = Import-Csv -Path $DataPath

$findings = foreach ($node in $nodes) {
    $slurmResult = if ($node.SlurmState -eq "IDLE") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Control Plane Signal"
        Check = "Slurm node state"
        Result = $slurmResult
        Evidence = "Observed $($node.SlurmState); expected IDLE"
    }
}

$failedFindings = @($findings | Where-Object -FilterScript { $_.Result -eq "FAIL" })
$decision = if ($failedFindings.Count -eq 0) { "GO" } else { "NO-GO" }
"Decision: $decision"
"Failure count: $($failedFindings.Count)"

What does this code do?

  • The $ProjectRoot and $DataPath variables point to the evidence snapshot.
  • The $findings loop creates one Slurm finding for each node.
  • The $failedFindings collection keeps every finding with a FAIL result.
  • The $decision variable returns GO only when the failure count is zero.
  • Save the script as $HOME\NCP-AII-GoNoGo\validate-cluster.ps1.
  • Select all text in validate-cluster.ps1.

Before you run this foundation, do you expect its decision to differ from the Slurm-only result?

  • Run the selected script text by pressing F8.

You will still see Decision: GO followed by Failure count: 0. The decision remains incomplete because the new server gates do not exist yet.

Seeing a syntax problem?

  • Check that the closing brace for $findings appears before $failedFindings.
  • Confirm that every quoted value has matching double quotes.

Ask for help checking the validator foundation.

Add GPU and BMC findings

A scheduler state cannot confirm that a server contains its expected accelerators. The GPU finding compares detected inventory with the acceptance baseline.

  • Add the GPU inventory finding immediately above the closing brace for the $findings loop by pasting this code:
    $gpuResult = if ($node.DetectedGpuCount -eq $node.ExpectedGpuCount) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "GPU inventory"
        Result = $gpuResult
        Evidence = "$($node.DetectedGpuCount) detected; $($node.ExpectedGpuCount) expected"
    }

How does the GPU gate work?

  • The $gpuResult comparison checks the detected count against the expected count for the current node.
  • The evidence string preserves both values so the failed finding explains the mismatch.
  • The stage label places this check in the server bring-up sequence.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.
  • Run the selected text by pressing F8.

There is the first hidden fault. You will see Decision: NO-GO followed by Failure count: 1.

Still seeing a GO?

  • Confirm that the GPU block sits inside the foreach loop.
  • Check that DetectedGpuCount and ExpectedGpuCount match the CSV column names.

Ask for help tracing the missing GPU failure.

Management access provides a separate readiness signal. The BMC finding records whether each node's management interface is reachable.

  • Add the BMC reachability finding below the GPU block by pasting this code:
    $bmcResult = if ($node.BmcReachable -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "BMC reachability"
        Result = $bmcResult
        Evidence = "BmcReachable=$($node.BmcReachable)"
    }

How does the BMC gate work?

  • The $bmcResult comparison expects the imported CSV value to equal True.
  • Each BMC finding preserves the observed value as evidence.
  • All four fictional nodes pass this gate.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.
  • Run the selected text by pressing F8.

You will still see Decision: NO-GO followed by Failure count: 1. The unchanged count confirms that BMC reachability adds evidence without inventing another failure.

  • Display the generated finding objects by entering this variable in the Console Pane:
$findings

What should you see?

You will see Slurm state rows followed by GPU inventory rows. You will also see four BMC reachability rows with PASS results.

Missing the BMC rows?

  • Run the complete selected script again before entering $findings.
  • Confirm that the BMC block appears before the loop's closing brace.

Ask for help locating the missing findings.

Gate the firmware baseline

Matching hardware inventory still leaves room for configuration drift. The firmware finding compares the observed baseline with the expected baseline for each node.

  • Add the firmware alignment finding below the BMC block by pasting this code:
    $firmwareResult = if ($node.ObservedFirmware -eq $node.ExpectedFirmware) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "Firmware alignment"
        Result = $firmwareResult
        Evidence = "Observed $($node.ObservedFirmware); expected $($node.ExpectedFirmware)"
    }

How does the firmware gate work?

  • The $firmwareResult comparison checks the observed firmware against the expected baseline.
  • A mismatch becomes a separate failed finding for the affected node.
  • The evidence records both baselines for remediation.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.

Before you run the completed server-gate version, how many failed findings do you expect from the snapshot?

  • Run the selected script text by pressing F8.

You will see Decision: NO-GO followed by Failure count: 2. The server gates have replaced the unsafe GO with a defensible decision.

  • Reveal the two failed finding objects by entering this variable in the Console Pane:
$failedFindings

Which findings failed?

You will see the GPU inventory failure for gpu-a02. Its evidence shows seven detected GPUs against eight expected GPUs.

You will also see the firmware alignment failure for gpu-a03. Its evidence shows baseline-2 against the expected baseline-3.

Seeing the wrong failure count?

  • Confirm that all four finding blocks sit inside the same foreach loop.
  • Check that each result variable is assigned to its matching finding.
  • Compare every CSV property name with the snapshot header.

Ask for help tracing the decision logic.

✔️ Awesome, I've got everything!

Your saved validator now checks Slurm state plus the three server bring-up gates.

ⓧ I'd like to double check the full code

The reference below is the complete validate-cluster.ps1 file for this step.

$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

$nodes = Import-Csv -Path $DataPath

$findings = foreach ($node in $nodes) {
    $slurmResult = if ($node.SlurmState -eq "IDLE") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Control Plane Signal"
        Check = "Slurm node state"
        Result = $slurmResult
        Evidence = "Observed $($node.SlurmState); expected IDLE"
    }

    $gpuResult = if ($node.DetectedGpuCount -eq $node.ExpectedGpuCount) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "GPU inventory"
        Result = $gpuResult
        Evidence = "$($node.DetectedGpuCount) detected; $($node.ExpectedGpuCount) expected"
    }

    $bmcResult = if ($node.BmcReachable -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "BMC reachability"
        Result = $bmcResult
        Evidence = "BmcReachable=$($node.BmcReachable)"
    }

    $firmwareResult = if ($node.ObservedFirmware -eq $node.ExpectedFirmware) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "Firmware alignment"
        Result = $firmwareResult
        Evidence = "Observed $($node.ObservedFirmware); expected $($node.ExpectedFirmware)"
    }
}

$failedFindings = @($findings | Where-Object -FilterScript { $_.Result -eq "FAIL" })
$decision = if ($failedFindings.Count -eq 0) { "GO" } else { "NO-GO" }
"Decision: $decision"
"Failure count: $($failedFindings.Count)"

How to use this reference

Compare the order of your four finding blocks with this file. The decision logic belongs after the closing brace for the loop.

Your validator now catches the inventory and firmware faults that Slurm missed. Next up, you will test physical readiness plus cluster performance evidence.

Add Physical and Cluster Verification Gates

Your server checks already exposed the unsafe readiness decision that Slurm missed.

However, server inventory cannot prove that the fabric matches its planned topology. It also cannot prove that compute, communication, or storage tests passed.

This step expands the validator across the physical readiness plus performance evidence in the NCP-AII sequence.

In this step, get ready to:
  • Extend the validation sequence from topology comparison through single-node stress evidence.
  • Evaluate NCCL, HPL, plus storage evidence as separate checks.
  • Classify four failures by their bring-up stages.
Extend the validation sequence

A topology comparison checks whether the observed fabric matches its planned design. A single-node stress result provides separate evidence about compute behavior on each node.

  • Switch back to validate-cluster.ps1 in the Script Pane.
  • Locate Check = "Firmware alignment" inside the existing firmware finding.
  • Find the closing brace for that finding.
  • Add the topology finding below that brace by pasting this code:
    $topologyResult = if ($node.TopologyMatch -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Physical Validation"
        Check = "Topology comparison"
        Result = $topologyResult
        Evidence = "TopologyMatch=$($node.TopologyMatch)"
    }

What does this gate prove?

  • The $topologyResult comparison turns each node's TopologyMatch value into a separate acceptance result.
  • The Physical Validation stage keeps fabric evidence separate from server inventory.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.

Before you press F8, do you expect the topology gate to preserve two failures or reveal another one?

  • Press F8 to run the selected text.

You'll see Decision: NO-GO plus Failure count: 3. The topology mismatch on gpu-a04 is now part of the decision.

Still seeing two failures?

  • Check that the topology block sits inside the foreach loop.
  • Compare TopologyMatch with the matching column in data/cluster-snapshot.csv.
  • Ask for help with the topology gate.

Topology validates the fabric layout. The next finding checks the single-node stress evidence independently.

  • Locate Check = "Topology comparison" in the new finding.
  • Find the closing brace for that finding.
  • Add the stress finding below that brace by pasting this code:
    $stressResult = if ($node.SingleNodeStress -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Single-node stress"
        Result = $stressResult
        Evidence = "Observed $($node.SingleNodeStress); expected PASS"
    }

Why keep stress separate?

  • The $stressResult comparison evaluates the stress result for every node.
  • A dedicated finding preserves the exact evidence behind each node's result.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.
  • Press F8 to run the selected text.

Good progress. You'll still see Failure count: 3 because every SingleNodeStress value passes.

Did the failure count change?

  • Check that $node.SingleNodeStress is compared with PASS.
  • Confirm that the stress block closes before the end of the foreach loop.
  • Ask for help checking the stress finding.
Add distinct cluster-test gates

Collective communication, benchmark performance, plus storage behavior can fail independently. Separate findings preserve which test passed or failed.

The first new check evaluates NCCL evidence for each node.

  • Locate Check = "Single-node stress" in the stress finding.
  • Find the closing brace for that finding.
  • Add the NCCL finding below that brace by pasting this code:
    $ncclResult = if ($node.NcclTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "NCCL evidence"
        Result = $ncclResult
        Evidence = "Observed $($node.NcclTest); expected PASS"
    }

What does the NCCL gate add?

  • The $ncclResult comparison evaluates collective communication evidence per node.
  • A failed result remains traceable to the specific node plus test.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.

Before you press F8, do you think every node has passing NCCL evidence?

  • Press F8 to run the selected text.

You'll see Failure count: 4. The failed NcclTest value on gpu-a04 reveals the fourth acceptance failure.

Still seeing three failures?

  • Check that the new condition reads from $node.NcclTest.
  • Confirm that FAIL remains in the gpu-a04 row of data/cluster-snapshot.csv.
  • Ask for help tracing the NCCL result.

The HPL result needs its own gate so benchmark evidence remains visible even when it passes.

  • Locate Check = "NCCL evidence" in the NCCL finding.
  • Find the closing brace for that finding.
  • Add the HPL finding below that brace by pasting this code:
    $hplResult = if ($node.HplTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "HPL evidence"
        Result = $hplResult
        Evidence = "Observed $($node.HplTest); expected PASS"
    }

Why record passing evidence?

  • The $hplResult comparison evaluates benchmark evidence with the same acceptance pattern.
  • Passing findings create a complete evidence trail for the final handover.
  • Save validate-cluster.ps1.
  • Select all text in the Script Pane.
  • Press F8 to run the selected text.

You'll still see Failure count: 4 because all four HplTest values pass.

Seeing a fifth failure?

  • Check that $node.HplTest is compared with PASS.
  • Compare the HplTest column with the spelling used in the script.
  • Ask for help checking the HPL comparison.

The final gate keeps storage-test evidence visible as its own acceptance result.

  • Locate Check = "HPL evidence" in the HPL finding.
  • Find the closing brace for that finding.
  • Add the storage finding below that brace by pasting this code:
    $storageResult = if ($node.StorageTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Storage evidence"
        Result = $storageResult
        Evidence = "Observed $($node.StorageTest); expected PASS"
    }

What does the storage gate preserve?

  • The $storageResult comparison checks storage evidence without hiding it inside a generic health flag.
  • Each node now contributes a distinct storage finding to the acceptance record.
  • Save validate-cluster.ps1.

✔️ Awesome, I've got everything!

Your validator now contains all five physical plus cluster-test gates.

  • Confirm that validate-cluster.ps1 is saved.

ⓧ I'd like to double check the full code

$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

$nodes = Import-Csv -Path $DataPath

$findings = foreach ($node in $nodes) {
    $slurmResult = if ($node.SlurmState -eq "IDLE") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Control Plane Signal"
        Check = "Slurm node state"
        Result = $slurmResult
        Evidence = "Observed $($node.SlurmState); expected IDLE"
    }

    $gpuResult = if ($node.DetectedGpuCount -eq $node.ExpectedGpuCount) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "GPU inventory"
        Result = $gpuResult
        Evidence = "$($node.DetectedGpuCount) detected; $($node.ExpectedGpuCount) expected"
    }

    $bmcResult = if ($node.BmcReachable -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "BMC reachability"
        Result = $bmcResult
        Evidence = "BmcReachable=$($node.BmcReachable)"
    }

    $firmwareResult = if ($node.ObservedFirmware -eq $node.ExpectedFirmware) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "Firmware alignment"
        Result = $firmwareResult
        Evidence = "Observed $($node.ObservedFirmware); expected $($node.ExpectedFirmware)"
    }

    $topologyResult = if ($node.TopologyMatch -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Physical Validation"
        Check = "Topology comparison"
        Result = $topologyResult
        Evidence = "TopologyMatch=$($node.TopologyMatch)"
    }

    $stressResult = if ($node.SingleNodeStress -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Single-node stress"
        Result = $stressResult
        Evidence = "Observed $($node.SingleNodeStress); expected PASS"
    }

    $ncclResult = if ($node.NcclTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "NCCL evidence"
        Result = $ncclResult
        Evidence = "Observed $($node.NcclTest); expected PASS"
    }

    $hplResult = if ($node.HplTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "HPL evidence"
        Result = $hplResult
        Evidence = "Observed $($node.HplTest); expected PASS"
    }

    $storageResult = if ($node.StorageTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Storage evidence"
        Result = $storageResult
        Evidence = "Observed $($node.StorageTest); expected PASS"
    }
}

$failedFindings = @($findings | Where-Object -FilterScript { $_.Result -eq "FAIL" })
$decision = if ($failedFindings.Count -eq 0) { "GO" } else { "NO-GO" }
"Decision: $decision"
"Failure count: $($failedFindings.Count)"
Verify and classify the four failures

The expanded validator now follows the evidence sequence from server bring-up through physical validation into cluster testing. The final run proves that every new gate contributes to one combined disposition.

  • Select all text in the Script Pane.

Before you press F8, which decision should four mandatory failures produce?

  • Press F8 to run the expanded validator.

You'll see Decision: NO-GO plus Failure count: 4.

Seeing the wrong failure count?

  • Check that all five new finding blocks remain inside the foreach loop.
  • Confirm that every finding assigns its result to the matching result variable.
  • Ask for help comparing your validator with the expected structure.
  • Locate the $failedFindings assignment near the bottom of the script.
  • Select only the first $failedFindings variable on that line.
  • Press F8 to display the failed findings.

You'll see four failed rows. They identify GPU inventory on gpu-a02, firmware alignment on gpu-a03, topology comparison on gpu-a04, plus NCCL evidence on gpu-a04.

How do the stages preserve sequence?

The inventory plus firmware failures belong to System and Server Bring-Up. The topology failure belongs to Physical Validation.

The NCCL failure belongs to Cluster Test and Verification. This ordering shows where acceptance first breaks before later performance evidence is considered.

Your validator now reaches a defensible NO-GO across server, physical, plus cluster-test evidence. Next, you'll turn these findings into durable handover files.

Produce the Acceptance Handover

The expanded validator already proves that the fictional cluster requires a NO-GO decision. Your next goal is to preserve that result outside Windows PowerShell ISE.

A production handover needs durable evidence. It also needs an explicit disposition that remains available after the console clears.

In this step, get ready to:
  • Export every structured finding to a CSV register.
  • Generate a Markdown report that preserves the disposition.
  • Inspect both artifacts to set the remediation order.
Export the findings register

A CSV findings register gives every acceptance gate its own row. Each row preserves the complete evidence record for one check.

  • At the top of validate-cluster.ps1, locate the current path setup by finding these lines:
$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"

What Do These Paths Do?

The $ProjectRoot variable identifies the workspace created earlier. The $DataPath variable points to the fictional evidence snapshot.

  • Replace those two lines with the expanded path setup below:
$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"
$FindingsPath = "$ProjectRoot\out\findings.csv"
$ReportPath = "$ProjectRoot\out\handover-report.md"

What Changes Here?

  • The $FindingsPath variable gives the structured register a fixed destination.
  • The $ReportPath variable gives the readable handover a separate destination.

The $findings collection already holds all 36 node-and-check results. The export turns those in-memory objects into a file that another engineer can filter or share.

  • Near the bottom of validate-cluster.ps1, locate the line that defines $decision.
  • Delete the two display lines immediately below the $decision line.
  • Place the export directly below the $decision line by adding this code:
$findings | Export-Csv -Path $FindingsPath -NoTypeInformation

What Does This Code Do?

  • The pipeline sends every finding to Export-Csv.
  • The -Path parameter writes the rows to out\findings.csv.
  • The -NoTypeInformation parameter keeps the file focused on the column headers plus evidence rows.
  • Save validate-cluster.ps1.

Before you run the updated selection, how many finding rows should four nodes produce across nine acceptance checks?

  • Select all text in the ISE Script Pane.
  • Press F8 to run the selected validator.
  • Open out\findings.csv from the ISE file browser.

You should see 36 data rows beneath the headers. Four rows show FAIL in the Result column.

Don't See findings.csv?

  • Confirm that $FindingsPath points to $ProjectRoot\out\findings.csv.
  • Confirm that the out folder from the workspace setup still exists.
  • Select the complete script before pressing F8.

Help me diagnose why findings.csv was not created.

Build the Markdown handover report

A Markdown report places the disposition before the supporting evidence. This gives operations a decision they can read without loading the CSV into another tool.

  • In validate-cluster.ps1, find the Export-Csv line you just added.
  • Add the report-building code directly below that line:
$reportLines = @(
    "# AI Cluster Acceptance Report"
    ""
    "Decision: $decision"
    "Failure count: $($failedFindings.Count)"
    ""
    "## Evidence"
)

foreach ($finding in $findings) {
    $reportLines += "- $($finding.Node) | $($finding.Stage) | $($finding.Check) | $($finding.Result) | $($finding.Evidence)"
}

$reportLines | Set-Content -Path $ReportPath
$reportLines

How Is the Handover Built?

  • The $reportLines array stores the report heading plus the final disposition.
  • The foreach loop adds one evidence line for every finding.
  • Set-Content writes the latest report to out\handover-report.md.
  • The final $reportLines expression displays the same handover in the console.
  • Save validate-cluster.ps1.

Before you press F8, do you expect the durable report to preserve the same four failures as the console decision?

  • Select all text in the ISE Script Pane.
  • Press F8 to regenerate both handover artifacts.
  • Open out\handover-report.md from the ISE file browser.

The report starts with Decision: NO-GO. It shows Failure count: 4.

Below ## Evidence, you should see one line for every node and acceptance check. Strong work. Your NO-GO decision now survives the console session in a readable evidence report.

Report Missing or Incomplete?

  • Confirm that the report code sits after the $decision calculation.
  • Confirm that the foreach loop closes before the Set-Content line.
  • Confirm that $ReportPath points to $ProjectRoot\out\handover-report.md.

Help me fix my incomplete handover report.

✔️ Awesome, I've got everything!

Great. Double-check that validate-cluster.ps1 is saved before you inspect the final handover.

ⓧ I'd like to double check the full code

$ProjectRoot = "$HOME\NCP-AII-GoNoGo"
$DataPath = "$ProjectRoot\data\cluster-snapshot.csv"
$FindingsPath = "$ProjectRoot\out\findings.csv"
$ReportPath = "$ProjectRoot\out\handover-report.md"

$nodes = Import-Csv -Path $DataPath

$findings = foreach ($node in $nodes) {
    $slurmResult = if ($node.SlurmState -eq "IDLE") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Control Plane Signal"
        Check = "Slurm node state"
        Result = $slurmResult
        Evidence = "Observed $($node.SlurmState); expected IDLE"
    }

    $gpuResult = if ($node.DetectedGpuCount -eq $node.ExpectedGpuCount) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "GPU inventory"
        Result = $gpuResult
        Evidence = "$($node.DetectedGpuCount) detected; $($node.ExpectedGpuCount) expected"
    }

    $bmcResult = if ($node.BmcReachable -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "BMC reachability"
        Result = $bmcResult
        Evidence = "BmcReachable=$($node.BmcReachable)"
    }

    $firmwareResult = if ($node.ObservedFirmware -eq $node.ExpectedFirmware) { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "System and Server Bring-Up"
        Check = "Firmware alignment"
        Result = $firmwareResult
        Evidence = "Observed $($node.ObservedFirmware); expected $($node.ExpectedFirmware)"
    }

    $topologyResult = if ($node.TopologyMatch -eq "True") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Physical Validation"
        Check = "Topology comparison"
        Result = $topologyResult
        Evidence = "TopologyMatch=$($node.TopologyMatch)"
    }

    $stressResult = if ($node.SingleNodeStress -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Single-node stress"
        Result = $stressResult
        Evidence = "Observed $($node.SingleNodeStress); expected PASS"
    }

    $ncclResult = if ($node.NcclTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "NCCL evidence"
        Result = $ncclResult
        Evidence = "Observed $($node.NcclTest); expected PASS"
    }

    $hplResult = if ($node.HplTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "HPL evidence"
        Result = $hplResult
        Evidence = "Observed $($node.HplTest); expected PASS"
    }

    $storageResult = if ($node.StorageTest -eq "PASS") { "PASS" } else { "FAIL" }
    [pscustomobject]@{
        Node = $node.Node
        Stage = "Cluster Test and Verification"
        Check = "Storage evidence"
        Result = $storageResult
        Evidence = "Observed $($node.StorageTest); expected PASS"
    }
}

$failedFindings = @($findings | Where-Object -FilterScript { $_.Result -eq "FAIL" })
$decision = if ($failedFindings.Count -eq 0) { "GO" } else { "NO-GO" }

$findings | Export-Csv -Path $FindingsPath -NoTypeInformation

$reportLines = @(
    "# AI Cluster Acceptance Report"
    ""
    "Decision: $decision"
    "Failure count: $($failedFindings.Count)"
    ""
    "## Evidence"
)

foreach ($finding in $findings) {
    $reportLines += "- $($finding.Node) | $($finding.Stage) | $($finding.Check) | $($finding.Result) | $($finding.Evidence)"
}

$reportLines | Set-Content -Path $ReportPath
$reportLines

How to Use This Reference

Compare this reference with your saved file from top to bottom. The path variables should appear before the import logic.

The export plus report logic should appear after the decision calculation. This ordering ensures every artifact uses the completed findings collection.

Inspect the handover and set remediation order

The two artifacts serve different readers. The CSV supports detailed filtering while the report makes the acceptance disposition clear.

  • Review out\findings.csv for the columns Node, Stage, Check, Result, and Evidence.
  • Locate the four rows whose Result value is FAIL.
  • Return to out\handover-report.md to trace those failures through the acceptance sequence.

Remediation starts with GPU inventory on gpu-a02 plus firmware alignment on gpu-a03. Topology on gpu-a04 comes next.

The failed NCCL evidence on gpu-a04 follows. A complete revalidation closes the sequence.

Secret mission

Repair the Evidence Pack and Earn a GO

Four faulty evidence fields are holding the fictional cluster at NO-GO. Correct those fields without weakening the validator, then regenerate a clean acceptance handover.

Clean Up Your Resources

Clean Up Your Resources

Everything runs on your Windows computer at no ongoing cost. Decide whether to keep your files, pause your work, or delete the local folder.

Resources you used:

  • Your local NCP-AII-GoNoGo folder contains every project file and generated handover artifact.

Keep everything running

No cleanup is needed when you want to retain the validator for future evidence packs. Every file stays available at no ongoing cost.

  • Keep the NCP-AII-GoNoGo folder in your Windows home directory.
  • Rerun validate-cluster.ps1 whenever you want to validate a changed evidence snapshot.
  • Compare future results with out\handover-report.md to track changes in the acceptance decision.

Pause - I'll come back to this later

Your Windows PowerShell ISE session is the only open workspace. The validator has no background service to stop.

  • Save any unsaved changes in Windows PowerShell ISE.
  • Close Windows PowerShell ISE.
  • Leave the NCP-AII-GoNoGo folder in your Windows home directory.

Delete - I don't want to use this again

Permanent deletion removes the scripts and both generated artifacts together. A copy stored outside the project folder stays safe.

  • Copy any report you want to keep to a location outside NCP-AII-GoNoGo.
  • Close Windows PowerShell ISE.
  • Press Windows+E to open File Explorer.
  • Navigate to the Windows home directory that contains NCP-AII-GoNoGo.
  • Select the NCP-AII-GoNoGo folder.
  • Press Shift+Delete to permanently delete the selected folder.
  • Confirm permanent deletion in the Windows dialog.
  • Check that NCP-AII-GoNoGo no longer appears in your Windows home directory.

Nice Work!

Nice Work!

You did it! Your fictional AI cluster handover now rests on repeatable evidence from a PowerShell validator.

The unchanged controls moved from NO-GO to GO only after you repaired the four faulty evidence fields.

You've learned how to:

  • Expose the Slurm-only false positive by proving that an IDLE scheduler state can hide missing deployment evidence.
  • Build independent acceptance gates for BMC reachability, GPU inventory, firmware alignment, and topology. You also evaluated single-node stress, NCCL, HPL, and storage evidence as separate checks.
  • Export a filterable CSV findings register. You also generated a Markdown handover report with an explicit GO or NO-GO disposition.
  • Secret Mission: Repaired the four faulty evidence fields while preserving the unchanged validator. You proved that the same controls return Decision: GO with Failure count: 0.

Ready to quiz yourself?