Build an AI Output Verification Engine

Build a PowerShell module that fact-checks AI troubleshooting claims against live system state.

Introduction

30 Second Summary

Imagine an AI recommends restarting a critical service on your server because it detected errors in the logs. What if the AI is hallucinating, and those errors never actually happened?

In this project, you will build a PowerShell module called TrustLayer that automatically fact-checks AI troubleshooting recommendations against your live Windows system. Feed it a recommendation file, and it returns a confidence verdict with evidence for each claim.

What You'll Build

TrustLayer is a PowerShell script module that ingests AI-generated recommendations as JSON, checks each claim against your live Windows system using native cmdlets, and outputs a structured verdict with an evidence trail.

By the end of this project, you'll have:

  • A working TrustLayer module with 5 exported functions that verify AI claims against live Windows services, event logs, and DNS resolution.
  • A reusable verification pipeline pattern (parse, route, check, aggregate, verdict) you can apply to any claim-based validation scenario.
  • Sample AI recommendation JSON files for testing and demoing the tool on any Windows machine.
  • Secret Mission: Add batch processing mode that analyzes multiple AI recommendations and generates an accuracy trend report.

Are there any prerequisites?

You should be comfortable with PowerShell functions, parameters, and error handling. This project builds on those foundations by introducing module architecture and structured output patterns.

Set Up the TrustLayer Module

Every PowerShell module follows a standard folder structure. The module folder name must match the module name, and it contains a script file (.psm1) for your functions plus a manifest file (.psd1) that describes the module's metadata and exports.

In this step, you will create the TrustLayer module skeleton. You will set up the directory structure, write the module manifest, and create sample AI recommendation files that the module will verify in later steps.

In this step, get ready to:
  • Create the TrustLayer module directory structure with an empty module file and a Recommendations subfolder.
  • Write the module manifest that declares exports, version, and metadata.
  • Create two sample AI recommendation JSON files for testing.
Create the TrustLayer module directory

The module folder needs to contain the script module file where your functions will live, plus a subfolder for sample input data.

  • In VS Code, open the integrated terminal by pressing Ctrl+` (backtick key, above Tab).
  • Move to your Desktop by running this command:
cd ~\Desktop
  • Create the TrustLayer module folder, the empty module file, and the Recommendations subfolder by running these commands:
mkdir TrustLayer
cd TrustLayer
New-Item -ItemType File -Name "TrustLayer.psm1"
mkdir Recommendations
  • Confirm the files were created by running:
Get-ChildItem

You should see TrustLayer.psm1 and the Recommendations folder listed in the output.

What do these commands do?

  • mkdir TrustLayer creates the module folder on your Desktop.
  • New-Item -ItemType File -Name "TrustLayer.psm1" creates an empty script module file. You will add functions to this file in later steps.
  • mkdir Recommendations creates a subfolder to hold the sample AI recommendation JSON files.

Don't see TrustLayer.psm1?

Make sure you ran cd TrustLayer before creating the file. If you skipped it, the file may have been created in your Desktop folder instead.

help me find where my TrustLayer.psm1 file ended up

  • Open the TrustLayer folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the TrustLayer folder, then click Select Folder.
Write the module manifest

A module manifest is a .psd1 file that tells PowerShell what the module contains, which functions to export, and what version it is. Without a manifest, PowerShell cannot validate or properly load your module.

  • In VS Code's Explorer sidebar (left panel), right-click in the empty space below TrustLayer.psm1.
  • Click New File.
  • Name the file TrustLayer.psd1 and press Enter.
  • Paste the following manifest content into TrustLayer.psd1:
@{
    RootModule        = 'TrustLayer.psm1'
    ModuleVersion     = '1.0.0'
    GUID              = 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
    Author            = 'Your Name'
    Description       = 'AI Output Verification Engine - validates AI troubleshooting recommendations against live system state'
    PowerShellVersion = '5.1'
    FunctionsToExport = @(
        'ConvertFrom-AIRecommendation',
        'Test-ServiceClaim',
        'Test-EventLogClaim',
        'Test-NetworkClaim',
        'Invoke-TrustLayerVerification'
    )
    CmdletsToExport   = @()
    VariablesToExport  = @()
    AliasesToExport    = @()
}

What does this manifest do?

  • RootModule points to the .psm1 file that holds your functions.
  • FunctionsToExport lists the 5 functions users can call. Any function not listed here stays internal to the module.
  • PowerShellVersion sets the minimum version required to load the module.
  • GUID is a unique identifier for the module. In production, you would generate a unique one with [guid]::NewGuid().

What is New-ModuleManifest?

PowerShell provides a New-ModuleManifest cmdlet that generates .psd1 files automatically. It produces a verbose file with many comments and optional fields. For this project, you are writing a clean, minimal manifest so you can see exactly what each key does.

  • Save TrustLayer.psd1 by pressing Ctrl+S.
  • Verify the manifest is valid by running this command in the terminal:
Test-ModuleManifest -Path .\TrustLayer.psd1

You should see a table showing the module's Name, Path, Description, and Version with no red error text.

  • Import the module into your session to confirm it loads:
Import-Module .\TrustLayer.psd1 -Verbose

You should see verbose output showing PowerShell loading the module from its path. Since the .psm1 is still empty, no functions are exported yet. That is expected.

Seeing a red error on Test-ModuleManifest?

Double-check that TrustLayer.psd1 is in the same folder as TrustLayer.psm1. The RootModule path in the manifest must point to a file that actually exists.

Also verify there are no typos in the hashtable syntax. Every key-value pair must end with a newline, and the array items must be separated by commas.

help me fix my TrustLayer.psd1 manifest error

✔️ Awesome, I've got everything!

Great. Make sure you have saved TrustLayer.psd1 before moving on.

ⓧ I'd like to double check the full code

Here is the complete TrustLayer.psd1 file:

@{
    RootModule        = 'TrustLayer.psm1'
    ModuleVersion     = '1.0.0'
    GUID              = 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
    Author            = 'Your Name'
    Description       = 'AI Output Verification Engine - validates AI troubleshooting recommendations against live system state'
    PowerShellVersion = '5.1'
    FunctionsToExport = @(
        'ConvertFrom-AIRecommendation',
        'Test-ServiceClaim',
        'Test-EventLogClaim',
        'Test-NetworkClaim',
        'Invoke-TrustLayerVerification'
    )
    CmdletsToExport   = @()
    VariablesToExport  = @()
    AliasesToExport    = @()
}
Create sample AI recommendation files

The TrustLayer module verifies structured AI recommendations stored as JSON files. Each file contains a source, a recommendation sentence, and an array of testable claims. You will create two sample files that simulate AI-generated troubleshooting advice.

  • In the Explorer sidebar, right-click the Recommendations folder.
  • Click New File.
  • Name it sample-spooler.json and press Enter.
  • Paste the following JSON into sample-spooler.json:
{
    "source": "AI-CoPilot",
    "recommendation": "The Print Spooler service has stopped and is causing print failures. Restart the service to resolve.",
    "claims": [
        {
            "type": "ServiceState",
            "service": "Spooler",
            "expectedState": "Stopped"
        },
        {
            "type": "EventLog",
            "log": "System",
            "source": "Service Control Manager",
            "level": "Error",
            "hoursBack": 24
        }
    ]
}

What does this JSON represent?

This simulates an AI recommending a fix for print failures. It makes two testable claims: that the Print Spooler service (Spooler) is currently stopped, and that error events from the Service Control Manager exist in the System log within the last 24 hours.

Your module will check each claim against the live system and determine whether the AI's diagnosis is accurate.

  • Save sample-spooler.json by pressing Ctrl+S.
  • Create a second file. Right-click the Recommendations folder again.
  • Click New File.
  • Name it sample-dns.json and press Enter.
  • Paste the following JSON into sample-dns.json:
{
    "source": "AI-CoPilot",
    "recommendation": "DNS resolution is failing for internal resources. The DNS Client service may be stopped.",
    "claims": [
        {
            "type": "ServiceState",
            "service": "Dnscache",
            "expectedState": "Stopped"
        },
        {
            "type": "Network",
            "target": "google.com",
            "expectedResult": "ResolveFails"
        }
    ]
}

What does this file test?

This simulates an AI claiming that DNS resolution is broken. It asserts that the DNS Client service (Dnscache) is stopped and that resolving google.com fails. On a healthy system, both claims will be contradicted because the DNS Client is typically running and google.com resolves successfully.

  • Save sample-dns.json by pressing Ctrl+S.
  • Confirm both files exist by running this in the terminal:
Get-ChildItem .\Recommendations

You should see sample-dns.json and sample-spooler.json listed in the output.

Don't see both JSON files?

Make sure you created both files inside the Recommendations folder, not in the root TrustLayer folder. If a file ended up in the wrong location, drag it into the Recommendations folder in the Explorer sidebar.

help me fix my TrustLayer file structure

✔️ Awesome, I've got everything!

Great. Double check you have saved both JSON files before moving on.

ⓧ I'd like to double check the full code

Here is sample-spooler.json:

{
    "source": "AI-CoPilot",
    "recommendation": "The Print Spooler service has stopped and is causing print failures. Restart the service to resolve.",
    "claims": [
        {
            "type": "ServiceState",
            "service": "Spooler",
            "expectedState": "Stopped"
        },
        {
            "type": "EventLog",
            "log": "System",
            "source": "Service Control Manager",
            "level": "Error",
            "hoursBack": 24
        }
    ]
}

Here is sample-dns.json:

{
    "source": "AI-CoPilot",
    "recommendation": "DNS resolution is failing for internal resources. The DNS Client service may be stopped.",
    "claims": [
        {
            "type": "ServiceState",
            "service": "Dnscache",
            "expectedState": "Stopped"
        },
        {
            "type": "Network",
            "target": "google.com",
            "expectedResult": "ResolveFails"
        }
    ]
}

Your module skeleton is ready. You have the folder structure, manifest, and sample data in place. Next up, you will write the first real function that parses these JSON recommendations into structured objects.

Build the Recommendation Parser

Your module structure is in place. Now it's time to build the first function that powers TrustLayer's verification pipeline.

Every verification starts with parsing. Before TrustLayer can check any claim against your live system, it needs to read an AI recommendation file and extract its contents into a structured object that downstream functions can process.

In this step, get ready to:
  • Write the ConvertFrom-AIRecommendation function.
  • Test the parser against your sample JSON files.
Write the ConvertFrom-AIRecommendation function

This function is the entry point of the verification pipeline. It takes a file path to a JSON recommendation, reads the content, and returns a PSCustomObject with typed properties that the rest of the module can work with.

  • Open TrustLayer.psm1 in VS Code (it should still be open from the previous step).
  • Add the following function to the file:
function ConvertFrom-AIRecommendation {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        if (-not (Test-Path $Path)) {
            Write-Error "Recommendation file not found: $Path"
            return
        }

        $json = Get-Content -Path $Path -Raw | ConvertFrom-Json

        [PSCustomObject]@{
            Source         = $json.source
            Recommendation = $json.recommendation
            Claims         = $json.claims
            Timestamp      = Get-Date
        }
    }
}

What does this code do?

This function follows PowerShell's verb-noun naming convention. ConvertFrom- tells other scripters that this function transforms input from one format into another.

  • [CmdletBinding()] enables advanced function features like -Verbose and -ErrorAction support automatically.
  • [Parameter(Mandatory, ValueFromPipeline)] means the function requires a path and can accept it from the pipeline (e.g., piping file paths directly).
  • The process {} block ensures the function works correctly when receiving multiple items from the pipeline.
  • Test-Path validates that the file exists before attempting to read it, preventing confusing errors downstream.
  • The output PSCustomObject gives downstream functions a predictable structure with Source, Recommendation, Claims, and Timestamp properties.
  • Save TrustLayer.psm1.
  • Reload the module in your PowerShell terminal by running:
Import-Module .\TrustLayer.psd1 -Force
  • Confirm the function is exported by running:
Get-Command -Module TrustLayer

You should see ConvertFrom-AIRecommendation listed in the output.

Don't see the function listed?

  • Make sure you saved TrustLayer.psm1 before re-importing. The -Force flag reloads the file from disk, so unsaved changes won't appear.
  • Check that your terminal's working directory is the TrustLayer folder. If you're in the wrong directory, Import-Module won't find the manifest.
  • Look for red error text in the terminal after Import-Module. A syntax error in the function will prevent the module from loading.

Help me troubleshoot why my TrustLayer module function isn't showing up after Import-Module.

Test the parser against your sample JSON files

The function is loaded. Now you'll run it against both sample files to confirm it correctly parses the JSON structure into typed objects.

  • Test the parser against sample-spooler.json by running:
ConvertFrom-AIRecommendation -Path .\Recommendations\sample-spooler.json

You should see output with four properties: Source showing AI-CoPilot, Recommendation showing the Print Spooler message, Claims showing an array of objects, and Timestamp showing the current date and time.

  • Now test against sample-dns.json by running:
ConvertFrom-AIRecommendation -Path .\Recommendations\sample-dns.json

You should see the same four-property structure, this time with the DNS recommendation text and two claims about Dnscache and google.com.

Why does Claims show curly braces instead of details?

PowerShell's default formatting truncates nested objects in table view. The claims are fully intact inside the object. You can verify by piping to Select-Object -ExpandProperty Claims to see the full claim details.

  • Confirm the claims parsed correctly by running:
$(ConvertFrom-AIRecommendation -Path .\Recommendations\sample-spooler.json).Claims

You should see both claims expanded with their individual properties: type, service, expectedState, log, source, and so on.

Seeing an error about the file path?

  • Make sure your terminal is in the TrustLayer directory. The relative path .\Recommendations\ only works from inside the module folder.
  • If you see "Recommendation file not found", the function is working correctly. It's your custom error message. Double-check the file name matches sample-spooler.json exactly.

Help me debug why ConvertFrom-AIRecommendation can't find my sample JSON file.

✔️ Awesome, I've got everything!

Great. Double check you've saved TrustLayer.psm1 and your parser returns structured output for both sample files.

ⓧ I'd like to double check the full code

Here's the complete TrustLayer.psm1 file at this point:

function ConvertFrom-AIRecommendation {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        if (-not (Test-Path $Path)) {
            Write-Error "Recommendation file not found: $Path"
            return
        }

        $json = Get-Content -Path $Path -Raw | ConvertFrom-Json

        [PSCustomObject]@{
            Source         = $json.source
            Recommendation = $json.recommendation
            Claims         = $json.claims
            Timestamp      = Get-Date
        }
    }
}

Your parser takes raw JSON and turns it into a structured pipeline-ready object. Next up, you'll build the verification functions that actually test each claim against your live system.

Build the System Verification Functions

Your parser can now ingest an AI recommendation and break it into structured claims. The next question is: how do you check whether those claims are actually true?

Each claim type needs its own specialized checker. A service-state claim requires querying Get-Service, an event log claim needs Get-WinEvent, and a network claim uses Resolve-DnsName. In this step, you will build all three verification functions.

In this step, get ready to:
  • Write a service-state checker that compares expected vs. actual service status.
  • Write an event log checker that queries Windows Event Log for matching entries.
  • Write a network checker that verifies DNS resolution claims.
Write the Test-ServiceClaim function

The simplest claim to verify is a service-state assertion. The AI might say "the Print Spooler is stopped." Your function will call Get-Service and compare the actual status against what the AI claimed.

  • In TrustLayer.psm1, add the following function below your ConvertFrom-AIRecommendation function:
function Test-ServiceClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $serviceName = $Claim.service
    $expectedState = $Claim.expectedState

    try {
        $service = Get-Service -Name $serviceName -ErrorAction Stop
        $actualState = $service.Status.ToString()
        $verified = $actualState -eq $expectedState

        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = $actualState
            Verified  = $verified
            Evidence  = "Get-Service $serviceName returned Status: $actualState"
        }
    }
    catch {
        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = "ERROR: Service not found"
            Verified  = $false
            Evidence  = "Service '$serviceName' does not exist on this system"
        }
    }
}

What does this code do?

  • $Claim.service and $Claim.expectedState extract the service name and the state the AI is asserting from the claim object.
  • Get-Service -Name $serviceName -ErrorAction Stop queries the live service. The -ErrorAction Stop flag converts a missing service into a catchable exception.
  • $service.Status.ToString() converts the ServiceControllerStatus enum into a string like Running or Stopped so it can be directly compared to the JSON claim.
  • The function always returns a consistent PSCustomObject with six properties, whether the check succeeds or the service does not exist.
  • Save TrustLayer.psm1.
  • Reload the module and test Test-ServiceClaim by running these commands in your PowerShell terminal:
Import-Module .\TrustLayer.psm1 -Force
$claim = [PSCustomObject]@{ service = "Spooler"; expectedState = "Running" }
Test-ServiceClaim -Claim $claim

You should see a PSCustomObject with Verified set to True and Actual showing Running (assuming your Print Spooler service is active).

Seeing an error about the function not being recognized?

  • Make sure you saved TrustLayer.psm1 before running Import-Module.
  • Confirm you included -Force so PowerShell reloads the updated file.
  • Check that your terminal's working directory is the TrustLayer folder (run pwd to confirm).

Help me debug why my Test-ServiceClaim function isn't being recognized after Import-Module.

Write the Test-EventLogClaim function

Event log claims are trickier. The AI might assert that error events appeared in a specific log within a time window. Your function needs to translate human-readable severity levels (like "Error") into the numeric level codes that Get-WinEvent expects.

  • In TrustLayer.psm1, add the following function below Test-ServiceClaim:
function Test-EventLogClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $logName = $Claim.log
    $source = $Claim.source
    $level = switch ($Claim.level) {
        "Critical"    { 1 }
        "Error"       { 2 }
        "Warning"     { 3 }
        "Information" { 4 }
        default       { 2 }
    }
    $hoursBack = if ($Claim.hoursBack) { $Claim.hoursBack } else { 24 }
    $startTime = (Get-Date).AddHours(-$hoursBack)

    try {
        $filter = @{
            LogName   = $logName
            Level     = $level
            StartTime = $startTime
        }
        if ($source) {
            $filter['ProviderName'] = $source
        }

        $events = Get-WinEvent -FilterHashtable $filter -MaxEvents 5 -ErrorAction Stop
        $eventCount = $events.Count

        [PSCustomObject]@{
            ClaimType = "EventLog"
            Target    = "$logName/$source"
            Expected  = "$($Claim.level) events in last ${hoursBack}h"
            Actual    = "$eventCount event(s) found"
            Verified  = $true
            Evidence  = "Found $eventCount $($Claim.level) event(s) from '$source' in '$logName' log (last ${hoursBack}h)"
        }
    }
    catch {
        $message = $_.Exception.Message
        if ($message -like "*No events were found*") {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "No matching events"
                Verified  = $false
                Evidence  = "No $($Claim.level) events from '$source' in '$logName' log (last ${hoursBack}h)"
            }
        }
        else {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "ERROR: $message"
                Verified  = $false
                Evidence  = "Failed to query event log: $message"
            }
        }
    }
}

What does this code do?

  • The switch block maps human-readable severity names to the numeric levels that Get-WinEvent requires: Critical = 1, Error = 2, Warning = 3, Information = 4.
  • The FilterHashtable is built dynamically. ProviderName is only added if the claim specifies a source, which keeps the query flexible.
  • -MaxEvents 5 limits results so the function stays fast. You only need to know events exist, not fetch thousands.
  • The catch block distinguishes between "no events matched" (a valid empty result) and a genuine query failure, returning appropriate evidence for each.

Why two paths in the catch block?

Get-WinEvent with -ErrorAction Stop throws when no events match the filter. That is a valid "the claim is false" result, not a system error. The -like "*No events were found*" check separates that expected outcome from actual failures (like a permission denied or invalid log name).

  • Save TrustLayer.psm1.
  • Reload the module and test Test-EventLogClaim by running:
Import-Module .\TrustLayer.psm1 -Force
$claim = [PSCustomObject]@{ log = "System"; source = "Service Control Manager"; level = "Error"; hoursBack = 24 }
Test-EventLogClaim -Claim $claim

You should see a PSCustomObject output. If your system has recent Service Control Manager errors, Verified will be True. If no matching events exist, Verified will be False with the evidence explaining why. Both outcomes confirm the function works correctly.

Seeing a syntax error when pasting the function?

  • Check that all curly braces { } are properly matched. The function has nested try/catch and if/else blocks that each need their own closing brace.
  • Make sure the switch statement has its closing } on its own line after the default case.
  • Confirm there is a blank line between the closing } of Test-ServiceClaim and the function Test-EventLogClaim declaration.

Help me debug a syntax error in my Test-EventLogClaim function.

Write the Test-NetworkClaim function

The final claim type covers DNS resolution. An AI might assert that a hostname cannot be resolved. Your function uses Resolve-DnsName to verify whether resolution succeeds or fails, and compares that outcome to what the AI predicted.

  • In TrustLayer.psm1, add the following function below Test-EventLogClaim:
function Test-NetworkClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $target = $Claim.target
    $expectedResult = $Claim.expectedResult

    try {
        $result = Resolve-DnsName -Name $target -ErrorAction Stop
        $resolved = $result[0].IPAddress

        $verified = switch ($expectedResult) {
            "ResolveFails"    { $false }
            "ResolveSucceeds" { $true }
            default           { $true }
        }

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolved to $resolved"
            Verified  = $verified
            Evidence  = "Resolve-DnsName $target returned $resolved"
        }
    }
    catch {
        $verified = ($expectedResult -eq "ResolveFails")

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolution failed"
            Verified  = $verified
            Evidence  = "DNS resolution for '$target' failed: $($_.Exception.Message)"
        }
    }
}

What does this code do?

  • Resolve-DnsName -Name $target performs a live DNS lookup against the system's configured DNS servers.
  • The switch on $expectedResult flips the verification logic. If the AI claimed resolution would fail but it succeeded, Verified is $false because the AI was wrong.
  • The catch block handles DNS failure gracefully. If the AI expected failure and DNS indeed fails, the claim is verified as correct.
  • Save TrustLayer.psm1.
  • Reload the module and test Test-NetworkClaim by running:
Import-Module .\TrustLayer.psm1 -Force
$claim = [PSCustomObject]@{ target = "google.com"; expectedResult = "ResolveSucceeds" }
Test-NetworkClaim -Claim $claim

You should see Verified set to True and Actual showing the resolved IP address of google.com.

Getting a DNS resolution error?

  • Make sure your machine has an active internet connection. Resolve-DnsName needs access to your DNS servers.
  • If you are on a restricted network, try a different target like microsoft.com to confirm DNS works at all.
  • Verify that the function receives the claim object correctly by checking that $claim.target outputs google.com when you run it standalone.

Help me debug why Test-NetworkClaim is returning an error instead of a result.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the full TrustLayer.psm1 as it should look at this point:

function ConvertFrom-AIRecommendation {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        if (-not (Test-Path $Path)) {
            Write-Error "Recommendation file not found: $Path"
            return
        }

        $json = Get-Content -Path $Path -Raw | ConvertFrom-Json

        [PSCustomObject]@{
            Source         = $json.source
            Recommendation = $json.recommendation
            Claims         = $json.claims
            Timestamp      = Get-Date
        }
    }
}

function Test-ServiceClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $serviceName = $Claim.service
    $expectedState = $Claim.expectedState

    try {
        $service = Get-Service -Name $serviceName -ErrorAction Stop
        $actualState = $service.Status.ToString()
        $verified = $actualState -eq $expectedState

        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = $actualState
            Verified  = $verified
            Evidence  = "Get-Service $serviceName returned Status: $actualState"
        }
    }
    catch {
        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = "ERROR: Service not found"
            Verified  = $false
            Evidence  = "Service '$serviceName' does not exist on this system"
        }
    }
}

function Test-EventLogClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $logName = $Claim.log
    $source = $Claim.source
    $level = switch ($Claim.level) {
        "Critical"    { 1 }
        "Error"       { 2 }
        "Warning"     { 3 }
        "Information" { 4 }
        default       { 2 }
    }
    $hoursBack = if ($Claim.hoursBack) { $Claim.hoursBack } else { 24 }
    $startTime = (Get-Date).AddHours(-$hoursBack)

    try {
        $filter = @{
            LogName   = $logName
            Level     = $level
            StartTime = $startTime
        }
        if ($source) {
            $filter['ProviderName'] = $source
        }

        $events = Get-WinEvent -FilterHashtable $filter -MaxEvents 5 -ErrorAction Stop
        $eventCount = $events.Count

        [PSCustomObject]@{
            ClaimType = "EventLog"
            Target    = "$logName/$source"
            Expected  = "$($Claim.level) events in last ${hoursBack}h"
            Actual    = "$eventCount event(s) found"
            Verified  = $true
            Evidence  = "Found $eventCount $($Claim.level) event(s) from '$source' in '$logName' log (last ${hoursBack}h)"
        }
    }
    catch {
        $message = $_.Exception.Message
        if ($message -like "*No events were found*") {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "No matching events"
                Verified  = $false
                Evidence  = "No $($Claim.level) events from '$source' in '$logName' log (last ${hoursBack}h)"
            }
        }
        else {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "ERROR: $message"
                Verified  = $false
                Evidence  = "Failed to query event log: $message"
            }
        }
    }
}

function Test-NetworkClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $target = $Claim.target
    $expectedResult = $Claim.expectedResult

    try {
        $result = Resolve-DnsName -Name $target -ErrorAction Stop
        $resolved = $result[0].IPAddress

        $verified = switch ($expectedResult) {
            "ResolveFails"    { $false }
            "ResolveSucceeds" { $true }
            default           { $true }
        }

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolved to $resolved"
            Verified  = $verified
            Evidence  = "Resolve-DnsName $target returned $resolved"
        }
    }
    catch {
        $verified = ($expectedResult -eq "ResolveFails")

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolution failed"
            Verified  = $verified
            Evidence  = "DNS resolution for '$target' failed: $($_.Exception.Message)"
        }
    }
}

You now have three specialized verification functions, each designed to check one type of AI claim against your live system. Next up, you will build the orchestrator that ties these checkers together into a single verification pipeline.

Build the Verification Orchestrator

Your individual claim checkers each know how to test one type of assertion against your system.

Now you need a conductor that reads an AI recommendation, sends each claim to the right checker, and rolls all the evidence into a single trust verdict. That's the orchestrator function.

In this step, get ready to:
  • Write the orchestrator function that routes claims to the correct checker.
  • Add verdict logic that scores and classifies the overall recommendation.
  • Test the full pipeline end-to-end by manipulating a real service.
Write the claim routing function

The orchestrator's first job is to parse the recommendation and dispatch each claim to the matching Test-* function. A switch statement on the claim's type field handles this routing cleanly.

  • In TrustLayer.psm1, scroll to the bottom of the file (after the Test-NetworkClaim function's closing brace).
  • Start writing the Invoke-TrustLayerVerification function by adding this code:
function Invoke-TrustLayerVerification {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        $recommendation = ConvertFrom-AIRecommendation -Path $Path

        if (-not $recommendation) {
            return
        }

        # Route each claim to its specialized checker
        $results = foreach ($claim in $recommendation.Claims) {
            switch ($claim.type) {
                "ServiceState" { Test-ServiceClaim -Claim $claim }
                "EventLog"     { Test-EventLogClaim -Claim $claim }
                "Network"      { Test-NetworkClaim -Claim $claim }
                default {
                    [PSCustomObject]@{
                        ClaimType = $claim.type
                        Target    = "Unknown"
                        Expected  = "N/A"
                        Actual    = "N/A"
                        Verified  = $false
                        Evidence  = "Unsupported claim type: $($claim.type)"
                    }
                }
            }
        }

What does this code do?

This is the first half of the orchestrator function. It handles three responsibilities:

  • ValueFromPipeline on the $Path parameter means you can pipe file paths directly into this function.
  • ConvertFrom-AIRecommendation parses the JSON file into a structured object. If parsing fails, the function exits early.
  • The switch block routes each claim to Test-ServiceClaim, Test-EventLogClaim, or Test-NetworkClaim based on the claim's type field. Unknown types get a fallback object so the pipeline never silently drops a claim.

Don't save yet. The function is incomplete. You'll add the verdict logic next.

Add verdict logic and export the module

With claim results collected, the function needs to score them and produce a final trust verdict. The algorithm is simple: all pass means VERIFIED, none pass means CONTRADICTED, anything in between means PARTIALLY VERIFIED.

  • Immediately below the code you just added (still inside the process block), add the verdict logic and close the function:
        # Calculate verification ratio
        $totalClaims = @($results).Count
        $verifiedCount = @($results | Where-Object { $_.Verified }).Count

        # Assign verdict based on ratio
        $verdict = if ($verifiedCount -eq $totalClaims) {
            "VERIFIED"
        } elseif ($verifiedCount -eq 0) {
            "CONTRADICTED"
        } else {
            "PARTIALLY VERIFIED"
        }

        # Map verdict to actionable guidance
        $action = switch ($verdict) {
            "VERIFIED"           { "Proceed with recommendation" }
            "PARTIALLY VERIFIED" { "Investigate further before acting" }
            "CONTRADICTED"       { "Do NOT follow this recommendation" }
        }

        [PSCustomObject]@{
            Source         = $recommendation.Source
            Recommendation = $recommendation.Recommendation
            Verdict        = $verdict
            Confidence     = "$verifiedCount/$totalClaims claims verified"
            Results        = $results
            Timestamp      = Get-Date
            Action         = $action
        }
    }
}

What does the verdict logic do?

  • @($results).Count wraps results in an array to ensure .Count works even when there's only one claim.
  • The three-tier verdict (VERIFIED, PARTIALLY VERIFIED, CONTRADICTED) maps directly to a human-readable $action string. This tells the operator exactly what to do next without interpretation.
  • The output PSCustomObject is pipeline-friendly. You can pipe it to Format-List, export it to CSV, or feed it into another function.
  • Now add the Export-ModuleMember statement at the very bottom of TrustLayer.psm1 (after the function's closing brace) to expose all five functions:
Export-ModuleMember -Function @(
    'ConvertFrom-AIRecommendation',
    'Test-ServiceClaim',
    'Test-EventLogClaim',
    'Test-NetworkClaim',
    'Invoke-TrustLayerVerification'
)

Why Export-ModuleMember?

Without this line, all functions in a .psm1 file are exported by default. Adding an explicit Export-ModuleMember locks the public surface to only the functions you list.

This is a best practice for module development. If you add helper functions later, they stay private unless you explicitly export them.

  • Save TrustLayer.psm1.
  • Reload the module in your PowerShell terminal by running:
Import-Module .\TrustLayer.psd1 -Force
  • Confirm all five functions are exported by running:
Get-Command -Module TrustLayer

You should see all five function names listed: ConvertFrom-AIRecommendation, Test-ServiceClaim, Test-EventLogClaim, Test-NetworkClaim, and Invoke-TrustLayerVerification.

Not seeing all five functions?

  • Make sure you saved TrustLayer.psm1 before running Import-Module. The -Force flag reloads the module, but only from the saved file on disk.
  • Check that the Export-ModuleMember line is outside and below the function's closing brace, not inside it.
  • Confirm your terminal is in the TrustLayer directory (the folder containing TrustLayer.psd1).

Help me troubleshoot why my module isn't exporting all five functions.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file, then move on to testing the full pipeline.

ⓧ I'd like to double check the full code

Here is the complete TrustLayer.psm1 file with all five functions and the export statement:

function ConvertFrom-AIRecommendation {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        if (-not (Test-Path $Path)) {
            Write-Error "Recommendation file not found: $Path"
            return
        }

        $json = Get-Content -Path $Path -Raw | ConvertFrom-Json

        [PSCustomObject]@{
            Source         = $json.source
            Recommendation = $json.recommendation
            Claims         = $json.claims
            Timestamp      = Get-Date
        }
    }
}

function Test-ServiceClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $serviceName = $Claim.service
    $expectedState = $Claim.expectedState

    try {
        $service = Get-Service -Name $serviceName -ErrorAction Stop
        $actualState = $service.Status.ToString()
        $verified = $actualState -eq $expectedState

        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = $actualState
            Verified  = $verified
            Evidence  = "Get-Service $serviceName returned Status: $actualState"
        }
    }
    catch {
        [PSCustomObject]@{
            ClaimType = "ServiceState"
            Target    = $serviceName
            Expected  = $expectedState
            Actual    = "ERROR: Service not found"
            Verified  = $false
            Evidence  = "Service '$serviceName' does not exist on this system"
        }
    }
}

function Test-EventLogClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $logName = $Claim.log
    $source = $Claim.source
    $level = switch ($Claim.level) {
        "Critical"    { 1 }
        "Error"       { 2 }
        "Warning"     { 3 }
        "Information" { 4 }
        default       { 2 }
    }
    $hoursBack = if ($Claim.hoursBack) { $Claim.hoursBack } else { 24 }
    $startTime = (Get-Date).AddHours(-$hoursBack)

    try {
        $filter = @{
            LogName   = $logName
            Level     = $level
            StartTime = $startTime
        }
        if ($source) {
            $filter['ProviderName'] = $source
        }

        $events = Get-WinEvent -FilterHashtable $filter -MaxEvents 5 -ErrorAction Stop
        $eventCount = $events.Count

        [PSCustomObject]@{
            ClaimType = "EventLog"
            Target    = "$logName/$source"
            Expected  = "$($Claim.level) events in last ${hoursBack}h"
            Actual    = "$eventCount event(s) found"
            Verified  = $true
            Evidence  = "Found $eventCount $($Claim.level) event(s) from '$source' in '$logName' log (last ${hoursBack}h)"
        }
    }
    catch {
        $message = $_.Exception.Message
        if ($message -like "*No events were found*") {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "No matching events"
                Verified  = $false
                Evidence  = "No $($Claim.level) events from '$source' in '$logName' log (last ${hoursBack}h)"
            }
        }
        else {
            [PSCustomObject]@{
                ClaimType = "EventLog"
                Target    = "$logName/$source"
                Expected  = "$($Claim.level) events in last ${hoursBack}h"
                Actual    = "ERROR: $message"
                Verified  = $false
                Evidence  = "Failed to query event log: $message"
            }
        }
    }
}

function Test-NetworkClaim {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory)]
        [PSCustomObject]$Claim
    )

    $target = $Claim.target
    $expectedResult = $Claim.expectedResult

    try {
        $result = Resolve-DnsName -Name $target -ErrorAction Stop
        $resolved = $result[0].IPAddress

        $verified = switch ($expectedResult) {
            "ResolveFails"    { $false }
            "ResolveSucceeds" { $true }
            default           { $true }
        }

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolved to $resolved"
            Verified  = $verified
            Evidence  = "Resolve-DnsName $target returned $resolved"
        }
    }
    catch {
        $verified = ($expectedResult -eq "ResolveFails")

        [PSCustomObject]@{
            ClaimType = "Network"
            Target    = $target
            Expected  = $expectedResult
            Actual    = "Resolution failed"
            Verified  = $verified
            Evidence  = "DNS resolution for '$target' failed: $($_.Exception.Message)"
        }
    }
}

function Invoke-TrustLayerVerification {
    [CmdletBinding()]
    [OutputType([PSCustomObject])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [string]$Path
    )

    process {
        $recommendation = ConvertFrom-AIRecommendation -Path $Path

        if (-not $recommendation) {
            return
        }

        $results = foreach ($claim in $recommendation.Claims) {
            switch ($claim.type) {
                "ServiceState" { Test-ServiceClaim -Claim $claim }
                "EventLog"     { Test-EventLogClaim -Claim $claim }
                "Network"      { Test-NetworkClaim -Claim $claim }
                default {
                    [PSCustomObject]@{
                        ClaimType = $claim.type
                        Target    = "Unknown"
                        Expected  = "N/A"
                        Actual    = "N/A"
                        Verified  = $false
                        Evidence  = "Unsupported claim type: $($claim.type)"
                    }
                }
            }
        }

        $totalClaims = @($results).Count
        $verifiedCount = @($results | Where-Object { $_.Verified }).Count

        $verdict = if ($verifiedCount -eq $totalClaims) {
            "VERIFIED"
        } elseif ($verifiedCount -eq 0) {
            "CONTRADICTED"
        } else {
            "PARTIALLY VERIFIED"
        }

        $action = switch ($verdict) {
            "VERIFIED"           { "Proceed with recommendation" }
            "PARTIALLY VERIFIED" { "Investigate further before acting" }
            "CONTRADICTED"       { "Do NOT follow this recommendation" }
        }

        [PSCustomObject]@{
            Source         = $recommendation.Source
            Recommendation = $recommendation.Recommendation
            Verdict        = $verdict
            Confidence     = "$verifiedCount/$totalClaims claims verified"
            Results        = $results
            Timestamp      = Get-Date
            Action         = $action
        }
    }
}

Export-ModuleMember -Function @(
    'ConvertFrom-AIRecommendation',
    'Test-ServiceClaim',
    'Test-EventLogClaim',
    'Test-NetworkClaim',
    'Invoke-TrustLayerVerification'
)
Test the full pipeline end-to-end

The best way to prove the verification engine works is to manipulate real system state and watch the verdict change. You'll stop the Print Spooler service so the AI recommendation's claim becomes true, then start it again so the claim becomes false.

Why the Print Spooler?

The Spooler service manages print jobs. It's safe to stop and restart for testing because most development machines aren't actively printing.

  • Stop the Print Spooler service to simulate the AI recommendation's claimed state by running this command (you'll need an elevated/admin PowerShell session):
Stop-Service Spooler

Running as Administrator

If you get an access denied error, close your current terminal and open a new one as Administrator. Right-click the PowerShell icon and select Run as administrator.

  • Navigate back to your TrustLayer folder if you opened a new terminal, reimport the module, and run the verification against sample-spooler.json:
Import-Module .\TrustLayer.psd1 -Force
Invoke-TrustLayerVerification -Path .\Recommendations\sample-spooler.json

You should see a verdict of VERIFIED or PARTIALLY VERIFIED. The service claim verifies as True because you just stopped Spooler. The event log claim depends on whether matching errors exist in your System log.

  • Now start the service back to flip the system state by running:
Start-Service Spooler
  • Run the verification again against the same file:
Invoke-TrustLayerVerification -Path .\Recommendations\sample-spooler.json

This time the verdict should be CONTRADICTED. The AI claimed the Spooler was stopped, but Get-Service now reports it as Running. The evidence field shows exactly what was expected vs. what was found.

What should I see?

Your output includes a Verdict field, a Confidence ratio (like 0/2 claims verified), and an Action field saying Do NOT follow this recommendation.

To see individual claim evidence, expand the Results array by piping to Select-Object -ExpandProperty Results.

Getting errors when running the verification?

  • If you see a "file not found" error, check that your terminal's working directory is the TrustLayer folder and that Recommendations\sample-spooler.json exists.
  • If you get "access denied" on Stop-Service or Start-Service, you need to run PowerShell as Administrator.
  • If the verdict shows CONTRADICTED even after stopping the Spooler, confirm the service is actually stopped by running Get-Service Spooler and checking the Status column.

Help me debug why my TrustLayer verification isn't returning the expected verdict.

Secret mission

Add Batch Processing Mode

Your TrustLayer module verifies one recommendation at a time. In a real operations environment, you'd have dozens of AI recommendations queued up. Can you add a batch processing mode that analyzes an entire folder of recommendations and generates an accuracy trend report?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs or cloud resources to worry about. Decide whether to keep your module files on disk or delete them.

Restore your Print Spooler

If you stopped the Print Spooler service during testing, start it back now. Other applications on your machine may need it.

Start-Service Spooler

Resources you used:

  • TrustLayer module folder (contains TrustLayer.psm1, TrustLayer.psd1, and the Recommendations subfolder with JSON files).

Keep everything running

No action needed. Choose this if you plan to keep building on your TrustLayer module or want to reference it later.

  • Your TrustLayer folder stays on disk wherever you created it.
  • You can re-import the module at any time with Import-Module pointing at the folder path.

Pause - I'll come back to this later

Remove the module from your current PowerShell session to free up memory. Your files remain on disk for next time.

  • Run this command to unload the module from memory:
Remove-Module TrustLayer
  • Your files stay on disk. When you return, re-import with Import-Module and pick up where you left off.

Delete - I don't want to use this again

Remove all project files from your machine entirely.

  • First, remove the module from your session if it is still loaded:
Remove-Module TrustLayer -ErrorAction SilentlyContinue
  • Delete the entire TrustLayer folder by running this command (replace the path with wherever you created your folder):
Remove-Item -Path [[TRUSTLAYER_PATH="C:\Projects\TrustLayer"]] -Recurse -Force
  • Confirm the folder is gone by checking the parent directory:
Get-ChildItem [[TRUSTLAYER_PARENT="C:\Projects"]]
  • You should no longer see a TrustLayer folder in the output.

Nice Work!

Nice Work!

You just built a fully functional AI output verification engine in PowerShell. TrustLayer can now ingest AI-generated troubleshooting recommendations, check every claim against your live Windows system, and return a confidence verdict backed by real evidence.

You've learned how to:

  • Build a PowerShell script module from scratch with a proper manifest, controlled exports, and structured PSCustomObject output.
  • Create a verification pipeline that parses AI recommendations, routes claims to specialized checkers, and produces evidence-based verdicts.
  • Test claims against live system state using Get-Service, Get-WinEvent, and Resolve-DnsName to independently verify each assertion.
  • Secret Mission: Added batch processing mode with aggregate accuracy reporting for trend analysis across multiple AI recommendations.

Ready to quiz yourself?