Govern DORA Metrics with DevLake

Build a governed five-metric delivery dashboard with GitLab and DevLake.

Introduction

30 Second Summary

Delivery numbers can shape funding, oversight, and operating decisions long before anyone checks how they were calculated. A missing source can make the same program look safer than it really is.

In this project, you will build a governed measurement system for five DORA measures using GitLab Self-Managed records, an Incoming Webhook, and Apache DevLake. You will publish a source-labelled Grafana dashboard with a private evidence release.

What You'll Build

Picture showing a decision-maker a source-labelled dashboard that traces five delivery measures to real records while a negative-control test exposes a falsely improved Change Failure Rate.

By the end of this project, you'll have:

A released governance pack you can use to defend each measure's calculation, clock boundary, source, exclusions, ownership, and caveats.
A five-measure dashboard that shows real GitLab deployment evidence plus independent incident records, with every panel naming its source.
A visible negative control that proves Change Failure Rate drops when incident ingestion is removed, then returns after the connection is restored.
Secret Mission: Ask an independent reviewer to challenge one measure and test its documented reversal trigger.

Are there any prerequisites?

You need a Windows workstation with Docker Desktop, Git for Windows, GitHub, and Linear access. Your GitLab Self-Managed 19.3.1 instance must be reachable on Linux because Docker for Windows is not officially supported as its container host.

Before We Start

Before any tools or dashboards enter the picture, define the federal program or service this measurement contract represents. Naming its decision-maker, one production-service boundary, and the decision at stake gives every later measure a clear governance purpose.

Stand Up the Measurement Workbench

Your service boundary now has a named decision behind it.

The next risk is hidden infrastructure that cannot be reached or reproduced. A defensible measurement build needs every runtime plus every external dependency to be explicit and testable.

You will first prove the Windows toolchain is ready. You will then start Apache DevLake and verify its two local interfaces.

In this step, get ready to:

Validate the Windows toolchain plus required account access.
Launch Apache DevLake 1.0.2 with its bundled services.
Verify Config UI plus Grafana in the browser.

Validate the workstation and accounts

Your local checks cover Docker Desktop, Git for Windows, plus Windows PowerShell. These tools run the stack plus create the evidence commits later.

Docker Desktop may need a minute to start its engine. Ongoing startup activity is expected during this check.

Press the Windows key to open Windows search.
Type Docker Desktop into Windows search.
Press Enter to start Docker Desktop.
Wait for Docker Desktop to report that its engine is ready.
Press the Windows key to return to Windows search.
Type Windows PowerShell into Windows search.
Press Enter to open Windows PowerShell.
Check the four local prerequisites by running these commands:

docker --version
docker compose version
git --version
$PSVersionTable.PSVersion

What do these checks prove?

The first command confirms that the Docker command-line tool can reach Docker Desktop.
The second command confirms that the Docker Compose plugin is available.
The third command confirms that Git for Windows is available.
The final command displays the PowerShell version running in this session.

✔️ Every version meets the minimum

Your workstation is ready when Docker is 19.03.10 or later. Docker Compose must be 2.2.3 or later.

Git should print a version number. Windows PowerShell must report 5.1 or later.

ⓧ I see an older version

An older runtime can fail after the stack starts pulling images. Update the component below its required minimum before continuing.

Update Docker Desktop through the official Windows installation guide if Docker or Docker Compose is below the required version.
Update Windows by following the official Windows PowerShell guidance if Windows PowerShell is below 5.1.
Return to the four prerequisite checks after the update.

ⓧ A command is not found

A missing command means that its tool is unavailable in this PowerShell session. Install the missing dependency before continuing.

Install Docker Desktop through the official Windows installation guide if a Docker command is missing.
Install Git for Windows through the official Git download page if the Git command is missing.
Ask your workstation administrator to restore Windows PowerShell 5.1 or later if its version check cannot run.
Close Windows PowerShell after completing the installation.
Open a fresh Windows PowerShell session through Windows search.
Return to the four prerequisite checks.

Git writes your name plus email into every commit. Setting this identity now keeps the later evidence history attributable.

Configure your global Git commit identity by running these commands:

git config --global user.name "[[GIT_NAME="your name"]]"
git config --global user.email "[[GIT_EMAIL="you@example.com"]]"

What does this configuration do?

The global settings give future commits an author name plus email address. Git stores these values in your user-level configuration.

Verify the stored Git identity by running these commands:

git config --global user.name
git config --global user.email

What should the identity check show?

Each command reads one value from your global Git configuration. The output should show the name plus email address you entered.

You should see your chosen name on the first line. Your chosen email address should appear on the second line.

Is a Git identity value missing?

Return to the configuration commands. Check that both placeholders were replaced with your own values before you ran them.

Use help me diagnose the missing Git identity value.

The remote checks cover GitLab Self-Managed 19.3.1, GitHub, plus Linear. Each service carries part of the evidence path or close-out record.

Confirm access to the Linux-hosted GitLab endpoint by signing in at https://gitlab.example.com.
Confirm the GitLab Self-Managed version with your GitLab administrator.
Confirm permission to create a private project with your GitLab administrator.
Confirm GitHub access by signing in to your account.
Confirm Linear access by signing in to your workspace.

You should reach each signed-in workspace. Your GitLab administrator should confirm version 19.3.1 plus private-project access.

GitLab must use a Linux host

GitLab does not officially support its container image on Docker for Windows. This project requires the reachable Linux-hosted GitLab Self-Managed endpoint from the prerequisites.

Stop this setup if the Linux-hosted GitLab endpoint is unreachable.

Launch the DevLake stack

The official release supplies the Compose definition plus its environment template. Keeping both files in one named Desktop folder makes the stack reproducible.

Prepare the devlake-1.0.2 folder on your Desktop by running these commands:

Set-Location -Path ~/Desktop
New-Item -Path ./devlake-1.0.2 -ItemType Directory -Force
Set-Location -Path ./devlake-1.0.2
Get-Location

What do these commands do?

The first command moves PowerShell to your Desktop.
The second command creates the devlake-1.0.2 folder.
The third command moves PowerShell into that folder.
The final command prints the current folder path for confirmation.

You should see a path ending in Desktop\devlake-1.0.2.

Open the Apache DevLake v1.0.2 release in your browser.
Save docker-compose.yml inside the devlake-1.0.2 folder.
Save env.example inside the devlake-1.0.2 folder.
Switch back to the Windows PowerShell session from earlier.
Confirm both release files arrived by running this command:

Get-ChildItem

What does this command show?

PowerShell lists the contents of the current folder. This check catches a browser download that landed in another location.

You should see docker-compose.yml plus env.example in the output.

Are the release files missing?

Check the download destination in your browser. Move both files into the devlake-1.0.2 folder on your Desktop.

Use help me locate the missing DevLake release assets.

The environment template becomes the active .env file. Compose reads this file when it starts the services.

Rename the environment template plus confirm the resulting files by running these commands:

Rename-Item -Path "env.example" -NewName ".env"
Get-ChildItem

What changed in the folder?

The first command changes the template name to .env without changing its contents. The second command confirms that the renamed file exists.

You should now see docker-compose.yml plus .env in the folder listing.

The ENCRYPTION_SECRET protects tokens plus passwords that DevLake stores later. Treat the generated value as a credential because losing it can make stored secrets unreadable.

Generate a random 128-character uppercase value with your approved password manager.
Store the generated value in your approved secret manager.
Open .env in an approved text editor from File Explorer.
Find ENCRYPTION_SECRET in the file.
Paste the generated value after the equals sign.
Save .env.

The first launch can spend several minutes downloading service images. Continuous download output means Docker is still working.

Start the Apache DevLake stack in the background by running this command:

docker compose up -d

What does this command start?

Docker Compose reads docker-compose.yml plus .env. It starts Apache DevLake 1.0.2 with bundled MySQL 8 plus Grafana 11.6.0.

Before you check the containers, how many running services do you expect the release stack to report?

Inspect the Compose service status by running this command:

docker compose ps

What does the status check prove?

This command lists the running containers in the current Compose project. Their status shows whether the release stack started successfully.

You should see the DevLake services listed as running. The output should include services for MySQL, Config UI, plus Grafana.

Did a container fail to start?

Confirm Docker Desktop still reports that its engine is ready. Check that PowerShell remains inside the devlake-1.0.2 folder.

Check that docker-compose.yml plus .env appear in the folder. Confirm that ENCRYPTION_SECRET has a saved value.

Use help me diagnose the failed DevLake container.

Confirm every service is reachable

Container status proves that the processes started. The browser check proves that you can reach the interfaces needed for configuration plus reporting.

The DevLake service can take a few seconds to answer after its container starts. A brief wait during the first page load is normal.

Before you open the local addresses, which two interfaces do you expect the stack to expose?

Open http://localhost:4000 in your browser.
Confirm that the DevLake Config UI loads.
Open http://localhost:3002 in another browser tab.
Enter admin as the initial Grafana username if prompted.
Enter admin as the initial Grafana password if prompted.
Complete the Grafana sign-in.
Confirm that the Grafana interface loads.

Config UI is available at http://localhost:4000. Grafana is available at http://localhost:3002.

The Apache DevLake 1.0.2 release Docker Compose stack is now running locally with bundled MySQL 8 plus Grafana 11.6.0. The .env file contains a generated ENCRYPTION_SECRET.

Does an interface fail to load?

Refresh Config UI after a few seconds if the DevLake API is still starting. Return to the Compose status check if either address remains unavailable.

Confirm that Docker Desktop is running. Keep the Compose stack active while you use both interfaces.

Use help me diagnose the unreachable DevLake interface.

✔️ Awesome, I've got everything

Your workbench is ready. Keep the Compose stack running for the remaining steps.

ⓧ I'd like to double check the full setup

Apache DevLake 1.0.2 is running locally from its release docker-compose.yml file.
Bundled MySQL 8 appears as running in the Compose service status.
The .env file contains a generated ENCRYPTION_SECRET.
DevLake Config UI is available at http://localhost:4000.
Bundled Grafana 11.6.0 is available at http://localhost:3002.
No measurement governance files or DevLake data connections exist yet.

You have cleared the infrastructure risk: the local DevLake stack plus every external dependency are ready. Next, you will turn the five measurement boundaries into a governance pack.

Establish the Measure Contract

Your local Apache DevLake workbench is live. Its bundled MySQL and Grafana services are ready for evidence.

The next risk is definition drift. A measure contract prevents a convenient dashboard implementation from silently redefining the program's measures.

In this step, get ready to:

Define the five governed measures before collecting delivery data.
Record the four decisions that control source selection and rework classification.
Align GitLab, the Incoming Webhook, fidelity limits, and validation assertions across the governance pack.

Define the five measures

Each measure needs a fixed calculation, clock boundary, source, exclusion rule, owner, and publication gate. The first document captures those choices before the data pipeline can influence them.

Open the DevLake release folder that contains docker-compose.yml in your preferred text editor.
Create a governance folder beside docker-compose.yml using your editor's file sidebar.

You should see the new governance folder in the file sidebar.

Create measure-contract.md inside the governance folder using the file sidebar.
Define the five governed measures by pasting the content below:

# Federal Program Delivery Measure Contract

Decision owner: Principal DevOps Owner
Unit of analysis: One named production service
Reporting window: Dashboard-selected UTC interval
Publication rule: No measure is published without its source label and validation status.

| Measure | Contracted calculation | Clock boundary | Required source | Exclusions and caveats |
| --- | --- | --- | --- | --- |
| Deployment Frequency | Count successful production deployment days in the reporting window. | Successful production deployment finish time. | GitLab Deployments through DevLake `gitlab`. | Approval cadence is not reproduced by the mirror. Do not infer release demand from the fixture. |
| Change Lead Time | Median elapsed time from the first authored commit in a deployed change to its successful production deployment. | First commit authored time to production deployment finish time. | GitLab commits and merge requests through `gitextractor` and `gitlab`, joined to GitLab deployments by DevLake. | A timer beginning at pull-request open or merge is a proxy and must be labelled as such. This implementation uses DevLake's first-commit-to-deployment PR cycle calculation. |
| Change Failure Rate | Successful production deployments linked to at least one incident divided by successful production deployments. | Incident created after the causal deployment and before a later deployment. | GitLab deployments plus Incoming Webhook incidents. | Missing incidents make the rate improve falsely. Count each failed deployment once even if it creates multiple incidents. |
| Failed Deployment Recovery Time | Median elapsed time from the failed deployment finish to resolution of its linked incident. | Causal production deployment finish time to incident resolution time. | GitLab deployments plus Incoming Webhook incidents. | This was renamed and redefined from mean time to restore in 2023. It excludes unrelated outages. |
| Deployment Rework Rate | Successful production deployments whose deployed commit message begins with `rework:` divided by successful production deployments. | Rework production deployment finish time. | GitLab commit messages plus GitLab deployments, calculated by hand-written SQL. | The prefix is an explicit mirror classification for unplanned deployment work caused by a production incident. It is not inferred from timing alone. |

Historical interpretation:

- Deployment Rework Rate joined the software delivery measurement model in 2024.
- Four performance clusters emerged from the 2023 data and four again from the 2024 data. The cluster count is not predetermined.
- No claim is made that elite, high, medium, and low tiers were retired.
- The 2025 report's seven team profiles combine broader delivery, stability, and well-being signals. They are a different analytical lens, not renamed performance tiers.

Publication gate:

1. Every panel names its system of record.
2. All three validation assertions pass and retain observed evidence.
3. The fidelity note accompanies every release.
4. A decision owner approves any change to a numerator, denominator, clock, or classification rule.

What does this contract control?

The table fixes the calculation and clock boundary for each measure.
Change Lead Time starts at the first authored commit and ends at the successful production deployment.
Change Failure Rate and Failed Deployment Recovery Time depend on incident records from the Incoming Webhook.
Deployment Rework Rate uses the explicit rework: classification instead of a timing guess.

Save governance/measure-contract.md.
Scroll through the saved file to inspect the measure table and publication gate.

You should see five measure rows. You should also see four historical interpretation points followed by four publication requirements.

Does the contract look incomplete?

Check that measure-contract.md sits inside the governance folder. Confirm that the five table rows still begin and end with pipe characters.

If the table columns do not align in your editor, compare the separator row with the reference below.

help me check my measure contract against the required five measures

The contract states what the program measures. Decision records preserve why those definitions and sources were chosen.

Create decisions.md inside the governance folder using the file sidebar.
Record the four measurement decisions by pasting the content below:

# Measurement Decision Records

## ADR-001: Use Apache DevLake as the self-hosted measurement platform

Decision: Use Apache DevLake 1.0.2 with its bundled MySQL and Grafana services.
Alternative beaten: The archived DORA Four Keys implementation.
Tradeoff accepted: DevLake's dashboard SQL and data model must be inspected and governed rather than accepted as neutral defaults.
Reversal trigger: A supported release cannot ingest the required GitLab deployments, commit history, and webhook incidents or cannot expose auditable SQL.

## ADR-002: Use GitLab Self-Managed as both commit and deployment system of record

Decision: Use one GitLab connection that triggers the native `gitlab` and `gitextractor` plugins.
Alternative beaten: Posting deployments through DevLake's generic webhook.
Tradeoff accepted: The build needs two tokens, repository access, a production-environment rule, and a reachable Linux-hosted GitLab instance.
Reversal trigger: The federal program changes its authoritative source forge or GitLab deployment records cannot represent production releases.

## ADR-003: Use Incoming Webhook for incidents

Decision: Send incident records through DevLake Incoming Webhook with `type` set to `INCIDENT`.
Alternative beaten: Treating GitLab issues as a native incident source.
Tradeoff accepted: The incident sender owns payload completeness, timestamps, uniqueness, and delivery monitoring.
Reversal trigger: The authoritative incident platform gains a supported native DevLake connector with equivalent project mapping and timestamp fidelity.

## ADR-004: Govern five measures and classify rework explicitly

Decision: Publish the four keys plus Deployment Rework Rate, using the `rework:` merge commit prefix as the mirror's explicit unplanned-work marker.
Alternative beaten: Inferring rework from short time gaps after an incident or publishing only the four built-in panels.
Tradeoff accepted: Contributors must apply the prefix correctly, and the mirror cannot prove the production classification process.
Reversal trigger: The production delivery system exposes a governed structured rework field or incident-to-remediation deployment relationship.

What do these decisions preserve?

Each record names the selected approach.
Each record identifies the alternative that was defeated.
Each record accepts a concrete tradeoff.
Each record defines the evidence that would justify a new decision.

Save governance/decisions.md.
Inspect the saved file for records ADR-001 through ADR-004.

You should see four records. Every record contains a decision, defeated alternative, accepted tradeoff, and reversal trigger.

Missing part of a decision record?

Compare each record against the same four labels. A missing reversal trigger leaves the decision without a governed way to change course.

help me find the incomplete decision record

✔️ Awesome, I've got everything!

Your measure contract and four decision records are saved inside governance.

ⓧ I'd like to double check the full code

Compare your complete governance/measure-contract.md file with this reference:

# Federal Program Delivery Measure Contract

Decision owner: Principal DevOps Owner
Unit of analysis: One named production service
Reporting window: Dashboard-selected UTC interval
Publication rule: No measure is published without its source label and validation status.

| Measure | Contracted calculation | Clock boundary | Required source | Exclusions and caveats |
| --- | --- | --- | --- | --- |
| Deployment Frequency | Count successful production deployment days in the reporting window. | Successful production deployment finish time. | GitLab Deployments through DevLake `gitlab`. | Approval cadence is not reproduced by the mirror. Do not infer release demand from the fixture. |
| Change Lead Time | Median elapsed time from the first authored commit in a deployed change to its successful production deployment. | First commit authored time to production deployment finish time. | GitLab commits and merge requests through `gitextractor` and `gitlab`, joined to GitLab deployments by DevLake. | A timer beginning at pull-request open or merge is a proxy and must be labelled as such. This implementation uses DevLake's first-commit-to-deployment PR cycle calculation. |
| Change Failure Rate | Successful production deployments linked to at least one incident divided by successful production deployments. | Incident created after the causal deployment and before a later deployment. | GitLab deployments plus Incoming Webhook incidents. | Missing incidents make the rate improve falsely. Count each failed deployment once even if it creates multiple incidents. |
| Failed Deployment Recovery Time | Median elapsed time from the failed deployment finish to resolution of its linked incident. | Causal production deployment finish time to incident resolution time. | GitLab deployments plus Incoming Webhook incidents. | This was renamed and redefined from mean time to restore in 2023. It excludes unrelated outages. |
| Deployment Rework Rate | Successful production deployments whose deployed commit message begins with `rework:` divided by successful production deployments. | Rework production deployment finish time. | GitLab commit messages plus GitLab deployments, calculated by hand-written SQL. | The prefix is an explicit mirror classification for unplanned deployment work caused by a production incident. It is not inferred from timing alone. |

Historical interpretation:

- Deployment Rework Rate joined the software delivery measurement model in 2024.
- Four performance clusters emerged from the 2023 data and four again from the 2024 data. The cluster count is not predetermined.
- No claim is made that elite, high, medium, and low tiers were retired.
- The 2025 report's seven team profiles combine broader delivery, stability, and well-being signals. They are a different analytical lens, not renamed performance tiers.

Publication gate:

1. Every panel names its system of record.
2. All three validation assertions pass and retain observed evidence.
3. The fidelity note accompanies every release.
4. A decision owner approves any change to a numerator, denominator, clock, or classification rule.

How to compare this file

Check the owner and publication rule first. Compare every measure row before checking the historical interpretation and publication gate.

Compare your complete governance/decisions.md file with this reference:

# Measurement Decision Records

## ADR-001: Use Apache DevLake as the self-hosted measurement platform

Decision: Use Apache DevLake 1.0.2 with its bundled MySQL and Grafana services.
Alternative beaten: The archived DORA Four Keys implementation.
Tradeoff accepted: DevLake's dashboard SQL and data model must be inspected and governed rather than accepted as neutral defaults.
Reversal trigger: A supported release cannot ingest the required GitLab deployments, commit history, and webhook incidents or cannot expose auditable SQL.

## ADR-002: Use GitLab Self-Managed as both commit and deployment system of record

Decision: Use one GitLab connection that triggers the native `gitlab` and `gitextractor` plugins.
Alternative beaten: Posting deployments through DevLake's generic webhook.
Tradeoff accepted: The build needs two tokens, repository access, a production-environment rule, and a reachable Linux-hosted GitLab instance.
Reversal trigger: The federal program changes its authoritative source forge or GitLab deployment records cannot represent production releases.

## ADR-003: Use Incoming Webhook for incidents

Decision: Send incident records through DevLake Incoming Webhook with `type` set to `INCIDENT`.
Alternative beaten: Treating GitLab issues as a native incident source.
Tradeoff accepted: The incident sender owns payload completeness, timestamps, uniqueness, and delivery monitoring.
Reversal trigger: The authoritative incident platform gains a supported native DevLake connector with equivalent project mapping and timestamp fidelity.

## ADR-004: Govern five measures and classify rework explicitly

Decision: Publish the four keys plus Deployment Rework Rate, using the `rework:` merge commit prefix as the mirror's explicit unplanned-work marker.
Alternative beaten: Inferring rework from short time gaps after an incident or publishing only the four built-in panels.
Tradeoff accepted: Contributors must apply the prefix correctly, and the mirror cannot prove the production classification process.
Reversal trigger: The production delivery system exposes a governed structured rework field or incident-to-remediation deployment relationship.

How to compare these records

Confirm that all four ADR headings are present. Check that every record preserves the same four decision labels.

Map the topology and implementation plan

A topology makes source ownership visible. This one keeps GitLab evidence separate from the only valid incident path.

The diagram uses Mermaid to trace evidence through collection, storage, presentation, release, and close-out.

Create topology.md inside the governance folder using the file sidebar.
Map the controlled evidence flow by pasting the content below:

# Measurement Topology

~~~mermaid
flowchart LR
    GL[GitLab Self-Managed\ncommits, MRs, deployments]
    WH[Incoming Webhook\nINCIDENT records]
    GP[DevLake gitlab plugin\ndeployments and MRs]
    GE[DevLake gitextractor plugin\ncommits and diffs]
    DL[Apache DevLake 1.0.2\nDORA enrichment]
    DB[(MySQL 8)]
    GF[Grafana 11.6.0\nfive labelled measures]
    GH[Private GitHub release\ngovernance evidence]
    LI[Linear\nCompleted close-out]

    GL --> GP
    GL --> GE
    WH --> DL
    GP --> DL
    GE --> DL
    DL --> DB
    DB --> GF
    GF --> GH
    GH --> LI
~~~

Control boundary: GitLab never supplies incident records in this build. The webhook is the only incident path.

What does this topology prove?

GitLab supplies commits, merge requests, and deployments through two native plugin paths.
The Incoming Webhook supplies every INCIDENT record.
DevLake writes the enriched evidence to MySQL before Grafana presents it.
The released evidence then moves through GitHub before the Linear close-out.

Save governance/topology.md.
Inspect the saved topology for two arrows leaving GitLab and one arrow leaving the Incoming Webhook.

You should see the GitLab paths converge at DevLake. The incident path should enter DevLake directly from the Incoming Webhook.

Does the topology lose a source path?

Check that the Mermaid fence begins and ends with three tilde characters. Confirm that both GL --> GP and GL --> GE remain in the file.

help me inspect the missing topology path

The topology establishes where evidence moves. The implementation plan fixes the order used to build and close the measurement system.

Create implementation-plan.md inside the governance folder using the file sidebar.
Set the controlled implementation sequence by pasting the content below:

# Implementation Plan

1. Validate the Windows workstation, DevLake stack, accounts, and external Linux-hosted GitLab endpoint.
2. Approve the five-measure contract, four decisions, topology, fidelity limits, and validation assertions.
3. Seed the GitLab mirror and ingest merge-request commits and deployments through `gitlab` and `gitextractor`.
4. Add the Incoming Webhook, publish an `INCIDENT`, and re-run collection to create the incident-to-deployment relationship.
5. Add Deployment Rework Rate, label all sources, remove or relabel benchmark panels, execute the three assertions, publish the private GitHub release, and complete the Linear project.

Completion rule: The project closes only when the dashboard and released governance artifacts agree.

Why fix the order now?

The sequence puts governance approval before ingestion. It places dashboard publication after source wiring and validation.

The completion rule requires the released documents to agree with the displayed values.

Save governance/implementation-plan.md.
Inspect the saved plan for five numbered stages and one completion rule.

You should see governance approval in stage two. You should see dashboard publication and close-out in stage five.

Does the plan contradict the topology?

Confirm that stage three names both gitlab and gitextractor. Check that stage four keeps INCIDENT records on the Incoming Webhook path.

help me compare the implementation plan with the topology

✔️ Awesome, I've got everything!

Your source topology and implementation plan now agree on the controlled build sequence.

ⓧ I'd like to double check the full code

Compare your complete governance/topology.md file with this reference:

# Measurement Topology

~~~mermaid
flowchart LR
    GL[GitLab Self-Managed\ncommits, MRs, deployments]
    WH[Incoming Webhook\nINCIDENT records]
    GP[DevLake gitlab plugin\ndeployments and MRs]
    GE[DevLake gitextractor plugin\ncommits and diffs]
    DL[Apache DevLake 1.0.2\nDORA enrichment]
    DB[(MySQL 8)]
    GF[Grafana 11.6.0\nfive labelled measures]
    GH[Private GitHub release\ngovernance evidence]
    LI[Linear\nCompleted close-out]

    GL --> GP
    GL --> GE
    WH --> DL
    GP --> DL
    GE --> DL
    DL --> DB
    DB --> GF
    GF --> GH
    GH --> LI
~~~

Control boundary: GitLab never supplies incident records in this build. The webhook is the only incident path.

How to compare the topology

Compare each source node and arrow. Finish by checking the control boundary beneath the Mermaid diagram.

Compare your complete governance/implementation-plan.md file with this reference:

# Implementation Plan

1. Validate the Windows workstation, DevLake stack, accounts, and external Linux-hosted GitLab endpoint.
2. Approve the five-measure contract, four decisions, topology, fidelity limits, and validation assertions.
3. Seed the GitLab mirror and ingest merge-request commits and deployments through `gitlab` and `gitextractor`.
4. Add the Incoming Webhook, publish an `INCIDENT`, and re-run collection to create the incident-to-deployment relationship.
5. Add Deployment Rework Rate, label all sources, remove or relabel benchmark panels, execute the three assertions, publish the private GitHub release, and complete the Linear project.

Completion rule: The project closes only when the dashboard and released governance artifacts agree.

How to compare the plan

Confirm the five stages remain in order. Check the completion rule after the numbered list.

Record fidelity limits and assertions

A controlled mirror can prove source wiring and query behavior. Its fidelity note must disclose the parts of the federal operating environment that the mirror does not reproduce.

Create fidelity-note.md inside the governance folder using the file sidebar.
Document the mirror's interpretation limits by pasting the content below:

# Mirror Fidelity Note

This mirror proves source wiring, measure boundaries, incident dependency, query behavior, and governance traceability. It does not reproduce the federal program's operating environment.

Not reproduced:

- The approval-gated cadence that is the real constraint on Deployment Frequency.
- Production change windows, change advisory review, separation of duties, and emergency authorization.
- Production repository count, parallel release trains, long-running branches, and clock skew between systems.
- Real incident detection latency, severity policy, customer impact, and post-incident adjudication.
- The production process for deciding whether work is rework. The mirror uses the explicit `rework:` prefix.
- Historical data volume, missing-data behavior, organizational boundaries, and records-retention controls.

Interpretation limit: The mirror validates the measurement mechanism. Its values are test evidence, not a benchmark or forecast for the federal program.

Why disclose these limits?

The mirror validates measurement mechanics under controlled conditions. It cannot establish the federal program's real deployment pace or operational performance.

The note keeps fixture values from being presented as a benchmark or forecast.

Save governance/fidelity-note.md.
Inspect the saved note for six limitations and one interpretation limit.

You should see approval-gated cadence at the top of the limitation list. You should see records-retention controls in the final limitation.

Is a fidelity limit missing?

Check that the file covers cadence, production controls, scale, incidents, rework classification, and retention. The final paragraph should restrict the mirror values to test evidence.

help me check the mirror fidelity limits

Validation assertions define success before results exist. Their observed fields remain pending until later steps exercise the real data paths.

Create validation.md inside the governance folder using the file sidebar.
Define the three assertions by pasting the content below:

# Validation Record

## Assertion 1: Native deployment and commit lineage

Expected: Four successful `PRODUCTION` deployments originate in GitLab. Their deployment SHAs exist in merge-request commit history collected through `gitextractor`. Deployment Frequency is non-zero and Change Lead Time is derived from first authored commit to production deployment.
Observed: Pending execution.
Evidence: GitLab deployment list, DevLake collection status, dashboard values, and queried deployment rows.

## Assertion 2: Incident-dependent measures and rework

Expected: One webhook record with `type` equal to `INCIDENT` maps to the immediately preceding successful production deployment. With four deployments and one affected deployment, Change Failure Rate is 25%. Failed Deployment Recovery Time is non-empty. One `rework:` deployment produces a Deployment Rework Rate of 25%.
Observed: Pending execution.
Evidence: Webhook response, relationship row, four dashboard panels, and the hand-written rework panel.

## Assertion 3: Deliberate incident-path break

Expected: Record the correct Change Failure Rate. Remove the Incoming Webhook connection from the DevLake project and run `Collect Data`. Refresh Grafana and observe Change Failure Rate improve because the project-scoped incident relationships were rebuilt without incident input. Reattach the webhook, run `Collect Data`, and observe the original rate return.
Observed: Pending execution.
Evidence: Before, broken, and restored values with timestamps. A written description without observed values fails this assertion.

What will these assertions test?

Assertion 1 tests native deployment and commit lineage.
Assertion 2 tests incident correlation and explicit rework classification.
Assertion 3 tests whether removing incident ingestion creates a falsely improved Change Failure Rate.
Every observed result stays pending until the associated test runs.

Save governance/validation.md.
Inspect the saved record for three expected results and three pending observed fields.

You should see one assertion for lineage, one for incidents and rework, and one for the deliberate webhook break.

Do the assertions reveal results too early?

Confirm that every observed field says Pending execution. The expected fields define the test without claiming that evidence already exists.

help me separate expected and observed validation evidence

✔️ Awesome, I've got everything!

Your fidelity limits and three pre-defined assertions complete the governance pack.

ⓧ I'd like to double check the full code

Compare your complete governance/fidelity-note.md file with this reference:

# Mirror Fidelity Note

This mirror proves source wiring, measure boundaries, incident dependency, query behavior, and governance traceability. It does not reproduce the federal program's operating environment.

Not reproduced:

- The approval-gated cadence that is the real constraint on Deployment Frequency.
- Production change windows, change advisory review, separation of duties, and emergency authorization.
- Production repository count, parallel release trains, long-running branches, and clock skew between systems.
- Real incident detection latency, severity policy, customer impact, and post-incident adjudication.
- The production process for deciding whether work is rework. The mirror uses the explicit `rework:` prefix.
- Historical data volume, missing-data behavior, organizational boundaries, and records-retention controls.

Interpretation limit: The mirror validates the measurement mechanism. Its values are test evidence, not a benchmark or forecast for the federal program.

How to compare the fidelity note

Compare all six limitation bullets. Check the interpretation limit beneath them.

Compare your complete governance/validation.md file with this reference:

# Validation Record

## Assertion 1: Native deployment and commit lineage

Expected: Four successful `PRODUCTION` deployments originate in GitLab. Their deployment SHAs exist in merge-request commit history collected through `gitextractor`. Deployment Frequency is non-zero and Change Lead Time is derived from first authored commit to production deployment.
Observed: Pending execution.
Evidence: GitLab deployment list, DevLake collection status, dashboard values, and queried deployment rows.

## Assertion 2: Incident-dependent measures and rework

Expected: One webhook record with `type` equal to `INCIDENT` maps to the immediately preceding successful production deployment. With four deployments and one affected deployment, Change Failure Rate is 25%. Failed Deployment Recovery Time is non-empty. One `rework:` deployment produces a Deployment Rework Rate of 25%.
Observed: Pending execution.
Evidence: Webhook response, relationship row, four dashboard panels, and the hand-written rework panel.

## Assertion 3: Deliberate incident-path break

Expected: Record the correct Change Failure Rate. Remove the Incoming Webhook connection from the DevLake project and run `Collect Data`. Refresh Grafana and observe Change Failure Rate improve because the project-scoped incident relationships were rebuilt without incident input. Reattach the webhook, run `Collect Data`, and observe the original rate return.
Observed: Pending execution.
Evidence: Before, broken, and restored values with timestamps. A written description without observed values fails this assertion.

How to compare the assertions

Check each expected result against its evidence list. Confirm that all three observed fields remain pending.

Before the final check, predict whether every incident-backed measure still points only to the Incoming Webhook.

Inspect all six saved files under the governance folder.

You should see five measure rows, four decision records, one source topology, one five-stage plan, six fidelity limitations, and three pending assertions. The same source boundaries should appear across every file.

Your governance pack now fixes the measures before collection begins. Next, you will wire GitLab commits and production deployments through DevLake's two native plugin paths.

Ingest GitLab Delivery Evidence

Your governance pack now fixes the five delivery measures before any dashboard value can shape a decision. The next test is whether real delivery evidence follows those boundaries.

A private GitLab Self-Managed mirror supplies the commit history plus deployment evidence for Apache DevLake. This step seeds a controlled delivery flow so you can trace each value back to its source.

In this step, get ready to:

Create a private GitLab mirror with separate read-only and writer tokens.
Seed a controlled Git history with two merged changes plus three production deployments.
Collect the delivery evidence through DevLake before checking the resulting Grafana measures.

Create the GitLab evidence source

The mirror needs a blank history because the seeding script creates every commit used by the measures. Keeping the project private also protects the controlled evidence while you test the measurement path.

Return to your signed-in GitLab Self-Managed instance.
Select Create new.
Select New project/repository.
Select Create blank project.
Enter federal-delivery-mirror in the Project name field.
Select Private under Visibility Level.
Keep repository initialization disabled so the project has no README.
Select Create project.

Your private mirror now exists with an empty repository. That blank starting point lets the script create a predictable baseline.

Record the repository clone URL as your GitLab repository URL.
Record the numeric project identifier as your GitLab project ID.
Record the instance address without an API path as https://gitlab.example.com.

Two identities keep collection separate from evidence creation. The DevLake token only reads repository data.

The writer token creates branches plus merge requests. It also registers deployments through the GitLab API.

Select your GitLab avatar.
Select Edit profile.
Select Personal access tokens from the sidebar.
Select Add new token.

Store Each Token Safely

Credential handling can feel risky here. The split limits the collector to read access if that token is ever exposed.

GitLab displays a new token value only once. Avoid screenshots because the reveal contains a live credential.

Enter devlake-read in the Token name field.
Choose an Expiration date that meets your organization's token policy.
Select the read_api scope.
Select Create personal access token.
Copy the revealed token before leaving the page.
Store the token in your password manager under the name devlake-read.

Return to Personal access tokens.
Select Add new token.
Enter mirror-writer in the Token name field.
Choose an Expiration date that meets your organization's token policy.
Select the api scope.
Select Create personal access token.
Copy the revealed token before leaving the page.
Store the token in your password manager under the name mirror-writer.

The identity split is now in place. DevLake can collect evidence without receiving permission to alter the mirror.

Add and run the mirror automation

Three PowerShell scripts create a repeatable evidence trail. One publishes deployments while the other two build changes around it.

Return to the editor containing the governance folder from the previous step.
Create a scripts folder beside governance.

You should now see scripts beside governance in the editor's file tree.

Create scripts/publish-deployment.ps1 using the complete reference below.

✔️ Awesome, I've got everything!

Confirm that scripts/publish-deployment.ps1 contains the deployment publisher before saving it.

ⓧ I'd like to double check the full code

Compare scripts/publish-deployment.ps1 with this complete reference:

param(
    [Parameter(Mandatory = $true)]
    [string]$GitLabBaseUrl,
    [Parameter(Mandatory = $true)]
    [string]$ProjectId,
    [Parameter(Mandatory = $true)]
    [string]$Token,
    [Parameter(Mandatory = $true)]
    [string]$Sha,
    [ValidateSet("running", "success", "failed", "canceled")]
    [string]$Status = "success",
    [string]$Environment = "production",
    [string]$Ref = "main"
)

$encodedProjectId = [Uri]::EscapeDataString($ProjectId)
$uri = "$($GitLabBaseUrl.TrimEnd('/'))/api/v4/projects/$encodedProjectId/deployments"
$headers = @{ "PRIVATE-TOKEN" = $Token }
$body = @{
    environment = $Environment
    sha = $Sha
    ref = $Ref
    tag = $false
    status = $Status
}

Invoke-RestMethod `
    -Method Post `
    -Uri $uri `
    -Headers $headers `
    -Body $body `
    -ContentType "application/x-www-form-urlencoded"

How Does This Script Work?

The parameters identify the GitLab project plus the commit being deployed.
The PRIVATE-TOKEN header authenticates the writer token.
The form body records a successful deployment to production on main by default.
Invoke-RestMethod posts that record to the GitLab Deployment API.

Save scripts/publish-deployment.ps1.

The deployment publisher should now appear as a saved file under scripts.

Create scripts/publish-change.ps1 using the complete reference below.

✔️ Awesome, I've got everything!

Confirm that scripts/publish-change.ps1 contains the change publisher before saving it.

ⓧ I'd like to double check the full code

Compare scripts/publish-change.ps1 with this complete reference:

param(
    [Parameter(Mandatory = $true)]
    [string]$GitLabBaseUrl,
    [Parameter(Mandatory = $true)]
    [string]$ProjectId,
    [Parameter(Mandatory = $true)]
    [string]$Token,
    [Parameter(Mandatory = $true)]
    [string]$Branch,
    [Parameter(Mandatory = $true)]
    [string]$Message,
    [Parameter(Mandatory = $true)]
    [string]$State
)

$encodedProjectId = [Uri]::EscapeDataString($ProjectId)
$headers = @{ "PRIVATE-TOKEN" = $Token }
$mergeRequestsUri = "$($GitLabBaseUrl.TrimEnd('/'))/api/v4/projects/$encodedProjectId/merge_requests"

git switch -c $Branch
Set-Content -Path "service-state.txt" -Value $State
git add service-state.txt
git commit -m $Message
git push -u origin $Branch

$mergeRequest = Invoke-RestMethod `
    -Method Post `
    -Uri $mergeRequestsUri `
    -Headers $headers `
    -Body @{
        source_branch = $Branch
        target_branch = "main"
        title = $Message
        remove_source_branch = $true
    } `
    -ContentType "application/x-www-form-urlencoded"

$mergeRequestUri = "$mergeRequestsUri/$($mergeRequest.iid)"
$mergeStatus = $null
for ($attempt = 1; $attempt -le 10; $attempt++) {
    $mergeStatus = Invoke-RestMethod -Method Get -Uri $mergeRequestUri -Headers $headers
    if ($mergeStatus.detailed_merge_status -eq "mergeable") {
        break
    }
    Start-Sleep -Seconds 1
}

if ($mergeStatus.detailed_merge_status -ne "mergeable") {
    throw "Merge request did not become mergeable."
}

$merged = Invoke-RestMethod `
    -Method Put `
    -Uri "$mergeRequestUri/merge" `
    -Headers $headers `
    -Body @{
        merge_commit_message = $Message
        should_remove_source_branch = $true
    } `
    -ContentType "application/x-www-form-urlencoded"

git switch main
git pull origin main

& "$PSScriptRoot\publish-deployment.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Sha $merged.merge_commit_sha

How Does This Script Work?

The script creates a branch containing a new service-state.txt value.
The GitLab API opens the merge request plus completes its merge after the status becomes mergeable.
The polling loop allows up to ten checks for GitLab to calculate mergeability.
The final call publishes the merged commit as a production deployment.

Save scripts/publish-change.ps1.

The change publisher should now appear beside publish-deployment.ps1.

Create scripts/seed-mirror.ps1 using the complete reference below.

✔️ Awesome, I've got everything!

Confirm that scripts/seed-mirror.ps1 contains the baseline plus two controlled changes before saving it.

ⓧ I'd like to double check the full code

Compare scripts/seed-mirror.ps1 with this complete reference:

param(
    [Parameter(Mandatory = $true)]
    [string]$GitLabRepoUrl,
    [Parameter(Mandatory = $true)]
    [string]$GitLabBaseUrl,
    [Parameter(Mandatory = $true)]
    [string]$ProjectId,
    [Parameter(Mandatory = $true)]
    [string]$Token
)

git init -b main
git remote add origin $GitLabRepoUrl
Set-Content -Path "service-state.txt" -Value "baseline"
git add service-state.txt
git commit -m "baseline: create service"
git push -u origin main

$baselineSha = (git rev-parse --verify HEAD).Trim()
& "$PSScriptRoot\publish-deployment.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Sha $baselineSha

& "$PSScriptRoot\publish-change.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Branch "change-eligibility" `
    -Message "change: add eligibility path" `
    -State "eligibility"

& "$PSScriptRoot\publish-change.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Branch "change-routing-defect" `
    -Message "change: introduce routing defect" `
    -State "defect"

How Does This Script Work?

The first section initializes main with a baseline service state.
The baseline commit becomes the first successful production deployment.
The change-eligibility branch creates the first merge-request history.
The change-routing-defect branch creates the second merge-request history.

Save scripts/seed-mirror.ps1.

All three automation files should now appear under scripts.

The seed script makes the delivery history observable in one run. Its final checkout pulls the latest merged state back into your local mirror.

Return to the Windows PowerShell panel positioned in the local folder that contains governance plus scripts.
Execute scripts/seed-mirror.ps1 from that folder.
Enter your GitLab repository URL when PowerShell requests GitLabRepoUrl.
Enter https://gitlab.example.com when PowerShell requests GitLabBaseUrl.
Enter your GitLab project ID when PowerShell requests ProjectId.
Paste the stored mirror-writer token when PowerShell requests Token.

The run creates a baseline plus two merged change sets. It also registers one successful production deployment for each resulting state.

Return to the editor after the script completes.
Open service-state.txt from the local mirror folder.

You should see defect as the current service state. This confirms the latest seeded change was merged back into main.

Return to federal-delivery-mirror in GitLab.
Confirm the project history contains the baseline commit.
Confirm the project history contains two merged change sets.
Confirm the deployment history contains three successful records for production.

Did the Seed Run Stop?

Check that the project is blank if the initial push is rejected. Existing repository content can conflict with the scripted baseline.

If you see Merge request did not become mergeable. verify that the writer token has the api scope.

Confirm that your Git commit identity remains configured if the baseline commit fails.

help me diagnose the failed GitLab mirror seed

Collect and verify native evidence

GitLab is one DevLake data source that triggers two native paths. The gitlab plugin collects platform records while gitextractor supplies commit history for lead-time calculation.

Switch back to the DevLake Config UI at http://localhost:4000.
Add a GitLab Server connection.
Enter https://gitlab.example.com/api/v4/ as the connection endpoint.
Paste the stored devlake-read token into the connection's token field.
Complete the connection setup so DevLake can list repositories.

The endpoint must finish with /api/v4/. The trailing slash is part of DevLake's expected GitLab Server endpoint format.

Select federal-delivery-mirror from the repository list.
Retain the Source Code Management entity.
Retain the Code Review entity.
Retain the CI/CD entity.
Set the production environment regular expression to (?i)^production$.
Add the configured connection to the DevLake project.

Before you collect, consider which measures GitLab evidence can support on its own.

Start ingestion with the project's Collect Data action.
Wait for the collection pipeline to reach its completed state.
Review the collection details for the gitlab plugin.
Review the collection details for the gitextractor plugin.

The completed collection should include three successful PRODUCTION deployments. It should also include the merged-request commit history.

Collection Did Not Complete?

Confirm that the GitLab endpoint ends with /api/v4/ if DevLake cannot reach the server.

Confirm that the collector token has read_api if the repository cannot be listed.

Check (?i)^production$ if the deployments do not map to PRODUCTION.

help me diagnose my DevLake GitLab collection

Before you refresh the dashboard, consider what valid evidence each measure currently has.

Switch back to Grafana at http://localhost:3002.
Open the built-in DORA views for the DevLake project.
Refresh the dashboard after the completed collection.

Deployment Frequency plus Change Lead Time can now render from GitLab evidence. The incident-backed measures have no valid incident input.

That visible shortfall is intentional. It proves that delivery activity alone cannot establish failure or recovery performance.

Your mirror now supplies traceable commits plus production deployments through both native plugin paths. Next, you will add the independent incident evidence needed to complete the stability measures.

Connect Incident Evidence

Your GitLab mirror now feeds commits plus production deployments into Apache DevLake. That lineage already supports two DORA measures: Deployment Frequency plus Change Lead Time.

The stability measures depend on independent incident evidence. This step adds an Incoming Webhook so DevLake can connect a resolved incident to the preceding production deployment.

In this step, get ready to:

Create an Incoming Webhook connection in the existing DevLake project.
Post one resolved INCIDENT fixture through the generated issue URL.
Recollect the rework deployment to populate both stability measures.

Create the Incoming Webhook path

The Incoming Webhook acts as the controlled incident source for this mirror. Mapping it to the existing DevLake project lets one collection combine incident evidence with the GitLab deployment timeline.

Switch back to the DevLake Config UI from earlier.
Select the DevLake project that already contains the GitLab Server connection.

You should see the existing GitLab connection mapped to this project. Its previous collection remains available.

Add an Incoming Webhook connection to the same DevLake project.
Copy the generated issue URL into this project value: your generated issue URL.

You should now see the GitLab Server connection plus the Incoming Webhook connection mapped to one project. This shared mapping gives DevLake both sides of the incident-to-deployment relationship.

Why Incidents Need a Separate Path

GitLab supplies the commit history plus deployment records in this build. The Incoming Webhook supplies the incident timestamps that the stability measures require.

Missing incident evidence can make Change Failure Rate appear lower. Keeping this path explicit protects the dashboard from that false improvement.

Post the incident and publish the rework

The controlled incident needs explicit creation plus resolution timestamps. Posting those values together gives DevLake a complete record for Change Failure Rate plus Failed Deployment Recovery Time.

Create scripts/send-incident.ps1 inside the local mirror working directory by adding this code:

param(
    [Parameter(Mandatory = $true)]
    [string]$WebhookUrl,
    [string]$IssueKey = "FED-INC-001"
)

$now = (Get-Date).ToUniversalTime().ToString("o")
$payload = @{
    issueKey = $IssueKey
    title = "Production routing regression"
    description = "Controlled incident for measurement-path validation"
    type = "INCIDENT"
    status = "DONE"
    originalStatus = "resolved"
    createdDate = $now
    updatedDate = $now
    resolutionDate = $now
    severity = "controlled-test"
} | ConvertTo-Json

Invoke-RestMethod `
    -Method Post `
    -Uri $WebhookUrl `
    -Body $payload `
    -ContentType "application/json"

What Does This Script Do?

The $now value records one UTC timestamp for the incident lifecycle.
The $payload object identifies FED-INC-001 as a resolved INCIDENT with controlled-test severity.
The createdDate, updatedDate, plus resolutionDate fields make the timing boundary explicit.
The Invoke-RestMethod call converts the payload into incident evidence at the generated webhook URL.

Save scripts/send-incident.ps1.

✔️ Awesome, I've got everything!

Great. Make sure scripts/send-incident.ps1 is saved inside the scripts directory.

ⓧ I'd like to double check the full code

Compare your complete scripts/send-incident.ps1 file with this reference:

param(
    [Parameter(Mandatory = $true)]
    [string]$WebhookUrl,
    [string]$IssueKey = "FED-INC-001"
)

$now = (Get-Date).ToUniversalTime().ToString("o")
$payload = @{
    issueKey = $IssueKey
    title = "Production routing regression"
    description = "Controlled incident for measurement-path validation"
    type = "INCIDENT"
    status = "DONE"
    originalStatus = "resolved"
    createdDate = $now
    updatedDate = $now
    resolutionDate = $now
    severity = "controlled-test"
} | ConvertTo-Json

Invoke-RestMethod `
    -Method Post `
    -Uri $WebhookUrl `
    -Body $payload `
    -ContentType "application/json"

Why This File Is Complete

This file builds the resolved incident payload. It also posts that payload to the Incoming Webhook connection.

Return to Windows PowerShell in the local mirror working directory.
Execute scripts/send-incident.ps1.
Paste your generated issue URL when PowerShell requests WebhookUrl.

The request should finish without a PowerShell error. DevLake now has one resolved incident named FED-INC-001.

Incident Request Rejected?

Confirm that WebhookUrl contains the generated issue URL from the Incoming Webhook connection. A connection page URL points to a different destination.

Check that your file preserves the INCIDENT type plus the DONE status exactly.

Help me diagnose my DevLake incident webhook request.

The incident now points back to the latest earlier successful production deployment. A fourth change provides the recovery deployment plus the explicit rework classification used later.

Create scripts/publish-rework.ps1 inside the local mirror working directory by adding this code:

param(
    [Parameter(Mandatory = $true)]
    [string]$GitLabBaseUrl,
    [Parameter(Mandatory = $true)]
    [string]$ProjectId,
    [Parameter(Mandatory = $true)]
    [string]$Token
)

& "$PSScriptRoot\publish-change.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Branch "rework-routing-incident" `
    -Message "rework: remediate production routing incident" `
    -State "routing-remediated"

What Does This Script Do?

The script reuses publish-change.ps1 to preserve the existing merge-request plus deployment workflow.
The rework-routing-incident branch holds the controlled remediation change.
The rework: remediate production routing incident message applies the contract's explicit rework classification.
The routing-remediated state becomes the new content of service-state.txt.

Save scripts/publish-rework.ps1.

✔️ Awesome, I've got everything!

Your rework publisher is ready. Keep it beside publish-change.ps1 in the scripts directory.

ⓧ I'd like to double check the full code

Compare your complete scripts/publish-rework.ps1 file with this reference:

param(
    [Parameter(Mandatory = $true)]
    [string]$GitLabBaseUrl,
    [Parameter(Mandatory = $true)]
    [string]$ProjectId,
    [Parameter(Mandatory = $true)]
    [string]$Token
)

& "$PSScriptRoot\publish-change.ps1" `
    -GitLabBaseUrl $GitLabBaseUrl `
    -ProjectId $ProjectId `
    -Token $Token `
    -Branch "rework-routing-incident" `
    -Message "rework: remediate production routing incident" `
    -State "routing-remediated"

Why This File Is Complete

This wrapper supplies the governed branch name plus commit classification. The existing publishing script handles the merge request plus deployment record.

Execute scripts/publish-rework.ps1 in PowerShell.
Enter the existing GitLab base URL when PowerShell requests GitLabBaseUrl.
Enter the private mirror project ID when PowerShell requests ProjectId.
Paste the writer token from earlier when PowerShell requests Token.

You should see the branch push plus merge workflow complete. GitLab should then show a fourth successful production deployment.

The checked-out service-state.txt file now contains routing-remediated.

Rework Change Did Not Merge?

If you see Merge request did not become mergeable. check the GitLab merge request for a branch conflict or repository rule.

Confirm that the supplied writer token still has the api scope. The read-only DevLake token cannot create this change.

Help me diagnose my rework publishing failure.

Collect the incident relationship

DevLake maps an incident to the latest successful PRODUCTION deployment with an earlier finish time. A fresh collection rebuilds that relationship from the GitLab timeline plus the webhook timestamp.

Switch back to the DevLake Config UI from earlier.
Select the project containing both configured connections.

You should see the GitLab Server connection plus the Incoming Webhook connection in the same project configuration.

Before you collect, do you expect the incident-backed measures to remain empty once both evidence paths belong to the project?

Click Collect Data for this DevLake project.

The collection may take a few minutes while DevLake refreshes the project data. Ongoing task activity during this wait is expected.

Confirm the collection pipeline completes successfully.
Return to the Grafana interface from earlier.
Refresh the built-in DORA dashboard.
Inspect Change Failure Rate for a non-zero value.
Inspect Failed Deployment Recovery Time for a populated value.

You should now see a non-zero Change Failure Rate plus a non-empty Failed Deployment Recovery Time. Those values confirm that DevLake linked the resolved incident to the preceding successful production deployment.

That closes the missing evidence path. All four native DORA measures now have traceable inputs from their contracted systems of record.

Incident Measures Still Empty?

Confirm that the Incoming Webhook connection belongs to the same DevLake project as the GitLab connection. Separate project mappings prevent the relationship from forming.

Run Collect Data after posting the incident. Also confirm that the payload retains an explicit resolutionDate value.

Help me trace the missing incident relationship.

Your measurement flow now joins GitLab delivery evidence with an independent incident source. Next, you can turn those traceable inputs into a publication-ready dashboard.

Publish and Validate the Measures

Your GitLab path plus the Incoming Webhook path now give Apache DevLake traceable evidence for all four native DORA measures. The incident relationship makes delivery stability visible.

A governance dashboard must also expose every source. It must show its failure behavior before publication.

You will add Deployment Rework Rate as the fifth measure. You will then release the validated evidence package.

In this step, get ready to:

Publish a five-measure Grafana dashboard with a source label on every panel.
Execute all three assertions with a visible incident-path negative control.
Publish the evidence package through GitHub before completing its Linear record.

Build and publish the five-measure dashboard

The dashboard specification is the publication contract for each panel. It prevents a visual edit from changing the agreed source or interpretation.

In your code editor, create a grafana folder inside the local project folder that contains governance and scripts.
Create dashboard-spec.md inside the grafana folder by copying this content:

# Federal Delivery Measures Dashboard Specification

| Panel title | Displayed source label | Publication treatment |
| --- | --- | --- |
| Deployment Frequency | Source: GitLab Deployments via DevLake `gitlab` | Publish the observed value without an unqualified tier. |
| Change Lead Time | Source: GitLab commits and merge requests via `gitextractor` and `gitlab`; first commit to production | Publish the median and clock boundary. |
| Change Failure Rate | Source: GitLab Deployments plus Incoming Webhook `INCIDENT` records | Publish only when the webhook-path assertion passes. |
| Failed Deployment Recovery Time | Source: GitLab deployment finish plus Incoming Webhook incident resolution | Use the 2023 name and definition. |
| Deployment Rework Rate | Source: GitLab `rework:` merge commit classification plus GitLab Deployments; custom SQL | Publish the classification rule beside the value. |

Remove before publication:

- The `Overall DORA Metrics` benchmark table.
- Color or text mappings that imply elite, high, medium, or low current performance.
- Any description that presents a report-year cluster as a fixed maturity tier.

Allowed historical context: A panel may cite a named DORA report year if it is clearly labelled historical context and is not used as the federal program's current rating.

What Does This Specification Control?

Each row fixes the source label that belongs beside one published measure.
The publication treatment records the boundary or dependency that readers need to interpret the value.
The removal list blocks benchmark cues from becoming unsupported current ratings.

Save grafana/dashboard-spec.md.
Preview grafana/dashboard-spec.md with your editor's Markdown preview.

You should see five panel rows followed by three publication-removal requirements. Every panel row names its source.

Does the Table Render Incorrectly?

Check that every table row starts with a pipe character. Check that the separator row remains directly below the header row.

Confirm the file is saved as dashboard-spec.md inside grafana.

Ask for help with the table structure: help me compare my dashboard specification with the required Markdown table

✔️ Awesome, I've got everything!

Your dashboard specification now fixes the five source labels plus the publication safeguards.

ⓧ I'd like to double check the full code

Compare your complete grafana/dashboard-spec.md file with this reference:

# Federal Delivery Measures Dashboard Specification

| Panel title | Displayed source label | Publication treatment |
| --- | --- | --- | --- |
| Deployment Frequency | Source: GitLab Deployments via DevLake `gitlab` | Publish the observed value without an unqualified tier. |
| Change Lead Time | Source: GitLab commits and merge requests via `gitextractor` and `gitlab`; first commit to production | Publish the median and clock boundary. |
| Change Failure Rate | Source: GitLab Deployments plus Incoming Webhook `INCIDENT` records | Publish only when the webhook-path assertion passes. |
| Failed Deployment Recovery Time | Source: GitLab deployment finish plus Incoming Webhook incident resolution | Use the 2023 name and definition. |
| Deployment Rework Rate | Source: GitLab `rework:` merge commit classification plus GitLab Deployments; custom SQL | Publish the classification rule beside the value. |

Remove before publication:

- The `Overall DORA Metrics` benchmark table.
- Color or text mappings that imply elite, high, medium, or low current performance.
- Any description that presents a report-year cluster as a fixed maturity tier.

Allowed historical context: A panel may cite a named DORA report year if it is clearly labelled historical context and is not used as the federal program's current rating.

Deployment Rework Rate needs a custom query because the dashboard does not ship with this fifth measure. The query joins deployment records to commit messages before calculating the percentage classified as rework.

Create deployment-rework-rate.sql inside the grafana folder by copying this query:

WITH production_deployments AS (
    SELECT
        cdc.cicd_deployment_id AS deployment_id,
        MAX(cdc.finished_date) AS deployment_finished_date,
        MAX(
            CASE
                WHEN LOWER(c.message) LIKE 'rework:%' THEN 1
                ELSE 0
            END
        ) AS is_rework
    FROM cicd_deployment_commits cdc
    JOIN commits c
        ON c.sha = cdc.commit_sha
    JOIN project_mapping pm
        ON cdc.cicd_scope_id = pm.row_id
        AND pm.`table` = 'cicd_scopes'
    WHERE
        pm.project_name IN (${project:sqlstring})
        AND cdc.result = 'SUCCESS'
        AND cdc.environment = 'PRODUCTION'
        AND $__timeFilter(cdc.finished_date)
    GROUP BY cdc.cicd_deployment_id
)
SELECT
    ROUND(
        100.0 * SUM(is_rework) / NULLIF(COUNT(*), 0),
        1
    ) AS deployment_rework_rate_percent
FROM production_deployments;

What Does This Query Do?

The production_deployments expression groups records by deployment_id so each deployment contributes once.
The is_rework classification becomes 1 when a joined commit message begins with rework:.
The filters retain successful production deployments from the selected project plus dashboard time range.
The final calculation divides classified rework deployments by all included deployments.

Save grafana/deployment-rework-rate.sql.
Confirm the query ends with the deployment_rework_rate_percent result column.

Your grafana folder should now contain the dashboard contract plus the custom query.

Does the Query File Look Incomplete?

Check that the file contains the complete production_deployments expression. Confirm the final statement reads from that expression.

Ask for a structural comparison: help me check my Deployment Rework Rate SQL against the required query

✔️ Awesome, I've got everything!

The custom query is saved beside the dashboard specification.

ⓧ I'd like to double check the full code

Compare your complete grafana/deployment-rework-rate.sql file with this reference:

WITH production_deployments AS (
    SELECT
        cdc.cicd_deployment_id AS deployment_id,
        MAX(cdc.finished_date) AS deployment_finished_date,
        MAX(
            CASE
                WHEN LOWER(c.message) LIKE 'rework:%' THEN 1
                ELSE 0
            END
        ) AS is_rework
    FROM cicd_deployment_commits cdc
    JOIN commits c
        ON c.sha = cdc.commit_sha
    JOIN project_mapping pm
        ON cdc.cicd_scope_id = pm.row_id
        AND pm.`table` = 'cicd_scopes'
    WHERE
        pm.project_name IN (${project:sqlstring})
        AND cdc.result = 'SUCCESS'
        AND cdc.environment = 'PRODUCTION'
        AND $__timeFilter(cdc.finished_date)
    GROUP BY cdc.cicd_deployment_id
)
SELECT
    ROUND(
        100.0 * SUM(is_rework) / NULLIF(COUNT(*), 0),
        1
    ) AS deployment_rework_rate_percent
FROM production_deployments;

The files define what to publish. The next actions apply them to the running dashboard backed by bundled MySQL.

Return to http://localhost:3002 in your browser.
Open the existing DORA dashboard.
Click Edit.
Open the Save drop-down list.
Select Save as copy.
Enter Federal Delivery Measures as the copied dashboard name.
Click Save.

Why Work From a Copy?

The copied dashboard preserves the shipped DORA view as a reference. Your publication edits remain isolated in a governed version.

Click Edit on the copied dashboard.
Click Add new element.
Select Panel.
Select Configure visualization.
Select the bundled MySQL source from the Data source drop-down menu.
Switch the query editor to Code mode.
Paste the complete contents of grafana/deployment-rework-rate.sql into the query editor.
Select the Stat visualization.

Before you run the query, what percentage do you expect from one rework deployment across four deployments? Hold that prediction in mind.

Click Run queries.

You should see 25.0 as the Deployment Rework Rate. This proves that the explicit commit classification reached the dashboard.

Enter Deployment Rework Rate as the panel title.
Copy the matching source label from grafana/dashboard-spec.md into the panel description.
Click Save in the panel editor.
Update the other four panel descriptions from their matching rows in grafana/dashboard-spec.md.
Remove the Overall DORA Metrics benchmark table.
Remove the elite, high, medium, and low current-performance mappings.
Click Save on the dashboard.

You should see five measures on the copied dashboard. Each panel names its source without assigning the program an unqualified current tier.

Is the Rework Panel Empty?

Confirm the panel uses the bundled MySQL data source. Confirm the selected project contains the four GitLab deployments.

Expand the dashboard time range so it includes every fixture deployment. Check that the SQL still contains the project variable plus the deployment time filter.

Ask for help with the panel: help me diagnose an empty Deployment Rework Rate panel

Execute the validation assertions

A displayed value becomes defensible when its inputs survive a planned challenge. A negative control deliberately removes a required input so you can observe whether the measure fails honestly.

Open governance/validation.md in your code editor.
Verify Assertion 1 against the four successful GitLab production deployments.
Verify each deployment SHA against the collected merge-request commit history.
Replace Observed: Pending execution. under Assertion 1 with the observed deployment count plus lineage result.
Verify Assertion 2 against the incident relationship plus the five dashboard panels.
Replace Observed: Pending execution. under Assertion 2 with the observed Change Failure Rate, recovery duration, and Deployment Rework Rate.
Add the inspected dashboard views plus relationship evidence to the matching Evidence: lines.

Assertions 1 and 2 should now record four deployments with commit lineage. They should also record a 25% Change Failure Rate, a non-empty recovery time, and a 25% Deployment Rework Rate.

Capture the current Change Failure Rate here: your baseline CFR.
Capture its UTC observation time here: your baseline CFR UTC timestamp.

Before you detach the incident path, do you expect the displayed rate to describe the fixture honestly? Hold that prediction in mind.

Return to the DevLake Config UI at http://localhost:4000.
Detach the Incoming Webhook connection from the measurement project.
Run the project's Collect Data action.
Return to the Federal Delivery Measures dashboard in Grafana.
Refresh the dashboard after collection finishes.

You should see Change Failure Rate fall below your baseline CFR. Delivery performance did not improve during this test.

Why Did the Rate Improve?

DevLake rebuilt the project-scoped incident relationships without the webhook input. The affected deployment lost its incident relationship.

The lower rate is the intended shortfall. It proves that missing incident evidence can create a falsely reassuring stability result.

Capture the lower Change Failure Rate here: your broken-path CFR.
Capture its UTC observation time here: your broken-path CFR UTC timestamp.
Return to the DevLake Config UI.
Reattach the Incoming Webhook connection to the measurement project.
Run the project's Collect Data action again.
Return to the Federal Delivery Measures dashboard.
Refresh the dashboard after the restored collection finishes.

You should see the original Change Failure Rate return. The restored value should match your baseline CFR.

Capture the restored Change Failure Rate here: your restored CFR.
Capture its UTC observation time here: your restored CFR UTC timestamp.
Replace Observed: Pending execution. under Assertion 3 with the baseline, broken, and restored values.
Add all three UTC timestamps to the Assertion 3 evidence.
Save governance/validation.md.
Preview governance/validation.md to confirm no assertion remains pending.

That is the critical evidence captured. Your validation record now shows the correct path, the broken path, and the restored path.

Did the Rate Stay Unchanged?

Confirm the Incoming Webhook was detached from the same DevLake project used by the dashboard. Confirm collection completed after the detachment.

Refresh Grafana only after the collection finishes. Check that the dashboard project filter still selects the measurement project.

Ask for help with the negative control: help me diagnose why detaching the webhook did not change CFR

Release the evidence and close the project

The release needs a plain-language entry point for reviewers. The README connects the contract, evidence paths, dashboard treatment, and fidelity limit.

Create README.md inside the local project folder that contains governance.
Add this release summary to README.md:

# Federal Delivery Measurement Evidence

This build fixes five measures before presenting them: Deployment Frequency, Change Lead Time, Change Failure Rate, Failed Deployment Recovery Time, and Deployment Rework Rate. GitLab Self-Managed is the system of record for commits, merge requests, and deployments. Apache DevLake ingests it through the `gitlab` and `gitextractor` plugins. Incidents arrive separately through an Incoming Webhook as `INCIDENT` records. Grafana presents each value with its source and without an unqualified performance tier.

The evidence package contains the contract, four decision records, topology, implementation plan, dashboard specification, custom rework query, fidelity limits, and three observed validation assertions. The deliberate negative control proves that removing incident ingestion makes Change Failure Rate look better even though delivery performance has not improved.

This mirror validates measurement mechanics. It does not reproduce the federal program's approval-gated deployment cadence, production authorization controls, scale, or incident adjudication process.

What Does the README Establish?

The first paragraph fixes the five published measures plus their systems of record.
The second paragraph identifies the released evidence plus the negative-control result.
The final paragraph limits the mirror's interpretation before a reviewer sees the values.

Save README.md.
Preview README.md to confirm all three paragraphs render below the title.
Review the existing governance/fidelity-note.md to confirm the approval-gated cadence remains explicit.

You should see a concise release summary followed by the mirror's interpretation limit.

Does the README Omit an Evidence Path?

Compare the second paragraph with the folders in your project. It should cover governance records, the dashboard specification, custom SQL, fidelity limits, and observed assertions.

Ask for a comparison: help me check my evidence README against the completed project artifacts

✔️ Awesome, I've got everything!

Your README now gives reviewers a plain-language map of the evidence package.

ⓧ I'd like to double check the full code

Compare your complete README.md file with this reference:

# Federal Delivery Measurement Evidence

This build fixes five measures before presenting them: Deployment Frequency, Change Lead Time, Change Failure Rate, Failed Deployment Recovery Time, and Deployment Rework Rate. GitLab Self-Managed is the system of record for commits, merge requests, and deployments. Apache DevLake ingests it through the `gitlab` and `gitextractor` plugins. Incidents arrive separately through an Incoming Webhook as `INCIDENT` records. Grafana presents each value with its source and without an unqualified performance tier.

The evidence package contains the contract, four decision records, topology, implementation plan, dashboard specification, custom rework query, fidelity limits, and three observed validation assertions. The deliberate negative control proves that removing incident ingestion makes Change Failure Rate look better even though delivery performance has not improved.

This mirror validates measurement mechanics. It does not reproduce the federal program's approval-gated deployment cadence, production authorization controls, scale, or incident adjudication process.

The files are ready for controlled release. Keep the repository private because the package represents a federal-program governance mirror.

In GitHub, click New repository.
Choose the appropriate organization or account under Owner.
Enter your evidence repository name as the repository name.
Select Private as the repository visibility.
Click Create repository.
Add the private GitHub repository as a second remote in your Git client.
Commit governance, grafana, scripts, README.md, and service-state.txt as one evidence package.
Push the evidence commit to the private GitHub repository.

You should see the governance pack, scripts, dashboard files, README, and fixture state in the private repository. The released evidence now has one reviewable home.

Did the GitHub Push Fail?

Confirm the GitHub remote points to the new private repository. Check that your authenticated account has write access to its selected owner.

Ask for help first: help me diagnose my private GitHub repository push Contact the NextWork community if your organization applies a custom authentication policy.

Open Releases in the private GitHub repository.
Click Draft a new release.
Use Choose a tag to select or create the release tag.
Enter a concise evidence version under Release title.
Summarize the five measures plus the negative control under Describe this release.
Click Publish release.
Return to the Linear project that tracks this measurement build.
Move the project to a status in the Completed category.

Before the final check, which three surfaces should prove that the measures are published, released, and closed? Hold that answer in mind.

Return to http://localhost:3002 to confirm the five source-labelled measures remain visible.
Open the private repository's Releases view to confirm the release is published.
Return to the Linear project to confirm its status belongs to the Completed category.

Strong finish. You now have a five-measure dashboard, observed failure evidence, a private release, and a completed governance record.

Secret mission

Red-Team One Measurement Decision

Put one published measurement decision in front of an independent reviewer. They will challenge its alternative, tradeoff, and reversal trigger before you preserve the verdict in an auditable review log.

Clean Up Your Resources

Choose whether to leave the local measurement environment running, pause it for later, or remove its runtime completely. This project introduces no metered cloud resources, so existing organizational hosting remains the only cost consideration.

Resources you used:

Local Docker Compose containers and network for Apache DevLake 1.0.2, bundled MySQL 8, and Grafana 11.6.0.
Docker volumes containing the DevLake project configuration, native GitLab ingestion settings, Incoming Webhook connection, collected evidence, and source-labelled dashboard.
Local Compose folder containing docker-compose.yml and .env with ENCRYPTION_SECRET.

Preserve the audit record

The private GitLab mirror, private GitHub release, and Completed Linear project are evidence records outside the local stack.

Their deletion requires records-retention approval, so the cleanup options below preserve them.

Keep everything running

Leave the environment as it is if you expect to inspect the dashboard or repeat the validation assertions soon.

Keep the local DevLake stack running so Config UI remains available at http://localhost:4000.
Keep the source-labelled dashboard available at http://localhost:3002.
Preserve the private GitLab mirror so its four production deployments and incident-linked history remain available.
Preserve the private GitHub release and Completed Linear project as the released close-out record.

Pause - I'll come back to this later

Pause the local services to free workstation resources while keeping the containers, volumes, configuration, and collected data available for a later restart.

Return to Windows PowerShell in the folder that contains docker-compose.yml.
Stop the local measurement stack by running this command:

docker compose stop

What Does This Command Do?

The command stops the Compose services without removing their containers. Their volumes retain the DevLake configuration, collected records, and Grafana dashboard.

Refresh http://localhost:4000 to confirm that Config UI no longer loads.
Refresh http://localhost:3002 to confirm that Grafana no longer loads.

Stack Still Running?

Confirm that PowerShell is in the folder containing docker-compose.yml. A different folder prevents Compose from locating this stack.

Share the folder context and command output when you ask for help diagnosing the pause command.

Your local measurement services are now paused, while the evidence needed to resume the review remains intact.

Delete - I don't want to use this again

This option permanently removes the local DevLake containers, networks, volumes, configuration, collected records, and customized dashboard. Your released GitLab, GitHub, and Linear evidence remains available outside the local stack.

Confirm Retention Before Deletion

Deleting the volumes is intentionally irreversible. The private GitHub release protects the published evidence package, but confirm records-retention approval before removing any local copy.

Confirm that the private GitHub release contains the final governance pack and observed validation evidence.
Confirm that records-retention approval covers removal of the local measurement environment.
Return to PowerShell in the folder that contains docker-compose.yml.
Remove the local stack and its volumes by running this command:

docker compose down --volumes

What Does This Command Remove?

The command removes the Compose containers and networks. The --volumes flag also removes declared or attached non-external volumes that hold the local DevLake data.

Refresh http://localhost:4000 to confirm that Config UI no longer loads.
Refresh http://localhost:3002 to confirm that Grafana no longer loads.

Cleanup Command Failed?

Confirm that Docker Desktop is running. Confirm that PowerShell is using the folder containing the same docker-compose.yml file that launched the stack.

Share the command output when you ask for help diagnosing the cleanup command.

Delete local Compose files

The stack cleanup leaves its downloaded files on disk. The remaining .env file contains the generated ENCRYPTION_SECRET.

Return to the folder containing docker-compose.yml in File Explorer.
Delete that folder after confirming records-retention approval.
Confirm that the folder no longer appears in its parent location.

The local runtime and its stored measurement data are now removed. Your released governance evidence remains preserved in the approved external records.

Nice Work!

You did it! You built a governed delivery-measurement environment that traces five measures from GitLab Self-Managed plus Incoming Webhook evidence into a source-labelled Grafana dashboard.

You've learned how to:

Govern a five-measure DORA contract with fixed calculations. You also documented source boundaries through decision records with explicit reversal triggers.
Wire real delivery evidence through Apache DevLake using GitLab's native gitlab plus gitextractor paths. You paired those records with webhook incidents on a source-labelled five-measure dashboard.
Prove measurement fidelity with a visible negative control that made Change Failure Rate falsely improve when incident ingestion disappeared. You released the observed evidence in a private GitHub repository. You then moved the Linear project to a Completed-category status.
Secret Mission: Invited an independent reviewer to challenge one measurement decision using the released evidence. You recorded the verdict in governance/review-log.md. The same review log shows whether the reversal trigger fired.

Ready to quiz yourself?