Data Processing Pipeline with Shell

Build a shell script that cleans CSV data and generates reports using awk, sed, and vi.

Introduction

30 Second Summary

Every organization has spreadsheets and data files full of work orders, customer records, or transaction logs. Someone has to turn that raw data into something useful before any decisions can be made.

In this project, you will build a shell script that acts as a data processing pipeline. You feed it a messy CSV file of utility work order data and it outputs a clean, formatted summary report with statistics.

What You'll Build

Picture running a single command in your terminal and watching a formatted report appear with department summaries, average completion times, and status breakdowns, all generated from raw data that was riddled with errors just moments before.

By the end of this project, you'll have:

  • A working shell script you run with a single command that produces a formatted report showing work orders by department, average completion times, and status breakdowns.
  • A complete data cleaning pipeline where awk extracts and calculates across columns while sed fixes messy capitalization, empty fields, and trailing whitespace.
  • Hands-on experience building and editing your script in vi, the exact editor named in data analyst job listings.
  • Secret Mission: Add command-line arguments so your script can filter reports by department or status on the fly.

Do I need any prior experience?

No prerequisites are required. This project includes full setup instructions for all tools. You just need a Windows 11 machine. You will install WSL (Windows Subsystem for Linux), which gives you a full Ubuntu Linux terminal directly on your Windows PC for free.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for the job you're targeting.

Set Up WSL and Verify Your Tools

To use the shell tools named in the job listing, you need a UNIX environment. Windows 11 includes a feature called WSL (Windows Subsystem for Linux) that gives you a full Linux terminal without leaving your machine.

In this step, you will install WSL with Ubuntu, confirm that awk, sed, and vi are ready to use, and create a project directory to work in.

In this step, get ready to:
  • Install WSL with Ubuntu on Windows 11.
  • Verify that awk, sed, and vi are available in your terminal.
  • Create a project directory for your data pipeline.
Install WSL with Ubuntu

WSL lets you run a full Linux operating system inside Windows. You need it because awk, sed, and vi are UNIX tools that run natively in Linux.

  • Open PowerShell as Administrator by pressing the Windows key, typing PowerShell, then selecting Run as administrator.
  • Check if WSL is already installed by running this command:
wsl --list

✔️ I see Ubuntu in the list

WSL and Ubuntu are already set up on your machine. Skip ahead to the next substep.

  • Press the Windows key, type Ubuntu, and press Enter to open the Ubuntu terminal.

ⓧ I see an error or no distributions installed

WSL is not set up yet. You will install it now.

  • Install WSL and Ubuntu by running this command in the same Administrator PowerShell window:
wsl --install

What does this command do?

This single command enables WSL, downloads the Linux kernel, and installs Ubuntu as your default Linux distribution. It handles everything in one step.

Your machine will need to restart

After the install finishes, Windows will ask you to restart. Save any open work before rebooting.

After reboot, an Ubuntu terminal window opens automatically. It will ask you to create a UNIX username and password. Your password will not appear as you type it. This is normal Linux security behavior.

  • Restart your machine when prompted.
  • After reboot, wait for the Ubuntu window to appear.
  • Enter a username when prompted (all lowercase, no spaces).
  • Enter a password when prompted. No characters appear on screen as you type. This is expected.
  • Re-enter the password to confirm it.

Ubuntu window did not appear after reboot?

Press the Windows key, type Ubuntu, and press Enter to open it manually. If it still does not launch, confirm that virtualization is enabled in your BIOS settings.

Help me troubleshoot WSL installation on Windows 11

Verify your tools are available

Ubuntu ships with awk (mawk), sed, and vi (vim-tiny) pre-installed. You will now confirm each one is working.

Before you run the first check, what do you expect to see if awk is properly installed?

  • In your Ubuntu terminal, check the awk version by running:
awk --version

You should see output containing mawk and a version number. This confirms awk is installed and ready.

  • Check the sed version by running:
sed --version

You should see output starting with sed (GNU sed) followed by a version number.

  • Check the vi version by running:
vi --version

You should see output mentioning VIM and Vi IMproved. All three tools are confirmed available.

What are these three tools?

These are the exact UNIX tools named in the PG&E job listing. awk extracts and calculates across columns of data. sed transforms text line by line. vi is a terminal-based text editor.

You will use all three together to build an automated data pipeline.

Seeing an error with the awk command?

If you see awk: not an option: --version, try running awk -W version instead. Some older Ubuntu versions use a mawk build that requires this alternate flag.

If you see command not found for any tool, make sure you are running commands inside the Ubuntu terminal, not in PowerShell or Command Prompt.

Help me fix version check errors in WSL Ubuntu

Create your project directory

You need a dedicated folder for the files you will create throughout this project.

  • Create the project directory and move into it by running this command:
mkdir ~/data-pipeline && cd ~/data-pipeline
  • Confirm you are inside the new directory by running:
pwd

You should see /home/your-username/data-pipeline printed in the terminal. The exact path depends on the username you chose during Ubuntu setup.

Seeing a different path?

If pwd shows your home directory without data-pipeline at the end, the cd part may not have run. Try running cd ~/data-pipeline on its own, then pwd again.

Help me navigate to my project directory in WSL

Your UNIX environment is ready with all three tools confirmed. Next up, you will create sample data and start exploring it with shell commands.

Create Sample Data and Explore It

Your UNIX environment is set up and all three tools are confirmed working. Now you need something to actually process.

Before building a pipeline, you need data to work with and a sense of what it looks like raw. In this step, you'll create a sample CSV file with realistic data quality problems, then use shell commands to poke around it.

In this step, get ready to:
  • Create a CSV file simulating utility work order data with deliberate quality issues.
  • Explore the file using basic shell commands like cat, head, wc, and grep.
  • Use an awk pipe to count data by column and observe messy output.
Create the sample data file

Real data pipelines process messy data. Your sample file will simulate utility work order records with intentional problems: inconsistent capitalization, empty fields, and extra whitespace.

  • Create the workorders.csv file by running this command in your Ubuntu terminal:
cat > workorders.csv << 'EOF'
WorkOrderID,Department,Status,DaysToComplete,Priority,DateOpened
WO-001,Gas Operations,Completed,3,High,2024-01-15
WO-002,Electric Distribution,Open,,Medium,2024-01-16
WO-003,Gas Operations,completed,7,Low,2024-01-17
WO-004,Customer Service,COMPLETED,2,High,2024-01-18
WO-005,Electric Distribution,Closed,5,Medium,2024-01-19
WO-006,Gas Operations,Open,, High,2024-01-20
WO-007,Customer Service,completed,4,Medium,2024-01-21
WO-008,Electric Distribution,COMPLETED,6,Low,2024-01-22
WO-009,Gas Operations,Closed,1,High,2024-01-23
WO-010,Customer Service,Open,,Low,2024-01-24
WO-011,Electric Distribution,completed,3,High,2024-01-25
WO-012,Gas Operations,Completed,8,Medium,2024-01-26
WO-013,Customer Service,CLOSED,2,High,2024-01-27
WO-014,Electric Distribution,Open,,Medium,2024-01-28
WO-015,Gas Operations,completed,5,Low,2024-01-29
EOF

What does this command do?

This uses a heredoc (<< 'EOF') to write multiple lines into a file at once.

Everything between << 'EOF' and the closing EOF gets written into workorders.csv. The single quotes around EOF prevent the shell from interpreting any special characters in the data.

  • Confirm the file was created by running:
head -5 workorders.csv

You should see the header row plus the first four data rows printed in your terminal.

Seeing "No such file or directory"?

  • Make sure you're in the ~/data-pipeline directory. Run pwd to check your current location.
  • If you pasted the heredoc command and it didn't return to a prompt, you may have missed the closing EOF line. Press Ctrl+C and try again.

Help me troubleshoot creating workorders.csv

Explore the data with basic shell commands

Before processing data, you want to understand its shape. Shell commands let you quickly inspect file contents without opening an editor.

  • Count the total number of lines in the file by running:
wc -l workorders.csv

You should see 16 (15 data rows plus 1 header row).

  • Search for all rows mentioning "closed" (regardless of capitalization) by running:
grep -i "closed" workorders.csv

What does the -i flag do?

The -i flag makes grep case-insensitive. It matches "Closed", "closed", and "CLOSED" all at once. Notice that your results show different capitalizations of the same status. That's one of the data quality issues you'll fix later.

Count by department with awk

awk is built for column-based data. You can extract a single column and pipe it to other commands to get a quick count.

Before you run this next command, what do you expect the output to look like? Think about whether awk knows to skip the header row.

  • Count how many work orders each department has by running:
awk -F',' '{print $2}' workorders.csv | sort | uniq -c | sort -rn

What does this pipeline do?

  • -F',' tells awk that fields are separated by commas.
  • {print $2} extracts the second column (Department) from every row.
  • sort groups identical lines together. uniq -c counts them. sort -rn orders by count (highest first).

The pipe character (|) sends the output of one command as input to the next. This chaining is the foundation of shell scripting data pipelines.

Look at the output. You'll see "Department" listed as an entry because awk processed the header row too. The department counts themselves look OK here, but other columns have bigger problems.

Where are the data quality issues?

The Status column has "Completed", "completed", and "COMPLETED" treated as different values. Some DaysToComplete fields are empty. These issues will produce wrong results when you try to calculate averages or count by status in the next step.

✔️ Awesome, I've got everything!

Great. Make sure your terminal is still in the ~/data-pipeline directory for the next step.

ⓧ I'd like to double check the full code

Your workorders.csv file should contain this exact content:

  • View your full file by running:
cat workorders.csv
WorkOrderID,Department,Status,DaysToComplete,Priority,DateOpened
WO-001,Gas Operations,Completed,3,High,2024-01-15
WO-002,Electric Distribution,Open,,Medium,2024-01-16
WO-003,Gas Operations,completed,7,Low,2024-01-17
WO-004,Customer Service,COMPLETED,2,High,2024-01-18
WO-005,Electric Distribution,Closed,5,Medium,2024-01-19
WO-006,Gas Operations,Open,, High,2024-01-20
WO-007,Customer Service,completed,4,Medium,2024-01-21
WO-008,Electric Distribution,COMPLETED,6,Low,2024-01-22
WO-009,Gas Operations,Closed,1,High,2024-01-23
WO-010,Customer Service,Open,,Low,2024-01-24
WO-011,Electric Distribution,completed,3,High,2024-01-25
WO-012,Gas Operations,Completed,8,Medium,2024-01-26
WO-013,Customer Service,CLOSED,2,High,2024-01-27
WO-014,Electric Distribution,Open,,Medium,2024-01-28
WO-015,Gas Operations,completed,5,Low,2024-01-29

Compare your output against the content above. Look for the empty fields (two commas in a row) and the inconsistent capitalization in the Status column.

You've created your dataset and seen it raw. Next, you'll use awk to do real analysis on this data. But the messy quality issues will trip up your calculations.

Extract and Calculate with awk

In the last step, you created your work order data and got a first look at it with basic shell commands. Now it is time to analyze that data.

awk is the standard UNIX tool for column-based data processing. It reads structured files like CSVs column by column, letting you extract, filter, and calculate in a single pass.

In this step, get ready to:
  • Extract and format specific columns from CSV data using awk.
  • Filter rows based on conditions like priority level and completion time.
  • Calculate averages and group counts to reveal data quality problems.
Print specific columns with awk

Your CSV has six columns, but you rarely need all of them at once. awk lets you select exactly the columns you want by their position number.

  • Print just the work order ID, department, and status columns by running this command:
awk -F',' 'NR>1 {print $1, $2, $3}' workorders.csv

What does this command do?

  • -F',' tells awk to split each line at commas, so it knows where one column ends and the next begins.
  • NR>1 means "only process lines where the record number is greater than 1." This skips the header row.
  • $1, $2, $3 reference columns by position. Column 1 is WorkOrderID, column 2 is Department, column 3 is Status.

You should see 15 lines of output, each showing an ID, a department name, and a status value.

Seeing a file not found error?

Make sure you are in the ~/data-pipeline directory. Run pwd to check your current location. If you are somewhere else, run cd ~/data-pipeline to get back.

Also check that you typed the filename exactly as workorders.csv with no capital letters or extra characters.

Help me troubleshoot my awk command

Filter rows by condition

Beyond selecting columns, awk can filter rows based on conditions. This is like a WHERE clause in SQL, written directly in the terminal.

  • Show only high-priority work orders by running:
awk -F',' '$5 == "High"' workorders.csv

You should see a handful of rows where the Priority column (column 5) is "High".

  • Now find work orders that took more than 5 days to complete by running:
awk -F',' 'NR>1 && $4 > 5 {print $1, $2, "took", $4, "days"}' workorders.csv

What does this command do?

NR>1 && $4 > 5 combines two conditions with AND. The row must be past the header AND have a DaysToComplete value greater than 5.

The {print $1, $2, "took", $4, "days"} part builds a readable sentence from the data, mixing column values with literal words.

You should see a few rows showing which work orders took the longest.

Calculate averages and count by status

awk can also accumulate values across all rows and produce a summary at the end. This is where things get interesting.

Before you run the next command, think about this: your CSV has 15 data rows, but 4 of them have empty DaysToComplete fields. What do you think will happen when awk tries to average those values?

  • Calculate the average days to complete across all work orders by running:
awk -F',' 'NR>1 {sum+=$4; count++} END {print "Average days to complete:", sum/count}' workorders.csv

What does this command do?

sum+=$4 adds column 4 (DaysToComplete) to a running total for every row. count++ increments a counter for every row processed.

The END block runs once after all rows are processed, printing the final calculation.

The result looks lower than you would expect. When a DaysToComplete field is empty, awk treats it as 0 for arithmetic. Those four empty rows drag the average down because count still includes all 15 rows.

  • Now count work orders grouped by status to see another data quality problem:
awk -F',' 'NR>1 {status[$3]++} END {for (s in status) print s, status[s]}' workorders.csv

What does this command do?

status[$3]++ uses an associative array. Each unique value in column 3 becomes a key, and its count increments each time that value appears.

The END block loops through all keys and prints each status with its count.

Look at the output carefully. You should see "Completed", "completed", and "COMPLETED" listed as separate categories. Same for "Closed" and "CLOSED". These are all the same status, but inconsistent capitalization makes awk treat them as different values.

Why are the results wrong?

You have discovered two data quality issues hiding in your CSV. Empty fields are counted as zero in calculations, skewing averages downward. Inconsistent capitalization splits what should be one category into three.

This is exactly why data cleaning is a separate step in any data pipeline. Raw data needs to be standardized before analysis tools like awk can produce accurate results.

Your awk commands work perfectly. The problem is the data itself. Next up, you will use sed to clean the messy data so your analysis produces accurate results.

Clean the Data with sed

In the previous step, your awk commands produced incorrect results. The average completion time was skewed by empty fields counting as zero, and status categories like "Completed", "completed", and "COMPLETED" showed up as separate entries.

The data itself is the problem. sed (stream editor) fixes these issues by transforming text line-by-line. In this step, you will clean the raw CSV so that awk can produce accurate analysis.

In this step, get ready to:
  • Use sed substitution to normalize inconsistent text.
  • Chain multiple sed operations into a data-cleaning pipeline.
  • Verify your cleaned data produces correct awk results.
Fix inconsistent capitalization with sed

The core problem is that "Completed", "completed", and "COMPLETED" all mean the same thing, but awk treats them as different text values. sed can find a pattern and replace it with a normalized version.

  • Try normalizing the word "Completed" in all its forms by running this command:
sed 's/,[Cc][Oo][Mm][Pp][Ll][Ee][Tt][Ee][Dd],/,completed,/g' workorders.csv

What does this command do?

The s/pattern/replacement/g syntax is sed's substitution command. It finds every match of the pattern and replaces it with the replacement text.

  • Each [Cc] means "match either uppercase C or lowercase c". This covers every capitalization variant without needing a case-insensitive flag.
  • The commas around the pattern ,completed, ensure you only match within the Status column boundaries.
  • The g at the end means "global" (replace all matches on the line, not just the first).

You should see all 15 data rows printed, with every variation of "Completed" now showing as completed in the Status column.

That handles one word, but the data also has empty fields and trailing whitespace. Let's see how sed handles those.

  • Try trimming trailing whitespace and replacing empty fields by running:
sed -e 's/,[[:space:]]*$/,/' -e 's/,,/,NA,/g' workorders.csv

What do these expressions do?

  • The -e flag lets you chain multiple substitution commands in one sed call.
  • The first expression strips trailing whitespace at the end of each line.
  • The second expression s/,,/,NA,/g finds two consecutive commas (an empty field) and inserts NA between them.

Look at the output. Rows that previously had empty DaysToComplete fields (like WO-002) now show NA in that column.

Output looks the same as before?

These commands print the transformed output to the terminal. They do not modify the original file. If you do not see any change, double-check you typed ,, (two commas) in the second pattern.

Help me troubleshoot my sed output

Chain all fixes into one cleaning pipeline

Now you know how each fix works individually. Time to combine them all into a single pipeline that produces a fully cleaned file.

  • Run this command to clean the data and save the result to a new file:
sed -e 's/[[:space:]]*$//' \
    -e 's/,,/,NA,/g' \
    -e 's/[Cc][Oo][Mm][Pp][Ll][Ee][Tt][Ee][Dd]/completed/g' \
    -e 's/[Oo][Pp][Ee][Nn]/open/g' \
    -e 's/[Cc][Ll][Oo][Ss][Ee][Dd]/closed/g' \
    workorders.csv > workorders_clean.csv

What does this pipeline do?

Five expressions run in order on every line of the file:

  • Strip trailing whitespace from each line.
  • Replace empty fields (,,) with NA.
  • Normalize all variations of "Completed" to completed.
  • Normalize all variations of "Open" to open.
  • Normalize all variations of "Closed" to closed.

The > workorders_clean.csv redirects the output into a new file instead of printing to the terminal. Your original workorders.csv stays untouched.

  • Verify the cleaned file exists and looks correct by running:
head -5 workorders_clean.csv

You should see the header row plus 4 data rows. Status values are now all lowercase, and previously empty fields show NA.

Seeing a "No such file or directory" error?

This means the sed command did not create the file. Make sure you included > workorders_clean.csv at the end of the command. Without the > redirect, sed only prints to the terminal.

Help me fix my sed pipeline

Confirm awk now produces correct results

The whole point of cleaning was to fix the incorrect analysis from the previous step. Time to rerun those awk commands on the cleaned file to prove it works.

  • Run the status count on the cleaned data:
awk -F',' 'NR>1 {status[$3]++} END {for (s in status) print s, status[s]}' workorders_clean.csv

Before, this command showed duplicates like "Completed", "completed", and "COMPLETED" as separate categories. Now you should see only three consolidated statuses: completed, open, and closed.

  • Before running the next command, predict: will the average completion time be higher or lower now that empty fields are excluded?
  • Run the corrected average calculation:
awk -F',' 'NR>1 && $4 != "NA" && $4 != "" {sum+=$4; count++} END {print "Average days to complete:", sum/count}' workorders_clean.csv

Why is the average different now?

The condition $4 != "NA" skips rows where DaysToComplete was empty (now marked as NA). Previously, those empty fields were treated as 0, dragging the average down.

With them excluded, the average reflects only work orders that actually have completion data.

You should see an average around 4.4 days. The status breakdown should show three clean categories with accurate counts.

Still seeing duplicate status categories?

Make sure you are running awk against workorders_clean.csv (not the original workorders.csv). Check the filename at the end of your command.

Help me troubleshoot my cleaned data

Your data is now clean and your analysis is accurate. Next up, you will combine the entire cleaning and analysis workflow into a single reusable shell script using vi.

Automate Everything into a Shell Script with vi

Your cleaning and analysis commands work perfectly now. But running them one at a time is manual, error-prone, and impossible to hand off to a colleague.

A shell script captures the entire pipeline in a single file. You run one command and get a full report. In this step, you will write that script in vi, the editor specifically named in the PG&E job listing.

In this step, get ready to:
  • Write a shell script using vi's modal editing workflow.
  • Combine the sed cleaning and awk analysis into one automated pipeline.
  • Run the finished script and see a formatted summary report in one command.
Open vi and set up the script header

Vi is a modal editor. It starts in command mode, where keystrokes move the cursor or trigger actions. You press i to enter insert mode and type text. You press Esc to return to command mode. You save with :w.

  • Open vi to create a new file by running this command:
vi report.sh
  • Press i to enter insert mode. You will see -- INSERT -- appear at the bottom of the screen.
  • Type the following script header and variable definitions:
#!/bin/bash
# Work Order Data Processing Pipeline
# Cleans raw CSV data and generates a summary report

INPUT_FILE="workorders.csv"
CLEAN_FILE="/tmp/workorders_clean.csv"
  • Press Esc to return to command mode.
  • Type :w and press Enter to save.

Vi confirms the save by showing the filename and byte count at the bottom of the screen.

What does this code do?

The #!/bin/bash line (called a shebang) tells the system to run this file with Bash.

The two variables store file paths. INPUT_FILE points to your raw CSV. CLEAN_FILE points to a temporary location in /tmp/ so you never overwrite your original data.

Using variables means if you rename the CSV later, you only change one line instead of hunting through the whole script.

Why vi instead of nano?

The PG&E job listing names vi as a required skill. Interviewers may ask about modal editing or expect you to edit a file on a server that only has vi installed. Learning the basic workflow now gives you a real answer for that conversation.

Add the data cleaning section

The sed pipeline you built in the previous step becomes the first action inside your script. It reads the raw file and writes a cleaned version to the temp path.

  • In vi, press i to re-enter insert mode.
  • Position your cursor after the last line and add a blank line, then type the sed cleaning section:
# --- Data Cleaning with sed ---
sed -e 's/[[:space:]]*$//' \
    -e 's/,,/,NA,/g' \
    -e 's/[Cc][Oo][Mm][Pp][Ll][Ee][Tt][Ee][Dd]/completed/g' \
    -e 's/[Oo][Pp][Ee][Nn]/open/g' \
    -e 's/[Cc][Ll][Oo][Ss][Ee][Dd]/closed/g' \
    "$INPUT_FILE" > "$CLEAN_FILE"
  • Press Esc to return to command mode.
  • Type :wq and press Enter to save and quit vi.

What does this code do?

This is the same sed pipeline from the previous step, but now it reads from the $INPUT_FILE variable and writes to $CLEAN_FILE. The backslashes (\) at the end of each line tell Bash the command continues on the next line.

  • Verify the script runs so far by executing:
bash report.sh
  • Confirm the cleaning worked by running:
head -3 /tmp/workorders_clean.csv

You should see the CSV header followed by two data rows with normalized lowercase status values and no trailing whitespace.

Seeing a syntax error?

Check that every line inside the sed command (except the last) ends with a space followed by a backslash. A missing backslash or an extra space after the backslash breaks the line continuation.

Also confirm the quotes around "$INPUT_FILE" and "$CLEAN_FILE" are straight double quotes, not curly quotes.

Help me debug my sed command in report.sh

Add the report output section and run the script

The cleaned data is ready. Now you will add the formatted report that uses awk with printf for aligned columns. This is the payoff of the whole pipeline.

  • Re-open the script in vi:
vi report.sh
  • Press G (capital G) to jump to the last line of the file.
  • Press o to open a new line below and enter insert mode.
  • Type the following report output section:
# --- Report Output ---
echo "====================================="
echo "   WORK ORDER SUMMARY REPORT"
echo "====================================="
echo ""

# Department summary
echo "--- Work Orders by Department ---"
awk -F',' 'NR>1 {dept[$2]++} END {for (d in dept) printf "  %-25s %d\n", d, dept[d]}' "$CLEAN_FILE"
echo ""

# Status breakdown
echo "--- Status Breakdown ---"
awk -F',' 'NR>1 {status[$3]++} END {for (s in status) printf "  %-15s %d\n", s, status[s]}' "$CLEAN_FILE"
echo ""

# Average completion time (excluding NA values)
echo "--- Completion Time ---"
awk -F',' 'NR>1 && $4 != "NA" && $4 != "" {sum+=$4; count++} END {printf "  Average days to complete: %.1f\n", sum/count}' "$CLEAN_FILE"
echo ""

What does printf do here?

The printf function in awk formats output into aligned columns. %-25s means "left-align a string in a 25-character-wide field." This keeps department names and counts neatly lined up regardless of name length.

  • Still in insert mode, continue typing the final section below the code you just entered:
# High priority open items
echo "--- High Priority Open Items ---"
awk -F',' 'NR>1 && $3 == "open" && $5 == "High" {printf "  %s  %-25s  Opened: %s\n", $1, $2, $6}' "$CLEAN_FILE"
echo ""
echo "====================================="
echo "  Report generated: $(date)"
echo "====================================="

# Cleanup
rm -f "$CLEAN_FILE"

What does the cleanup section do?

The rm -f "$CLEAN_FILE" line deletes the temporary cleaned file after the report finishes. The -f flag prevents an error if the file does not exist. This keeps your system tidy.

  • Press Esc to return to command mode.
  • Type :wq and press Enter to save and quit vi.
  • Make the script executable by running:
chmod +x report.sh

Before you run the script, what do you think it will print? Take a moment to predict the output structure.

  • Run your finished pipeline by executing:
./report.sh

You should see a formatted summary report with a header, department counts, status breakdown, average completion time, and a list of high-priority open items.

Permission denied or command not found?

If you see Permission denied, confirm you ran chmod +x report.sh before executing it. The +x flag grants execute permission.

If you see No such file or directory, make sure you are in the ~/data-pipeline directory. Run pwd to check.

Help me fix my report.sh script

✔️ Awesome, I've got everything!

Great. Double check you have saved report.sh and that it runs cleanly with no errors.

ⓧ I'd like to double check the full code

Your report.sh file has three sections you built above. Scroll up and confirm each one is present in this order:

  • The script header: shebang, comments, and INPUT_FILE / CLEAN_FILE variables.
  • The sed cleaning section with all five substitution expressions piping into "$CLEAN_FILE".
  • The report output section: echo headers, four awk commands with printf, the date footer, and the rm -f cleanup.

If anything is missing, re-open the file with vi report.sh and add the missing lines.

You just automated a complete data pipeline into a single reusable command. Next up in the Secret Mission, you will add command-line arguments so your script can filter reports by department or status without editing the code.

Secret mission

Add Command-Line Arguments

Your report.sh works great for a full summary, but what if an analyst only needs to see Gas Operations data? Or only open work orders? Right now they would have to edit the script every time. In this secret mission, you will add command-line arguments so your script can filter reports dynamically without any code changes.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally inside WSL, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • WSL with Ubuntu installed on Windows 11.
  • Project directory ~/data-pipeline containing workorders.csv and report.sh.

Keep everything running

No action needed. Choose this if you are still actively building or want to reuse your pipeline later.

  • Your WSL Ubuntu installation and project files persist between reboots.
  • You can access them anytime by opening Ubuntu from the Start menu or typing wsl in PowerShell.
  • Your ~/data-pipeline folder and all scripts remain in place.

Pause - I'll come back to this later

WSL suspends automatically when you close the Ubuntu terminal. No manual action is needed to pause.

  • Close the Ubuntu terminal window. WSL shuts down the Linux kernel after a few seconds of inactivity.
  • Your files and scripts remain exactly as you left them. Reopen Ubuntu from the Start menu when you are ready to continue.

Delete - I don't want to use this again

Remove all project resources and the WSL installation entirely. This frees disk space and returns your system to its pre-project state.

  • Open PowerShell as Administrator.
  • Unregister the Ubuntu distribution by running this command:
wsl --unregister Ubuntu

What does this command do?

This permanently deletes all data, settings, and files inside your Ubuntu installation, including your ~/data-pipeline folder. After running it, Ubuntu will no longer appear in your WSL distribution list.

  • Remove the WSL platform itself by running:
wsl --uninstall
  • Restart your machine to complete the removal.

Nice Work!

Nice Work!

You just built a complete data processing pipeline using the exact UNIX tools that professional data analyst roles require. One command now transforms messy CSV data into a clean, formatted summary report.

Here's what you learned:

  • Used awk to extract columns, filter rows by condition, and calculate statistics from structured CSV data in a single pass.
  • Used sed to clean messy data by normalizing capitalization, filling empty fields, and trimming whitespace in a pipeline.
  • Wrote a complete shell script in vi that automates the entire data processing pipeline into a single reusable command.
  • Secret Mission: Added command-line arguments so your script can filter reports dynamically by department or status.

Ready to quiz yourself?