Migrate Legacy Data to Azure Cloud

Migrate CSV data to Azure Data Lake and generate an AI-powered cost analysis memo.

Introduction

30 Second Summary

Every organisation eventually faces the same challenge: their data is stuck in old systems and needs to move to the cloud. The tricky part is not just the migration itself. It is convincing stakeholders by showing them exactly what it will cost.

In this project, you will build a cloud migration pipeline that moves legacy CSV data into Azure Data Lake Storage Gen2, transforms it using PySpark in Azure Databricks, and then uses an LLM to automatically generate a professional cost analysis memo for stakeholders.

What You'll Build

An end-to-end data migration pipeline that takes legacy flat-file data, loads it into Azure's modern data lakehouse architecture, transforms it with Delta Lake, and leverages the Claude API to produce a stakeholder-ready cost estimate document.

By the end of this project, you'll have:

  • An Azure resource group containing a Data Lake Storage Gen2 account with your migrated legacy data.
  • A Databricks PySpark notebook that reads raw CSV and writes optimized Delta Lake tables.
  • A Python script that uses the Claude API to generate a professional migration cost estimate from your actual Azure resource configuration.
  • Secret Mission: Extend your script to produce a multi-phase migration roadmap with per-phase cost projections and timeline estimates.

Are there any prerequisites?

No prior Azure or data engineering experience is needed. This guide walks you through setting up every tool from scratch, including your Azure account, the Azure CLI, and the Anthropic API.

Before We Start

Every cloud migration project starts the same way: getting your tools and accounts ready. In this project, you will move legacy CSV data into Azure Data Lake Storage Gen2, transform it with PySpark in Azure Databricks, and then use an LLM to generate a professional cost estimate.

Before any of that can happen, you need the Azure CLI to manage cloud resources from your terminal, an Azure account upgraded to Pay-As-You-Go, and the Anthropic Python SDK to call the Claude API later.

In this step, get ready to:
  • Install Azure CLI and set up an Azure account with Pay-As-You-Go billing.
  • Confirm Python 3 is installed and add the Anthropic Python SDK.
  • Obtain an Anthropic API key for calling Claude.
Install Azure CLI and set up your Azure account

The Azure CLI lets you create and manage Azure resources directly from your terminal. You will use it throughout this project to provision storage accounts, upload data, and more.

  • Press Cmd+Space (macOS) to open Spotlight.
  • Type Terminal and press Enter to open it.
  • Check if Azure CLI is already installed by running this command:
az --version

✔️ I see a version number

Azure CLI is already installed on your machine. You are ready to move on to creating your Azure account.

ⓧ Command not found

Azure CLI is not installed yet. You will install it using Homebrew.

  • Install Azure CLI by running these commands:
brew update && brew install azure-cli

What does this do?

The first command updates Homebrew's package index. The second installs the Azure CLI so you can run az commands from your terminal.

  • After installation completes, verify it worked by running:
az --version

You should see version information printed in your terminal, including the core CLI version and Python version.

Seeing an error about Homebrew?

If you see a message saying Homebrew is not installed, visit brew.sh and follow the install instructions, then retry the commands above.

Help me troubleshoot

Now you need an Azure account. New Azure accounts come with $200 in free credit for 30 days, but you must upgrade to Pay-As-You-Go before you can create certain resources like Databricks.

  • Open https://azure.microsoft.com/free in your browser.
  • Click Start free and follow the prompts to create a Microsoft account (or sign in with an existing one).
  • Complete the identity verification and add a payment method when prompted.

Why do I need a payment method?

Azure requires a card on file to verify your identity, but you will not be charged while you stay within the $200 free credit. You must upgrade from the Free Trial to Pay-As-You-Go so that Databricks (which requires a non-trial subscription) can be created later.

  • Once your account is created, go to the Azure portal at https://portal.azure.com.
  • Search for Subscriptions in the top search bar and select it.
  • Select the subscription that was created with your free account.
  • Click Upgrade subscription in the command bar (or look for the upgrade banner at the top of the page).
  • Follow the prompts to complete the upgrade to Pay-As-You-Go.

What happens after upgrading?

You keep your remaining $200 credit for the full 30 days from when you signed up. You also get 12 months of popular free services. You are only charged for usage beyond the free allowances.

Now log into Azure from your terminal so the CLI can manage resources on your behalf.

  • Log into Azure by running this command:
az login

What happens when I run this?

Your default browser opens an Azure sign-in page. Sign in with the Microsoft account you just created. After signing in, the terminal displays your subscription information.

  • Verify your account is connected by running:
az account show

You should see JSON output containing your subscription name, ID, and state showing as Enabled.

Not seeing your subscription?

If the output shows no subscription or a different one than expected, close and reopen your terminal, then run az login again. Make sure you sign in with the same email you used to create the Azure account.

Help me fix my Azure CLI login

Verify Python and install the Anthropic SDK

You already have Python 3 on your machine. Now you need to add the Anthropic Python package so your scripts can call the Claude API.

  • Confirm Python 3 is available by running:
python3 --version

You should see a version number like Python 3.x.x printed.

  • Install the Anthropic Python SDK by running:
pip3 install anthropic

What is the Anthropic SDK?

This package provides a Python client for the Claude API. It handles authentication, request formatting, and response parsing so you can call Claude with just a few lines of code.

  • Verify the installation succeeded by running:
python3 -c "import anthropic; print(anthropic.__version__)"

You should see a version number printed in the terminal.

Seeing a ModuleNotFoundError?

If the import fails, make sure you ran pip3 install anthropic (not pip install without the 3). On some systems, pip installs to a different Python version than python3.

Help me fix my anthropic import

Get your Anthropic API key

The Claude API requires an API key for authentication. You will create one in the Anthropic console and store it as an environment variable that the SDK reads automatically.

  • Open https://console.anthropic.com in your browser.
  • Create an account or sign in if you already have one.
  • Add a payment method in the billing settings (API usage for this project costs under $0.10 total).
  • Navigate to the API Keys section.
  • Click Create Key, give it a name like migration-project, and copy the key.

Save your key now

Anthropic only shows the full key once at creation time. Copy it immediately and store it somewhere safe. You will not be able to view it again.

  • Set the key as an environment variable so the SDK can find it automatically by running this command, replacing the placeholder with your actual key:
export ANTHROPIC_API_KEY=your-api-key-here

Why ANTHROPIC_API_KEY?

The Anthropic Python SDK automatically reads this environment variable when you create a client. You do not need to pass the key directly in your code, which keeps your credentials out of source files.

Make the key persist across sessions

The export command only sets the variable for your current terminal session. To keep it available after you close the terminal, add the same line to your ~/.zshrc file (or ~/.bashrc if you use Bash).

  • Verify the SDK can authenticate by running:
python3 -c "import anthropic; client = anthropic.Anthropic(); print('Authentication OK')"

You should see Authentication OK printed with no errors.

Seeing an authentication error?

Double-check that you exported the key in the same terminal session where you are running the command. Run echo $ANTHROPIC_API_KEY to confirm it is set. If it prints nothing, re-run the export command with your key.

Help me debug my API key setup

Your environment is ready. Next up, you will create an Azure Data Lake Storage Gen2 account and upload sample legacy data into it.

Set Up Azure and Local Tools

Before building the migration pipeline, you need to confirm all your tools are configured and communicating with each other.

You have the Azure CLI installed and an Azure account upgraded to Pay-As-You-Go. You also have Python 3 and the Anthropic SDK ready. In this step, you will verify each tool is working and store your API key so the SDK can find it.

In this step, get ready to:
  • Verify your Azure CLI is authenticated and connected to your subscription.
  • Confirm the Anthropic Python SDK is installed and importable.
  • Store your Anthropic API key as an environment variable for the SDK to read.
Confirm your Azure CLI is authenticated

The Azure CLI connects your terminal to your Azure subscription. Running a quick check confirms that your login session is active and your subscription is visible.

  • Verify your Azure CLI is connected by running this command:
az account show

What does this command do?

`az account show` retrieves the details of your active Azure subscription. It returns JSON output showing your subscription name, ID, and state.

If it returns data successfully, your CLI is authenticated and ready for the rest of the project.

✔️ I see JSON with my subscription name

You should see output similar to this:

{
  "environmentName": "AzureCloud",
  "isDefault": true,
  "name": "My subscription",
  "state": "Enabled",
  "user": {
    "name": "you@example.com",
    "type": "user"
  }
}

Your Azure CLI is authenticated and ready to go.

ⓧ I see an error or no output

If you see an error about not being logged in, run the login command:

az login

This opens a browser window where you sign in to your Azure account. After signing in, return to your terminal and run az account show again.

Confirm Python and the Anthropic SDK

The cost estimate script you will write later uses the Anthropic Python SDK to call the Claude API. A quick import check confirms the package is installed and accessible.

  • Verify the Anthropic SDK is importable by running this command:
python3 -c "import anthropic; print(anthropic.__version__)"

✔️ I see a version number printed

You should see a version number like 0.109.1 printed to the terminal. This confirms the anthropic package is installed and Python can find it.

ⓧ I see ModuleNotFoundError

If Python cannot find the module, install it by running:

pip install anthropic

After the install completes, run the verification command again. You should now see a version number printed.

Store your Anthropic API key

The Anthropic Python SDK reads your API key from an environment variable called ANTHROPIC_API_KEY. Setting this variable means you never have to hardcode the key in your script.

  • Export your API key by running this command (replace the placeholder with your actual key from console.anthropic.com):
export ANTHROPIC_API_KEY="your-api-key-here"

Why an environment variable?

The SDK reads ANTHROPIC_API_KEY automatically when you create a client with anthropic.Anthropic(). This keeps your key out of your source code.

This export lasts for your current terminal session only. If you close your terminal, you will need to run the export again before running the script.

  • Confirm the key is stored by running:
echo $ANTHROPIC_API_KEY

You should see your key printed back in the terminal.

See a blank line instead of your key?

Make sure you ran the export command in the same terminal session where you are checking. Opening a new terminal tab starts a fresh session without the variable set.

Also check that you included the export keyword before ANTHROPIC_API_KEY. Without it, the variable is not available to child processes like Python.

Help me troubleshoot my API key setup.

Your tools are verified and your API key is stored. Next up, you will create an Azure Data Lake Storage account and upload legacy data into it.

Create Azure Data Lake Storage and Upload Legacy Data

Your Azure CLI is authenticated and ready to go. Now it is time to build the cloud infrastructure that will hold your migrated data.

Every cloud migration starts with a place to store data. In this step, you will create an Azure Data Lake Storage Gen2 account and upload sample legacy data into it. This simulates moving flat-file data from an on-premises data warehouse into Azure's modern lakehouse architecture.

In this step, get ready to:
  • Create an Azure resource group and Data Lake Storage Gen2 account.
  • Create a container (filesystem) inside the Data Lake.
  • Create sample legacy CSV files and upload them to the Data Lake.
Create a resource group and Data Lake Storage Gen2 account

Azure organizes related cloud resources into resource groups. You will create one to hold everything for this project, then provision a storage account with hierarchical namespace enabled.

Hierarchical namespace is what turns a regular storage account into a Data Lake Storage Gen2 account with folder-level permissions and performance optimizations.

  • Create a resource group named migration-rg by running this command:
az group create --name migration-rg --location eastus

What does this command do?

This creates a logical container called migration-rg in the eastus region. All resources you create next will live inside this group.

When you are done with the project, deleting the group removes everything at once.

  • You should see JSON output with "provisioningState": "Succeeded" confirming the group was created.
  • Create your Data Lake Storage Gen2 account by running this command:
az storage account create --name [[STORAGE_ACCOUNT_NAME="a globally unique name e.g. migrationlake followed by random digits"]] --resource-group migration-rg --location eastus --sku Standard_LRS --kind StorageV2 --enable-hierarchical-namespace true

What does this command do?

This provisions a storage account with several important flags:

  • --sku Standard_LRS selects locally redundant storage, the cheapest option for a learning project.
  • --kind StorageV2 uses a general-purpose v2 account, which is required for Data Lake features.
  • --enable-hierarchical-namespace true turns the storage account into a Data Lake Storage Gen2 account with folder-level permissions.

Storage account names must be globally unique

Azure storage account names must be between 3-24 characters, lowercase letters and numbers only. Choose something unique like migrationlake followed by a few random digits.

Remember this name because you will use it throughout the project.

  • Wait for the command to complete. You should see JSON output with "provisioningState": "Succeeded" confirming the storage account was created.

Seeing a naming error?

If you get an error about the storage account name, make sure it contains only lowercase letters and numbers, is between 3-24 characters, and has not already been taken by another Azure user globally.

Help me troubleshoot my storage account naming issue.

Create a container in the Data Lake

In Data Lake Storage Gen2, a container (also called a filesystem) is the top-level folder that organizes your data. Think of it like a root directory for your migrated files.

  • Create a container named legacy-data by running this command:
az storage fs create -n legacy-data --account-name [[STORAGE_ACCOUNT_NAME="your storage account name from above"]] --auth-mode login

What does this command do?

This creates a filesystem (container) called legacy-data inside your storage account. The --auth-mode login flag uses your Azure AD credentials (from az login earlier) instead of requiring a separate storage key.

  • You should see a brief JSON response or no output, which means the container was created successfully.

Getting an authorization error?

If you see a permissions error, you may need the Storage Blob Data Contributor role assigned to your account. You can work around this by retrieving your storage account key and passing it with --account-key instead of --auth-mode login.

Help me fix my authorization error when creating a container.

Create sample legacy data and upload it to the Data Lake

In a real migration, your data would come from an existing database or file server. For this project, you will create two CSV files that represent a fictional company's employee and project data.

  • Create a local directory for your sample data by running this command:
mkdir -p sample_data
  • Create the employee data file by running this command:
cat > sample_data/employees.csv << 'EOF'
id,name,department,salary,hire_date,status
1,Alice Johnson,Engineering,95000,2018-03-15,active
2,Bob Smith,Marketing,72000,2019-07-22,active
3,Carol Williams,Engineering,105000,2017-01-10,active
4,David Brown,Sales,68000,2020-11-03,active
5,Eve Davis,Engineering,98000,2019-05-18,active
6,Frank Miller,Marketing,75000,2021-02-28,active
7,Grace Wilson,Sales,71000,2018-09-14,inactive
8,Henry Taylor,Engineering,110000,2016-06-01,active
9,Iris Anderson,Marketing,69000,2022-01-15,active
10,Jack Thomas,Sales,73000,2020-08-20,active
EOF
  • Create the project data file by running this command:
cat > sample_data/projects.csv << 'EOF'
project_id,project_name,department,budget,start_date,status
P001,Cloud Migration,Engineering,500000,2024-01-15,in_progress
P002,Brand Refresh,Marketing,150000,2024-03-01,completed
P003,Sales Platform,Sales,300000,2024-02-10,in_progress
P004,Data Pipeline,Engineering,250000,2024-04-01,planned
P005,Customer Portal,Engineering,400000,2023-11-01,in_progress
EOF

What do these files represent?

These CSV files simulate the kind of flat-file exports you would find in a legacy on-premises data warehouse. The employees.csv file has HR records with salary and department info. The projects.csv file tracks active company projects and budgets.

Later in the project, you will transform this raw data into optimized Delta Lake format using PySpark in Databricks.

  • Confirm the files were created by running:
ls sample_data/
  • You should see employees.csv and projects.csv in the output.
  • Upload employees.csv to the Data Lake by running this command:
az storage fs file upload -s sample_data/employees.csv -p employees.csv -f legacy-data --account-name [[STORAGE_ACCOUNT_NAME="your storage account name"]] --auth-mode login
  • Upload projects.csv to the Data Lake by running this command:
az storage fs file upload -s sample_data/projects.csv -p projects.csv -f legacy-data --account-name [[STORAGE_ACCOUNT_NAME="your storage account name"]] --auth-mode login

What does this command do?

The az storage fs file upload command uploads a local file to a path inside your Data Lake filesystem. The -s flag points to the local source file. The -p flag sets the destination path. The -f flag specifies which filesystem (container) to upload into.

  • Verify the files are in your Data Lake by running this command:
az storage fs file list -f legacy-data --account-name [[STORAGE_ACCOUNT_NAME="your storage account name"]] --auth-mode login --output table
  • You should see employees.csv and projects.csv listed in the output table. This confirms your legacy data has been migrated to the cloud.

Files not showing up?

Double check that your --account-name matches exactly what you used when creating the storage account. Also verify you used the same -f legacy-data filesystem name in both the upload and list commands.

If you see an authorization error on upload, try retrieving your storage account key and passing it with --account-key instead of --auth-mode login.

Help me troubleshoot why my files are not appearing in my Data Lake container.

Your cloud storage is provisioned and loaded with legacy data. Next up, you will create an Azure Databricks workspace to transform this raw CSV data into optimized Delta Lake format.

Create Databricks Workspace and Connect to Data Lake

Your Data Lake is loaded with legacy CSV data. Now you need a compute engine to read, transform, and write that data in a modern format.

Azure Databricks provides a managed Apache Spark environment purpose-built for large-scale data processing. In this step, you will create a Databricks workspace, spin up a cluster, and connect it to your Data Lake so Spark can read your files directly.

In this step, get ready to:
  • Create an Azure Databricks workspace via the Azure Portal.
  • Launch the workspace and create a single-node cluster.
  • Connect Databricks to your Data Lake using an access key.
Create an Azure Databricks workspace

Azure Databricks runs inside its own workspace. You will create this workspace from the Azure Portal and attach it to your existing resource group.

  • Return to the Azure Portal in your browser.
  • Click + Create a resource in the top-left corner.
  • Click Analytics, then select Azure Databricks.
  • Fill in the Basics tab:
  • Click Review + create, then click Create.
  • Wait for the deployment to complete. This takes 2-5 minutes.

What is Azure Databricks?

Azure Databricks is a managed Apache Spark platform optimized for Azure. It gives you notebooks, clusters, and job scheduling without needing to manage Spark infrastructure yourself.

The Trial pricing tier gives you 14 days of free Premium DBUs (Databricks Units), so you will not be charged for compute during this project as long as you complete it within that window.

Seeing a subscription error?

Azure Databricks requires a Pay-As-You-Go subscription. If you are still on the Azure Free Trial subscription, you must upgrade it first. Go to your subscription in the portal and click Upgrade.

help me fix a subscription error when creating Azure Databricks

Launch the workspace and create a single-node cluster

A cluster is the compute engine that runs your Spark code. You will create a lightweight single-node cluster that is sufficient for this project's data volume.

  • Once deployment completes, click Go to resource.
  • On the Databricks workspace overview page, click Launch Workspace.
  • A new browser tab opens with the Databricks workspace UI.
  • In the left sidebar, click Compute.
  • Click Create compute.
  • Enter a cluster name (e.g. migration-cluster).
  • In the Performance section, check the Single node checkbox.
  • Leave the Databricks Runtime version at the latest LTS default.
  • Click Create compute.

The cluster will take 3-5 minutes to start. You will see the status change from Pending to Running.

Why a single-node cluster?

A single-node cluster runs the Spark driver with no separate workers. It is ideal for learning and small datasets like the sample files in this project.

This keeps costs low and avoids provisioning multiple VMs. For production workloads with larger datasets, you would use a multi-node cluster.

Cluster stuck in Pending?

If the cluster does not start after 5 minutes, check that your subscription has not hit its vCPU quota. The Azure Free Trial limits you to 4 vCPU cores. A single-node Standard_DS3_v2 uses 4 cores, which is exactly at the limit.

help me troubleshoot my Databricks cluster that is stuck in Pending state

Retrieve access key and configure Databricks to read from Data Lake

Databricks needs credentials to read files from your Data Lake. You will retrieve the storage account access key and set it as a Spark configuration property.

  • Retrieve your storage account access key by running this command in your local terminal:
az storage account keys list --account-name [[STORAGE_ACCT="your-storage-account-name"]] --resource-group migration-rg --output table

What does this command do?

This lists the two access keys for your storage account. You only need one of them (key1). Copy the value from the Value column for the first key.

  • Copy the Value of key1 from the output. Keep this somewhere safe for the next step.
  • Switch back to the Databricks workspace in your browser.
  • In the left sidebar, click New, then select Notebook.
  • Attach the notebook to your running cluster using the compute selector at the top of the notebook.
  • In the first cell, paste the following code to configure Data Lake access:
# Configure access to Data Lake
storage_account_name = "your-storage-account-name"
storage_account_key = "your-storage-account-key"

spark.conf.set(
    f"fs.azure.account.key.{storage_account_name}.dfs.core.windows.net",
    storage_account_key
)

What does this code do?

This sets a Spark configuration property that tells the cluster how to authenticate when reading from your ADLS Gen2 account. The fs.azure.account.key property uses your storage account key as the credential.

Replace your-storage-account-name with your actual storage account name, and your-storage-account-key with the key1 value you copied.

  • Replace the placeholder values with your actual storage account name and key.
  • Run the cell by pressing Shift+Enter.
  • Add a new cell below and paste this code to verify the connection by listing files:
# Verify connection by listing files in the container
files = dbutils.fs.ls(f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/")
for f in files:
    print(f.name)

What does this code do?

The dbutils.fs.ls command lists all files in the legacy-data container. The abfss:// protocol is how Spark addresses Azure Data Lake Storage Gen2 endpoints.

  • Run the cell by pressing Shift+Enter.

You should see employees.csv and projects.csv printed in the output. This confirms Databricks can authenticate and read from your Data Lake.

Seeing an authentication error?

Double-check that you replaced both placeholder strings with the correct values. The storage account name must match exactly (all lowercase, no dashes that were not in the original name). The access key is a long Base64 string ending in ==.

Also confirm your cluster is in Running state. A terminated cluster cannot execute cells.

help me fix an authentication error connecting Databricks to my Azure Data Lake

Your Databricks workspace is connected and can read files directly from the Data Lake. Next up, you will write a PySpark notebook to transform the raw CSV data into optimized Delta Lake format.

Transform Data with PySpark

Your Data Lake now holds raw CSV files, but CSV is a slow, unoptimized format for analytics.

In this step, you will write a PySpark notebook that reads those CSVs, transforms the data with computed columns, and writes the result in Delta Lake format. Delta Lake is a columnar storage layer that makes queries dramatically faster and adds versioning to your data.

In this step, get ready to:
  • Read raw CSV files into Spark DataFrames.
  • Transform employee data with computed columns for tenure and salary band.
  • Write optimized Delta Lake tables and verify them with SQL.
Create a notebook and read raw CSV data

Databricks notebooks let you run code cell by cell, so you can inspect results at each stage of your pipeline.

  • In your Databricks workspace sidebar, click + New.
  • Select Notebook.
  • Name the notebook migration_pipeline.
  • Make sure the language is set to Python.
  • Click the compute selector in the toolbar and attach your single-node cluster from the previous step.

The notebook opens with one empty cell. Your first cell configures the connection to your Data Lake.

  • Paste the following code into the first cell:
# Databricks notebook - Run this in your Databricks workspace
# Cell 1: Configure access to Data Lake
storage_account_name = "your-storage-account-name"
storage_account_key = "your-storage-account-key"

spark.conf.set(
    f"fs.azure.account.key.{storage_account_name}.dfs.core.windows.net",
    storage_account_key
)
  • Replace your-storage-account-name with your actual ADLS Gen2 storage account name.
  • Replace your-storage-account-key with your storage account access key from the previous step.

What does this code do?

This cell tells Spark how to authenticate with your Data Lake. The spark.conf.set call registers your storage account key so that all subsequent file reads use it automatically.

  • Run the cell by pressing Shift+Enter.

The cell completes with no output. A new empty cell appears below.

Now you will read your two CSV files into Spark DataFrames so you can inspect and transform them.

  • Paste the following code into the new cell:
# Cell 2: Read raw CSV data from Data Lake
employees_df = spark.read.format("csv").options(
    header="true",
    inferSchema="true"
).load(f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/employees.csv")

projects_df = spark.read.format("csv").options(
    header="true",
    inferSchema="true"
).load(f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/projects.csv")

print("Employees schema:")
employees_df.printSchema()
print(f"Employee count: {employees_df.count()}")

print("Projects schema:")
projects_df.printSchema()
print(f"Project count: {projects_df.count()}")

What does this code do?

  • spark.read.format("csv") tells Spark to read a CSV file from the path you provide.
  • header="true" uses the first row as column names. inferSchema="true" detects column types (integer, string, date) automatically.
  • The abfss:// URI is the protocol for accessing Azure Data Lake Storage Gen2.
  • printSchema() shows each column's name and data type so you can verify the data loaded correctly.
  • Run the cell by pressing Shift+Enter.

You should see the employees schema printed with columns like id, name, salary, and an employee count of 10. The projects schema shows 5 rows.

Seeing a path not found error?

  • Double-check that you replaced the storage account name placeholder in Cell 1 with your actual account name.
  • Verify the container name in the path is legacy-data and matches what you created in Step 2.
  • Make sure Cell 1 ran successfully before this cell. Cells must execute in order.

Help me debug my Databricks CSV read error

Transform the employee data with computed columns

Raw data rarely matches what analysts need. You will filter out inactive employees and add two computed columns: how long each employee has been at the company, and which salary band they fall into.

  • Paste the following code into the next empty cell:
# Cell 3: Transform data
from pyspark.sql.functions import col, current_date, datediff, when, lit

# Add tenure and salary band columns
transformed_employees = employees_df.filter(
    col("status") == "active"
).withColumn(
    "tenure_days", datediff(current_date(), col("hire_date"))
).withColumn(
    "salary_band",
    when(col("salary") < 75000, lit("junior"))
    .when(col("salary") < 100000, lit("mid"))
    .otherwise(lit("senior"))
)

transformed_employees.show()

What does this transformation do?

  • filter(col("status") == "active") removes the inactive employee (Grace Wilson) from the dataset.
  • withColumn("tenure_days", ...) calculates how many days each employee has worked, based on today's date minus their hire date.
  • The salary_band column categorises employees into junior (under $75k), mid (under $100k), or senior ($100k+).
  • Run the cell by pressing Shift+Enter.

You should see a table with 9 rows (Grace Wilson filtered out). Two new columns appear on the right: tenure_days and salary_band.

Seeing an import error?

  • Make sure the import line is inside this cell, not in a previous one. Each cell in Databricks runs in the same session, but the import must execute before the code that uses it.
  • Confirm your cluster is still running. If it auto-terminated, restart it and re-run all cells from the top.

Help me fix my PySpark transformation error

Write to Delta Lake and verify with SQL

Delta Lake stores data in a compressed columnar format with transaction logs. Queries against Delta are significantly faster than scanning raw CSV, and you get features like time travel and schema enforcement for free.

  • Paste the following code into the next empty cell to write your transformed data as Delta tables:
# Cell 4: Write to Delta Lake format
delta_path = f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/delta/employees"
transformed_employees.write.format("delta").mode("overwrite").save(delta_path)

projects_delta_path = f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/delta/projects"
projects_df.write.format("delta").mode("overwrite").save(projects_delta_path)

print("Delta Lake tables written successfully!")

What does this code do?

  • write.format("delta") tells Spark to write in Delta Lake format instead of CSV or Parquet.
  • mode("overwrite") replaces any existing data at that path. This makes your notebook re-runnable without errors.
  • Both DataFrames are written to a delta/ subdirectory inside your Data Lake container, keeping raw and transformed data separate.
  • Run the cell by pressing Shift+Enter.

You should see the message Delta Lake tables written successfully! after a few seconds.

Write operation taking too long or failing?

  • Check that your cluster is still running. Writing Delta tables requires an active Spark session.
  • If you see a permission error, verify that the storage account key in Cell 1 is correct and that Cell 1 was executed.

Help me debug my Delta Lake write failure

Now verify that the Delta tables are queryable using SQL. This proves your data is stored in a format ready for analytics.

  • Paste the following code into the next empty cell:
# Cell 5: Verify Delta Lake tables
delta_employees = spark.read.format("delta").load(delta_path)
delta_employees.createOrReplaceTempView("employees")

result = spark.sql("""
    SELECT salary_band, COUNT(*) as count, ROUND(AVG(salary), 0) as avg_salary
    FROM employees
    GROUP BY salary_band
    ORDER BY avg_salary
""")
result.show()

What does this verification do?

  • spark.read.format("delta").load(delta_path) reads the Delta table back from storage to prove it was written correctly.
  • createOrReplaceTempView("employees") registers the DataFrame as a temporary SQL table so you can query it with standard SQL.
  • The SQL query groups employees by salary band and calculates the count and average salary in each band.
  • Run the cell by pressing Shift+Enter.

You should see a table with three rows showing salary_band, count, and avg_salary columns. Junior, mid, and senior bands each show the number of employees and average salary.

✔️ Awesome, I've got everything!

Great. Make sure your notebook is saved (Databricks auto-saves, but you can verify the title shows no unsaved indicator).

ⓧ I'd like to double check the full code

Here is the full notebook with all five cells. Compare your notebook against this:

# Databricks notebook - Run this in your Databricks workspace
# Cell 1: Configure access to Data Lake
storage_account_name = "your-storage-account-name"
storage_account_key = "your-storage-account-key"

spark.conf.set(
    f"fs.azure.account.key.{storage_account_name}.dfs.core.windows.net",
    storage_account_key
)

# Cell 2: Read raw CSV data from Data Lake
employees_df = spark.read.format("csv").options(
    header="true",
    inferSchema="true"
).load(f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/employees.csv")

projects_df = spark.read.format("csv").options(
    header="true",
    inferSchema="true"
).load(f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/projects.csv")

print("Employees schema:")
employees_df.printSchema()
print(f"Employee count: {employees_df.count()}")

print("Projects schema:")
projects_df.printSchema()
print(f"Project count: {projects_df.count()}")

# Cell 3: Transform data
from pyspark.sql.functions import col, current_date, datediff, when, lit

# Add tenure and salary band columns
transformed_employees = employees_df.filter(
    col("status") == "active"
).withColumn(
    "tenure_days", datediff(current_date(), col("hire_date"))
).withColumn(
    "salary_band",
    when(col("salary") < 75000, lit("junior"))
    .when(col("salary") < 100000, lit("mid"))
    .otherwise(lit("senior"))
)

transformed_employees.show()

# Cell 4: Write to Delta Lake format
delta_path = f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/delta/employees"
transformed_employees.write.format("delta").mode("overwrite").save(delta_path)

projects_delta_path = f"abfss://legacy-data@{storage_account_name}.dfs.core.windows.net/delta/projects"
projects_df.write.format("delta").mode("overwrite").save(projects_delta_path)

print("Delta Lake tables written successfully!")

# Cell 5: Verify Delta Lake tables
delta_employees = spark.read.format("delta").load(delta_path)
delta_employees.createOrReplaceTempView("employees")

result = spark.sql("""
    SELECT salary_band, COUNT(*) as count, ROUND(AVG(salary), 0) as avg_salary
    FROM employees
    GROUP BY salary_band
    ORDER BY avg_salary
""")
result.show()

Your raw CSV data is now transformed and stored as optimized Delta Lake tables. In the next step, you will use the Claude API to generate a professional migration cost estimate from your actual Azure infrastructure.

Secret mission

Multi-Phase Migration Roadmap

Your cost estimate covers a single snapshot in time. Real migration projects unfold over months or years, with costs scaling at each phase. Can you extend your script to generate a full multi-phase roadmap with per-phase cost projections?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The biggest cost driver in this project is the Azure Databricks cluster VM, which charges while running.

Cost warning

Your Databricks cluster costs money for every minute it runs. The cluster auto-terminates after 120 minutes of inactivity by default, but leaving it running overnight could cost $10-20.

If you are done for the day, terminate the cluster manually or delete the entire resource group to avoid surprise charges.

Resources you used:

  • Azure resource group migration-rg (contains your Databricks workspace and Data Lake Storage Gen2 account).
  • Local project files (sample_data/, generate_cost_estimate.py, generate_roadmap.py, databricks_notebook.py, migration_cost_estimate.md, migration_roadmap.md).
  • Azure CLI (installed via Homebrew).
  • anthropic Python package.
  • ANTHROPIC_API_KEY environment variable.

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue the project soon.

  • Your Databricks cluster will auto-terminate after 120 minutes of inactivity. Storage costs are negligible for small files.
  • The ANTHROPIC_API_KEY environment variable stays set in your current terminal session. It disappears when you close the terminal.
  • Local files remain on your machine until you manually delete them.
  • Azure CLI and the anthropic package remain installed.

Pause - I'll come back to this later

Shut down running processes to stop VM charges immediately. Your files and storage remain intact for when you return.

  • In the Azure Databricks workspace, navigate to Compute in the left sidebar.
  • Click the three-dot menu next to your cluster and select Terminate. This stops all VM charges immediately.
  • Your Data Lake Storage Gen2 account remains active but costs only pennies per month for the small CSV and Delta files.
  • Local files and installed tools stay as-is.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your Azure infrastructure and local files.

Delete all Azure resources at once by deleting the resource group. Run this in your terminal:

az group delete --name migration-rg --yes --no-wait

This removes the Databricks workspace, Data Lake Storage Gen2 account, and all data inside them.

  • Delete your local project files by running:
rm -rf sample_data generate_cost_estimate.py generate_roadmap.py databricks_notebook.py migration_cost_estimate.md migration_roadmap.md
  • Unset the ANTHROPIC_API_KEY environment variable by running:
unset ANTHROPIC_API_KEY
  • Uninstall the anthropic Python package by running:
pip3 uninstall anthropic -y
  • Uninstall Azure CLI and remove its configuration by running:
brew uninstall azure-cli
rm -rf ~/.azure

That's a Wrap!

That's a Wrap!

Nice work! You just built an end-to-end cloud migration pipeline on Azure and used an LLM to generate the kind of stakeholder-ready documentation that technical leaders actually deliver.

You've learned how to:

  • Provision Azure Data Lake Storage Gen2 and an Azure Databricks workspace from scratch using the Azure CLI and Portal.
  • Transform raw CSV data into optimized Delta Lake format using a PySpark notebook in Databricks.
  • Use the Claude API to generate a professional migration cost analysis memo from real infrastructure metadata.
  • Secret Mission: Generated a multi-phase migration roadmap with per-phase cost projections and timeline estimates.

Ready to quiz yourself?