Build an ETL Pipeline with Azure Data Factory

Build a scheduled ETL pipeline that moves CSV data into Azure SQL Database.

Introduction

30 Second Summary

Every time you order something online, your purchase data flows through pipelines that extract it from raw logs, transform it into a usable format, and load it into databases where analysts can run queries. These ETL pipelines are the backbone of every data-driven company.

In this project, you will build a complete ETL pipeline using Azure Data Factory that ingests raw sales CSV data from Azure Blob Storage, loads it into an Azure SQL Database you designed yourself, and runs on an automated schedule.

What You'll Build

A production-style ETL pipeline that moves data from cloud storage into a SQL database using Azure Data Factory's Copy Activity, with a scheduled trigger for automation.

By the end of this project, you'll have:

A working Azure Data Factory pipeline that ingests raw CSV data from Blob Storage and loads it into a SQL Database using Copy Activity.
A SQL schema you designed from scratch with proper column types, NOT NULL constraints, and a primary key.
A scheduled trigger that automates the pipeline to run daily, with monitoring to verify successful execution.
Secret Mission: Add a Mapping Data Flow that filters invalid rows and creates a derived column before loading to SQL.

Are there any prerequisites?

You need an Azure for Students account (provided through your university subscription). This gives you $100 in free credit and access to the free-tier services used in this project.

No local software installations are required. Everything runs in the Azure Portal browser UI.

Create Your Azure Data Services

To build an ETL pipeline, you need three core services working together: a place to store raw data, a database to load it into, and an orchestration tool to move it between them.

In this step, you'll create all three Azure resources that form the foundation of your pipeline. By the end, you'll have the infrastructure ready for data to flow through.

In this step, get ready to:

Create an Azure Data Factory instance to orchestrate your pipeline.
Create a Storage Account with a Blob container for source data.
Create an Azure SQL Database using the free offer.

Create an Azure Data Factory instance

Azure Data Factory is the orchestration tool that will move your data from point A to point B. Think of it as the manager that tells your pipeline when and how to run.

In the Azure Portal, select Create a resource from the left menu.
Select Analytics, then select Data Factory.
On the Create Data Factory page, select your Azure subscription.
Under Resource group, select Create new and enter a name for your resource group (for example, etl-pipeline-rg). Select OK.
Under Region, select a location near you from the dropdown.
Under Name, enter a globally unique name for your data factory (for example, yourname-adf-etl).
Under Version, select V2.
Select Review + create. After validation passes, select Create.
After the deployment completes, select Go to resource.

What is Azure Data Factory?

Azure Data Factory is Azure's cloud-based data integration service. It lets you create, schedule, and monitor data pipelines that move and transform data across different stores.

It appears in 84% of Azure Data Engineer interview loops, making it the single most-tested tool for this role.

Seeing a name already taken error?

Data Factory names must be globally unique across all of Azure. Try adding your initials or a number to make it unique (for example, jd-adf-etl-2026).

If you're stuck, help me pick a unique Data Factory name.

Create a Storage Account with a Blob container

Your pipeline needs a source to pull data from. You'll create an Azure Storage Account with a Blob container where you'll upload your raw CSV file in the next step.

In the Azure Portal left menu, select Storage accounts.
Select Create.
On the Basics tab, select the same subscription you used earlier.
Under Resource group, select the resource group you just created (for example, etl-pipeline-rg).
Under Storage account name, enter a unique name using only lowercase letters and numbers (for example, yournameetlstorage). Names must be 3 to 24 characters.
Under Region, select the same region you chose for your Data Factory.
Leave Performance set to Standard.
Select Review + create. After validation passes, select Create.
After the deployment completes, select Go to resource.

Now create a Blob container inside your new Storage Account.

In the left menu of your storage account, scroll to the Data storage section and select Containers.
Select the + Container button.
Enter source-data as the container name.
Leave the anonymous access level set to Private (no anonymous access).
Select Create.

What is Blob Storage?

Azure Blob Storage is optimized for storing large amounts of unstructured data like text files, images, and CSVs. "Blob" stands for Binary Large Object.

Containers are like folders that organize your blobs. Your source-data container will hold the raw CSV file that your pipeline ingests.

Storage account name not available?

Storage account names must be globally unique across Azure and can only contain lowercase letters and numbers (no dashes or special characters). Try adding random digits to the end of your name.

If you're having trouble, help me pick a valid storage account name.

Create an Azure SQL Database using the free offer

The SQL Database is your pipeline's destination. This is where cleaned, structured data will land after your pipeline processes it. You'll use Azure's free offer, which gives you 100,000 vCore seconds of serverless compute and 32 GB of storage every month.

In your browser, navigate to the Azure SQL hub at aka.ms/azuresqlhub.
In the Create a database pane, select Start free.
Confirm you see the "Free offer applied!" banner at the top of the page.
On the Basics tab, select your subscription.
Under Resource group, select the same resource group you created earlier (etl-pipeline-rg).
Under Database name, enter a name for your database (for example, sales-etl-db).
Under Server, select Create new.
In the New server form, enter a globally unique server name (for example, yourname-sql-server).
Select a Location from the dropdown (use the same region as your other resources).
Under Authentication method, select Use SQL authentication.
Enter a Server admin login (for example, sqladmin) and a strong password. Write these down. You'll need them later.
Select OK to close the server form.
Verify the Cost summary card on the right shows Estimated Cost/Month as zero.
Select Review + create, then select Create.

What is the free offer?

The Azure SQL Database free offer provides up to 10 databases with 100,000 vCore seconds of serverless compute and 32 GB of storage each, every month for the life of your subscription.

General Purpose serverless means the database auto-pauses when idle and only charges for compute when active. With the free offer, once monthly limits are met, the database auto-pauses until the beginning of the next calendar month.

Remember your SQL credentials

Write down your server admin login and password somewhere safe. You'll need these credentials later when connecting to the database from the Azure Portal Query Editor and when creating Linked Services in Data Factory.

Don't see the free offer banner?

Make sure you're navigating to the Azure SQL hub directly via aka.ms/azuresqlhub and clicking Start free. If the banner still doesn't appear, try selecting your server from the dropdown again. This often triggers the banner to show.

The Microsoft Azure for Students Starter offer is incompatible with this free offer. If you're on a Students Starter subscription, use the Azure for College Students offer or the standard Azure Free offer instead.

If you're still stuck, help me access the Azure SQL Database free offer.

Now confirm all three resources are in your resource group.

In the Azure Portal search bar at the top, type Resource groups and select it.
Select your resource group (etl-pipeline-rg).
Confirm you see all three resources listed: your Data Factory, your Storage Account, and your SQL Database (plus the SQL Server).

Your infrastructure is ready. Next up, you'll upload source data and design the SQL schema that your pipeline will load into.

Upload Source Data and Design Your SQL Schema

Your three Azure services are up and running. Now it's time to give your pipeline something to work with.

Every ETL pipeline starts with raw data that has real-world messiness. In this step, you'll upload a sample CSV file with intentional data quality issues, then design a SQL table schema that's ready to receive that data.

In this step, get ready to:

Upload a sample sales CSV file to your Blob Storage container.
Write and run a CREATE TABLE statement in Azure SQL Database.
Verify your empty destination table exists with the correct schema.

Download and upload the source CSV file

Real-world data is rarely clean. Your source file has intentional issues that mimic what you'd encounter in production: missing values, inconsistent date formats, and duplicate rows.

Open any text editor on your computer (Notepad on Windows, TextEdit on macOS).
Paste the following CSV content into the editor:

OrderID,CustomerName,Product,OrderDate,OrderAmount,Quantity
1001,Alice Johnson,Laptop,2024-01-15,1299.99,1
1002,Bob Smith,Mouse,01/20/2024,24.99,3
1003,Carol White,Keyboard,2024-02-01,79.99,2
1004,,Monitor,2024-02-10,349.99,1
1005,David Brown,Laptop,2024/03/05,1299.99,1
1006,Eve Davis,Mouse,2024-03-12,24.99,5
1003,Carol White,Keyboard,2024-02-01,79.99,2
1007,Frank Miller,Headphones,,89.99,2
1008,Grace Lee,Webcam,2024-04-01,59.99,1
1009,Henry Wilson,Laptop,04-15-2024,1399.99,1

What data quality issues are in this file?

Null values: Row 1004 is missing a CustomerName, and row 1007 is missing an OrderDate.
Inconsistent date formats: You'll see 2024-01-15, 01/20/2024, 2024/03/05, and 04-15-2024 all in the same column.
Duplicate rows: OrderID 1003 appears twice with identical data.

Save the file as sales_data.csv on your Desktop.
Return to the Azure Portal and navigate to your Storage Account.
In the left menu, scroll to the Data storage section, then select Containers.
Select the container you created in Step 1.
Select the Upload button to open the upload blade.
Browse to your sales_data.csv file on your Desktop and select it.
Select Upload to upload the blob.

Your file should now appear listed inside the container.

Don't see your file in the container?

Make sure you saved the file with a .csv extension, not .txt. On Windows, Notepad may append .txt automatically. Choose All Files in the Save As type dropdown when saving.
Confirm you selected the correct container before clicking Upload.

Help me upload my CSV to Azure Blob Storage.

Connect to Query Editor and write your CREATE TABLE statement

Now you'll design the destination table that your pipeline will load data into. The schema needs column types that match your CSV data and constraints that enforce data quality.

In the Azure Portal, navigate to your Azure SQL Database resource.
On the database Overview page, select Query editor (preview) from the resource menu on the left.
On the sign-in screen, select SQL server authentication.
Enter the server admin Login and Password you set when creating the database in Step 1, then select OK.

What is the Query Editor?

The Query Editor is a browser-based tool built into the Azure Portal. It lets you run SQL queries directly against your Azure SQL Database without installing any local tools like SQL Server Management Studio.

Clear any existing text in the query pane, then paste the following CREATE TABLE statement:

-- Create the destination table for sales data
CREATE TABLE SalesOrders (
    OrderID INT NOT NULL PRIMARY KEY,
    CustomerName NVARCHAR(100),
    Product NVARCHAR(100) NOT NULL,
    OrderDate DATE,
    OrderAmount DECIMAL(10, 2) NOT NULL,
    Quantity INT NOT NULL
);

What does this schema do?

OrderID is an INT primary key. Every row must have a unique order ID.
CustomerName is NVARCHAR(100) and nullable. This allows the rows with missing customer names in your CSV to load without errors.
Product is NVARCHAR(100) NOT NULL. Every order must have a product name.
OrderDate is DATE and nullable. This accommodates the rows missing a date value.
OrderAmount is DECIMAL(10, 2) NOT NULL. This stores currency values with exactly two decimal places.
Quantity is INT NOT NULL. Every order must specify a quantity.

The NOT NULL constraints enforce data quality on critical business columns. Columns with known null values in your source data are left nullable so the pipeline can complete without errors.

Run the statement and verify your table

Select Run in the Query Editor toolbar to execute the CREATE TABLE statement.
You should see a message in the Results pane confirming the query completed successfully.

Now verify the empty table exists with the correct columns.

Clear the query pane and paste this SELECT statement:

-- Verify the table exists and check its columns
SELECT * FROM SalesOrders;

Select Run.
You should see all six column headers (OrderID, CustomerName, Product, OrderDate, OrderAmount, Quantity) with zero rows returned. This confirms your table schema is ready.

Seeing an error when running CREATE TABLE?

If you see a login failure, double-check you're using the server admin credentials you set during database creation in Step 1.
If you see "There is already an object named 'SalesOrders'", the table already exists. You can drop it first with DROP TABLE SalesOrders; and then re-run the CREATE TABLE statement.

Help me fix my CREATE TABLE error in Azure SQL Query Editor.

✔️ Awesome, I've got everything!

Great. Make sure you've saved your sales_data.csv in Blob Storage and confirmed the empty SalesOrders table in Query Editor.

ⓧ I'd like to double check the full code

Here's the complete CREATE TABLE statement your database should have:

-- Create the destination table for sales data
CREATE TABLE SalesOrders (
    OrderID INT NOT NULL PRIMARY KEY,
    CustomerName NVARCHAR(100),
    Product NVARCHAR(100) NOT NULL,
    OrderDate DATE,
    OrderAmount DECIMAL(10, 2) NOT NULL,
    Quantity INT NOT NULL
);

Your source data is staged and your destination table is ready. Next up, you'll build the ADF pipeline that connects these two and moves data from Blob Storage into SQL Database.

Build the ADF Pipeline with Copy Activity

Your source data is sitting in Blob Storage and your destination table is waiting in Azure SQL Database. Now you need to connect the two.

In this step, you'll build the actual pipeline in Azure Data Factory Studio that moves data from point A to point B. This involves creating Linked Services (connections to your data stores) and Datasets (pointers to specific data within those stores).

In this step, get ready to:

Create a pipeline with a Copy Data activity.
Configure the source connection to your Blob Storage CSV file.
Configure the sink connection to your Azure SQL Database table and verify column mappings.

Open ADF Studio and create a new pipeline

Azure Data Factory Studio is the browser-based authoring environment where you build pipelines visually. You'll launch it from the Azure Portal and create your first pipeline with a Copy Activity.

In the Azure Portal, navigate to your resource group and select your Data Factory resource.
Click Launch studio on the Data Factory overview page. This opens Azure Data Factory Studio in a new browser tab.
In Azure Data Factory Studio, select the Author tab (the pencil icon on the left sidebar).
Select the plus sign, and then choose Pipeline.
In the Activities pane on the left, expand the Move and Transform section.
Drag the Copy data activity from the pane onto the pipeline canvas.

What are Linked Services and Datasets?

A Linked Service is like a connection string. It tells ADF how to connect to a data store (your Storage Account or SQL Database).

A Dataset points to a specific piece of data within that store (your CSV file or your SQL table). You'll create one Linked Service and one Dataset for the source, and another pair for the sink.

ADF Studio not loading?

ADF Studio requires Microsoft Edge or Google Chrome. If the page hangs, try clearing cookies for adf.azure.com or opening it in an incognito window.

help me troubleshoot ADF Studio not loading

Configure the source Linked Service and Dataset

The source is where your pipeline reads data from. You'll connect it to the Storage Account that holds your sales CSV file.

Select the Copy data activity on the canvas, then go to the Source tab in the properties panel below.
Click + New to create a source dataset.
In the New Dataset dialog, select Azure Blob Storage, then click Continue.
In the Select Format dialog, choose DelimitedText, then click Continue.
In the Set Properties dialog, give your dataset a name (e.g. SourceSalesCSV).
Select the checkbox for First row as header.
Under Linked service, click + New to create a new Linked Service.
In the New Linked Service dialog, enter a name (e.g. BlobStorageLinkedService).
For Authentication method, select Account key.
Select your Storage Account from the Storage account name dropdown.
Click Test connection to verify it works, then click Create.
Back in the Set Properties dialog, click Browse next to the file path.
Navigate to your Blob container and select your sales CSV file, then click OK.
Click OK to finish creating the source dataset.

Why Account key authentication?

Account key is the simplest authentication method for connecting ADF to your own Storage Account. It uses the access key from your storage account to establish the connection.

For production workloads, you'd typically use Managed Identity. Account key is perfect for learning and testing.

You should now see your source dataset listed in the Source tab of the Copy data activity.

Test connection failing?

Make sure you selected the correct Storage Account from the dropdown. If your account uses firewall rules, you may need to add an exception for Azure services. Check that the Authentication method is set to Account key.

help me fix a failed test connection to Azure Blob Storage in ADF

Configure the sink and verify column mappings

The sink is where your pipeline writes data to. You'll connect it to the Azure SQL Database table you created in the previous step, then verify the column mappings between your CSV and SQL table.

Select the Copy data activity on the canvas, then go to the Sink tab.
Click + New to create a sink dataset.
In the New Dataset dialog, search for SQL, select Azure SQL Database, then click Continue.
In the Set Properties dialog, give your dataset a name (e.g. SinkSalesSQL).
Under Linked service, click + New to create a new Linked Service for your SQL Database.
In the New Linked Service dialog, enter a name (e.g. AzureSQLLinkedService).
Under Server name, select your SQL Server instance.
Under Database name, select your database.
Enter the User name and Password you set when creating the SQL Database.
Click Test connection to verify it works, then click Create.
Back in the Set Properties dialog, select your destination table from the Table name dropdown.
Click OK to finish creating the sink dataset.

Now you'll verify the column mappings so ADF knows which CSV columns map to which SQL columns.

With the Copy data activity selected, go to the Mapping tab.
Click Import schemas to import both source and sink schemas.
Review the column mappings. Each source CSV column should map to its corresponding SQL table column.
Verify that column names and types align correctly (e.g. your CSV OrderDate column maps to the SQL DATE column you created).

What does Import schemas do?

This button reads the structure of both your source (CSV headers) and sink (SQL table columns) and displays them side by side. ADF automatically matches columns with the same name.

You can adjust any mappings that don't look right before running the pipeline.

Finally, validate your pipeline to confirm everything is configured correctly.

Click Validate in the top bar of the pipeline canvas.
You should see a message confirming the pipeline has no errors.

Seeing validation errors?

The most common causes are a missing table selection in the Sink dataset, or a failed Linked Service connection. Go back to the Source and Sink tabs and confirm that both datasets are selected and their Linked Services show a successful test connection.

help me fix ADF pipeline validation errors

Your pipeline is wired up and validated. Next, you'll run it and watch your data flow from Blob Storage into SQL Database.

Run, Monitor, and Verify

Your pipeline is configured and validated. Now it's time for the moment of truth: running it and confirming that data actually moves from Blob Storage into your SQL table.

A pipeline that passes validation only confirms the structure is correct. Running a debug test proves that the connections, permissions, and column mappings work end-to-end with real data.

In this step, get ready to:

Execute a debug run and confirm it succeeds.
Monitor pipeline metrics in ADF Studio.
Verify loaded data with SQL queries.

Trigger a debug run of your pipeline

A debug run executes your pipeline immediately without needing to publish it first. This is how you test changes during development.

Switch back to Azure Data Factory Studio and make sure you're on the Author tab with your pipeline open.
Click Debug in the top toolbar of the pipeline canvas.

What does Debug do?

Selecting Debug actually runs the pipeline. Your Copy Activity will read data from Blob Storage and write it into your SQL table. You don't need to publish your changes first.

Watch the Output tab at the bottom of the pipeline canvas. It shows real-time progress of your debug run.
Wait for the Status column to show Succeeded. This confirms data was copied from source to sink.

The Output tab also displays the row count that was read and written. Note this number for later verification.

Pipeline shows Failed status?

Click the error icon in the Status column to see the error details. Check that your Storage Account key and SQL Database credentials are still valid in the Linked Services.
Verify your CSV file is still in the Blob container and that column names match what the Dataset expects.
Make sure the SQL Database is not paused. If it auto-paused, it may take a minute to resume when the pipeline connects.

Help me troubleshoot my ADF pipeline debug failure.

Check the Monitor tab for run history

The Monitor tab gives you a historical view of all pipeline runs. In a production environment, this is where you would track pipeline health over time.

Click the Monitor icon in the left sidebar of ADF Studio (it looks like a gauge).
Select the Debug tab at the top to see your debug run history.
Find your pipeline run in the list. Confirm it shows Succeeded in the Status column.
Note the Duration column, which shows how long the pipeline took to execute.

What does the Monitor tab show?

The pipeline run grid displays Pipeline Name, Run Start, Run End, Duration, Status, and Error (if any). You can click on the pipeline name to drill into individual activity runs and see details like data read and data written.

For your Copy Activity, the activity-level details show the number of rows read from source and rows written to the sink. This is how you confirm the exact volume of data that moved.

Click on your pipeline name in the Monitor view to see the activity run details.
Hover over the Copy Activity run to see the data read and data written metrics. These numbers should match.

Verify data in SQL Database with queries

The Monitor tab confirms data moved, but you need to verify it landed correctly in your SQL table. This means checking row counts, inspecting sample data, and confirming that NULL values were handled as expected.

Switch back to the Azure Portal and navigate to your Azure SQL Database resource.
Open the Query Editor from the left sidebar and sign in with your SQL credentials.

Check the total row count by running this query:

-- Count total rows loaded from CSV
SELECT COUNT(*) AS TotalRows
FROM SalesOrders;

What does this query do?

This counts every row in your destination table. The number should match the row count you saw in the ADF Output tab. If your CSV had a header row (which ADF skips) and, say, 50 data rows, you should see 50 here.

Inspect sample rows to confirm data looks correct by running:

-- Preview sample data to verify column mapping
SELECT TOP 10 *
FROM SalesOrders;

What should I look for?

Check that each column contains the right type of data. Dates should look like dates, amounts should be numeric, and text fields should not be truncated. This confirms your column mappings from the Copy Activity are correct.

Verify how NULL values were handled by running:

-- Check for NULL values in critical columns
SELECT COUNT(*) AS NullCount
FROM SalesOrders
WHERE OrderAmount IS NULL OR OrderDate IS NULL;

Why check for NULLs?

Your source CSV contains some intentional null values. If you set NOT NULL constraints on critical columns during table creation, rows with NULLs in those columns would have been rejected by SQL Database. This query helps you understand what actually made it through.

If the count is zero and your CSV had nulls, your NOT NULL constraints blocked those rows. If you see a non-zero count, those columns allow NULLs in your schema.

Query returns 0 rows total?

Go back to the Monitor tab in ADF Studio and check whether the activity details show any rows written. If rows read is greater than zero but rows written is zero, there may be a schema mismatch.
Verify you are querying the correct database. Check the database name in the top-left of Query Editor matches the one your ADF Linked Service points to.

Help me figure out why my SQL table is empty after the ADF pipeline ran.

Your data is flowing. Next up, you'll automate this pipeline with a scheduled trigger so it runs without manual intervention.

Add a Scheduled Trigger

Your pipeline runs successfully when you trigger it manually with Debug. In production, data pipelines need to run automatically on a schedule without anyone clicking a button.

In this step, you'll create a scheduled trigger that fires your pipeline daily. You'll publish it, verify the triggered run in the Monitor tab, and then disable it to avoid unnecessary executions.

In this step, get ready to:

Create a schedule trigger with daily recurrence.
Publish your pipeline and trigger to activate them.
Verify the triggered run succeeded, then disable the trigger.

Create a schedule trigger on your pipeline

A trigger determines when a pipeline run should start. Schedule triggers fire on a wall clock schedule, so your pipeline runs at the same time every day without manual intervention.

In ADF Studio, make sure you're on the Author tab with your pipeline open.
Click Add Trigger in the top toolbar of your pipeline canvas.
Select New/Edit from the dropdown.
On the Add triggers page, click Choose trigger, then select + New.
Give your trigger a name (e.g. DailyPipelineRun).
Set the Type to Schedule.
Set the Recurrence to 1 Day.
Set the Start Date to today's date, and the start time to a few minutes from now. This lets you test the trigger without waiting until tomorrow.
Make sure Activated is set to Yes.
Click OK to save the trigger configuration.
On the next screen, review the trigger-run parameters and click OK to confirm.

Why set the start time a few minutes from now?

Schedule triggers fire based on the clock. By setting the start time just a few minutes ahead, you can confirm the trigger works without waiting a full day.

Once you've verified it fires successfully, you'll disable it so it doesn't keep running daily against your free tier allowance.

Publish all to activate the trigger

ADF requires publishing before triggers activate. Until you publish, your trigger exists only as a draft and will not fire.

Click Publish All in the top toolbar of ADF Studio.
Review the pending changes in the publish pane. You should see your trigger listed alongside any pipeline changes.
Click Publish to deploy everything to the live data factory.

Now confirm the trigger is active and visible in the management hub.

Click the Manage tab on the left side of ADF Studio.
Under Author, select Triggers.
Verify your trigger (e.g. DailyPipelineRun) appears in the list with a Started status.

Don't see your trigger in the list?

Make sure you clicked Publish All and the publish completed without errors. Unpublished triggers remain as drafts and won't appear in the Manage tab.

Also check that you set Activated to Yes when configuring the trigger. If it was set to No, the trigger won't start.

Help me troubleshoot my ADF trigger not appearing in the Manage tab.

Verify the triggered run and disable the trigger

Your trigger is now live and will fire at the start time you set. Wait for that time to pass, then check the Monitor tab to confirm the run completed.

Click the Monitor tab on the left side of ADF Studio.
Select Trigger runs (not Pipeline runs) to see runs initiated by your trigger.
Wait for your scheduled time to pass. Click Refresh to update the list.
Look for a run with status Succeeded and a Triggered By value matching your trigger name.

What if the trigger hasn't fired yet?

Triggers can take a minute or two past the scheduled time before they show up. Keep clicking Refresh every 30 seconds. If nothing appears after 5 minutes, double-check that your start time has actually passed (account for time zones).

Once you've confirmed a successful triggered run, disable the trigger to avoid repeated daily runs against your free tier.

Switch to the Manage tab on the left side.
Under Author, select Triggers.
Hover over your trigger and click the Stop button to disable it.
Click Publish All to save the change.

Why disable the trigger?

Azure Data Factory's free tier includes 5 low-frequency activities per month. A low-frequency activity is one that does not execute more than once per day.

Your daily trigger counts as one of those 5 free activities. Leaving it enabled is safe for this project, but disabling it after verification keeps your free allowance available for other experiments.

Triggered run shows Failed?

Click on the failed run in the Monitor tab to see the error details. Common causes include the SQL Database being auto-paused (it needs a moment to resume on first connection) or the Blob Storage file being moved or deleted since your last run.

If the SQL Database was paused, simply wait a minute and trigger another run. The database resumes automatically on the first connection attempt.

Help me debug my failed ADF trigger run.

Your pipeline is now fully automated. You've built a complete ETL pipeline that ingests data from Blob Storage, loads it into SQL Database, and runs on a schedule. Next up, an optional Secret Mission that adds data transformations to clean and enrich your data before it lands in SQL.

Secret mission

Add a Mapping Data Flow for Transformations

Your pipeline moves data, but what about cleaning it first? In this secret mission, you'll use ADF's visual transformation engine to filter out invalid rows and create a derived column. All without writing a single line of code.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All three services in this project remain within the Azure free tier for light usage, so there are no ongoing costs as long as you keep the scheduled trigger disabled.

Resources you used:

Azure Data Factory instance (free for the first 5 low-frequency activities per month).
Azure Storage Account with a Blob container (free tier includes 5 GB Blob Storage).
Azure SQL Database (free offer: 100,000 vCore seconds per month, 32 GB storage, auto-pauses when limits reached).

Keep everything running

No action needed. Choose this if you're still actively building or want to revisit your pipeline later.

All three services remain within the free tier for light usage.
The scheduled trigger will count against your 5 free ADF activities if left enabled. Make sure it stays disabled unless you need it running.
Azure SQL Database auto-pauses when free limits are reached, so no charges accrue.

Pause - I'll come back to this later

Free up resources and prevent any accidental activity usage while keeping your files and configuration intact.

Open Azure Data Factory Studio and click the Manage tab in the left sidebar.
Select Triggers and confirm your scheduled trigger is stopped (disabled). If it shows as Started, select it and stop it.
Your Azure SQL Database will auto-pause once monthly free limits are met. No manual action is needed for it.
Your Storage Account and Data Factory remain available but idle. No charges accrue while they stay within free tier limits.

Delete - I don't want to use this again

Remove all project resources at once by deleting the resource group. This is the fastest way to clean up everything.

In the Azure Portal, select Resource groups from the left navigation panel.
Select the resource group you created for this project.
Click Delete resource group at the top of the page.
Type the resource group name to confirm, then click Delete.
Click Delete again to confirm. This removes your Data Factory, Storage Account, and SQL Database in one action.

Why delete the resource group?

Deleting the resource group removes all resources contained within it. Since you created all three services in the same resource group, this single action cleans up everything without needing to delete each resource individually.

Nice Work!

You just built a production-style ETL pipeline in Azure Data Factory. From raw CSV data sitting in Blob Storage to a fully validated SQL Database table running on a scheduled trigger, you now have a complete, end-to-end data pipeline you can demo and discuss in any data engineering interview.

Here's what you accomplished:

Designed a SQL database schema from scratch with proper column types, NOT NULL constraints, and a primary key, then built an Azure Data Factory pipeline that moves data from Blob Storage to Azure SQL Database using Linked Services, Datasets, and Copy Activity.
Configured a scheduled trigger that automates your pipeline to run daily, then published and monitored the run in ADF Studio's Monitor tab.
Validated data quality by writing SQL queries in the Azure Portal Query Editor to confirm row counts, inspect sample data, and verify NULL handling.
Secret Mission: Built a Mapping Data Flow with Filter and Derived Column transformations that clean invalid rows and enrich data before loading to SQL.

Ready to quiz yourself?