Build a Data Pipeline with Azure Data Factory

Copy data between Azure Blob Storage containers using Azure Data Factory.

Introduction

30 Second Summary

Every app, website, and business tool you use generates data. Moving that data from one place to another reliably and on schedule is the core job of a data engineer.

In this project, you will build a complete data pipeline using Azure Data Factory that copies sales data between storage locations, runs it on demand, and then automates the whole process on a daily schedule.

What You'll Build

A data pipeline that copies a CSV file from one Azure Blob Storage container to another using Azure Data Factory's visual no-code interface.

By the end of this project, you'll have:

An Azure Resource Group, Storage Account, and Blob Containers organized for data engineering.
A working Azure Data Factory pipeline with a Copy Activity that moves data from a source container to a destination container.
A scheduled trigger that automates your pipeline to run daily without manual intervention.
Secret Mission: Add a Data Flow transformation that filters and enriches data before loading it to the destination.

Are there any prerequisites?

You need an Azure free account to complete this project. Everything runs in the browser through the Azure Portal, so there is nothing to install on your computer.

Before We Start

Before We Start...

Create a Resource Group and Storage Account

Before you can build a pipeline, you need somewhere to store your data and a way to keep your Azure resources organized.

In this step, you will create the foundational Azure infrastructure for your project. You will set up a Resource Group to hold everything together, a Storage Account to store files, and two Blob Containers that act as the source and destination for your pipeline.

In this step, get ready to:

Create a Resource Group to organize your project resources.
Set up a Storage Account with two Blob Containers.
Upload a sample sales data CSV file to use in your pipeline.

Create a Resource Group in the Azure Portal

A Resource Group is a container that holds related Azure resources together. Think of it like a folder on your computer that keeps all the files for one project in one place.

Open your browser and navigate to portal.azure.com.
Sign in with the Microsoft account you used to create your Azure free account.
In the search bar at the top of the page, type Resource groups and select Resource groups from the results.
Click Create.
In the Subscription dropdown, select your Azure subscription.
In the Resource group field, enter rg-data-pipeline-project.
In the Region dropdown, select a region close to you (e.g., East US).
Click Review + create, then click Create.

It takes a few seconds to create the resource group. You should see a notification confirming the deployment succeeded.

What is a Resource Group?

A Resource Group is Azure's way of organizing related resources. When you delete the resource group later, all resources inside it get deleted too.

This makes cleanup easy and keeps your Azure subscription tidy.

Don't see the Create button?

Make sure you are on the Resource groups page, not inside an existing resource group. The Create button appears in the top toolbar of the Resource groups list view.

help me find the Create button for Resource groups in the Azure Portal

Create a Storage Account

A Storage Account provides a unique namespace in Azure for your data. Everything you store in Azure Blob Storage lives inside a storage account.

In the portal search bar at the top, type Storage accounts and select Storage accounts from the results.
Click + Create.
In the Resource group dropdown, select rg-data-pipeline-project (the one you just created).
In the Storage account name field, enter a globally unique name. Type stpipelinedata followed by your initials and a number (e.g., stpipelinedatajd01).
In the Region dropdown, select the same region you chose for your resource group.
Leave Performance set to Standard and Redundancy set to Locally-redundant storage (LRS).
Click Review + create, then click Create.
Wait for the deployment to complete, then click Go to resource.

Why does the storage account name need to be unique?

Storage account names must be globally unique across all of Azure because they form part of a public URL. Names must be 3-24 characters, lowercase letters and numbers only.

Seeing a naming error?

If Azure says the name is already taken, try adding different numbers or extra initials. Remember that storage account names cannot contain uppercase letters, hyphens, or special characters.

help me choose a valid storage account name in the Azure Portal

Create two Blob Containers and upload sample data

Blob Containers are like folders inside your storage account. You need two containers: one for input data (the source) and one for output data (the destination).

On your storage account page, click Containers in the left menu under Data storage.
Click the + Container button.
Type input as the container name and click Create.
Click + Container again.
Type output as the container name and click Create.

You should now see both input and output containers listed on the page.

Next, you need a sample CSV file to upload. You will create this file on your Mac using TextEdit.

Press Cmd+Space to open Spotlight search.
Type TextEdit and press Enter to open it.
In TextEdit, click Format in the menu bar, then click Make Plain Text.
Paste the following sales data into the file:

OrderID,Product,Quantity,SalesAmount,Date
1001,Widget A,5,125.00,2026-01-15
1002,Widget B,3,89.50,2026-01-16
1003,Gadget C,10,450.00,2026-01-17
1004,Widget A,2,50.00,2026-01-18
1005,Gadget D,0,0.00,2026-01-19
1006,Widget B,7,210.00,2026-01-20
1007,Gadget C,1,45.00,2026-01-21
1008,Widget A,15,375.00,2026-01-22
1009,Gadget D,4,180.00,2026-01-23
1010,Widget B,0,-5.00,2026-01-24

What does this data contain?

This is a simple sales dataset with 10 orders. Each row records an order ID, product name, quantity sold, sale amount, and date.

Some rows intentionally have zero quantity or negative amounts. These will be useful later if you take on the Secret Mission to filter out bad data.

Press Cmd+S to save the file.
Name the file sales_data.csv and save it to your Desktop.
If TextEdit asks about the extension, click Use .csv to confirm.

Now upload the file to your input container.

Back in the Azure Portal, click the input container to open it.
Click the Upload button.
Click the folder icon to browse your local files.
Select the sales_data.csv file from your Desktop.
Click Upload to confirm.

You should see sales_data.csv listed inside the input container.

Don't see your file after uploading?

Click the Refresh button in the container view. Make sure you opened the input container (not output) before uploading.

help me upload a file to my Azure Blob Storage container

Your Azure infrastructure is ready. Next up, you will create an Azure Data Factory instance and explore the studio where you will build your pipeline.

Create an Azure Data Factory Instance

Your storage is ready with sales data waiting in the input container. Now you need a service to move that data from one place to another.

Azure Data Factory is Azure's fully managed data integration service. It lets you build visual pipelines that move and transform data without writing code. In this step, you'll create a Data Factory instance and open its studio interface.

In this step, get ready to:

Create an Azure Data Factory resource in your resource group.
Launch Azure Data Factory Studio in your browser.
Explore the ADF Studio interface and navigate to the Author section.

Create the Data Factory resource

A Data Factory resource is the container that holds all your pipelines, datasets, and connections. You need one before you can build anything.

In the Azure Portal search bar at the top, type Data factories and select it from the results.
Click Create.
Under Resource Group, select rg-data-pipeline-project from the dropdown.
Under Region, select the same region you used for your storage account (e.g., East US).
Under Name, enter a globally unique name like adf-pipeline-project- followed by your initials (e.g., adf-pipeline-project-jd).
Select the Git configuration tab at the top.
Check the Configure Git later checkbox.
Click Review + create.
Click Create.

Why configure Git later?

Git integration lets teams collaborate on pipelines using version control. For this project, you are working solo and do not need it. Skipping it keeps the setup simpler.

Deployment takes about 30 seconds. Wait for the notification that says deployment is complete.

Seeing a name validation error?

Data Factory names must be globally unique across all of Azure. If you see an error about the name being taken, try adding extra numbers or different initials to make it unique.

help me fix a name validation error when creating an Azure Data Factory

Launch Azure Data Factory Studio

Azure Data Factory Studio is the visual interface where you build and manage your pipelines. It runs entirely in your browser.

Once deployment completes, click Go to resource in the notification.
On the Data Factory overview page, click Launch Studio.

A new browser tab opens with Azure Data Factory Studio loaded.

What is ADF Studio?

ADF Studio is the dedicated workspace for building data pipelines visually. It is separate from the main Azure Portal. You will do all your pipeline authoring, monitoring, and management here.

Studio not loading?

ADF Studio requires Microsoft Edge or Google Chrome. If the page is stuck loading, make sure you are using one of these browsers and that pop-ups are not being blocked.

help me troubleshoot Azure Data Factory Studio not loading in my browser

Explore the ADF Studio interface

Before building your pipeline, take a moment to understand the ADF Studio layout. The left sidebar has four main icons that you will use throughout this project.

Look at the left sidebar. The Home icon shows the getting started page.
The Author icon (pencil) is where you build pipelines and data flows.
The Monitor icon (screen) shows your pipeline runs and their status.
The Manage icon (toolbox) is where you configure connections and triggers.
Click the Author icon (pencil) in the left sidebar to go to the pipeline authoring area.

What does the Author section do?

The Author section is your workspace for creating pipelines, datasets, and data flows. This is where you will spend most of your time in ADF Studio. The canvas lets you drag and drop activities to build pipelines visually.

Your Data Factory is ready and the studio is open. Next up, you will build your first pipeline that copies sales data from one container to another.

Build Your First Pipeline

Your Azure Data Factory Studio is open and ready to go. Now it's time to build the actual pipeline that moves your sales data from one container to another.

A pipeline in ADF is a sequence of activities that work together to move and transform data. You need to tell the pipeline where to find data (the source), where to put it (the sink), and how to connect to those locations. In this step, you'll wire all of that up using the visual canvas.

In this step, get ready to:

Create a pipeline with a Copy Data activity.
Configure a source dataset pointing to your input CSV file.
Configure a sink dataset pointing to your output container.

Create a new pipeline with a Copy Data activity

Every pipeline starts as an empty canvas. You'll add a Copy Data activity to it, which is the simplest way to move data from one place to another in ADF.

In the Author section, click the + (plus) button at the top of the Factory Resources panel.
Select Pipeline, then Pipeline again from the submenu.
A new pipeline tab opens. In the Properties panel on the right, rename it to pl_copy_sales_data.
In the Activities panel on the left side of the canvas, expand Move and transform.
Drag the Copy data activity onto the canvas.
Click on the activity on the canvas. In the General tab at the bottom, rename it to CopySalesCSV.

What is a Copy Data activity?

A Copy Activity is the workhorse of ADF. It reads data from a source and writes it to a destination (called a sink). It supports over 90 built-in connectors, so you can copy data between databases, cloud storage, APIs, and more.

You should see your CopySalesCSV activity sitting on the pipeline canvas as a blue rectangle.

Configure the source (where data comes from)

The source tells ADF which file to read. You'll create a Dataset that points to your CSV file in the input container, and a Linked Service that holds the connection details to your storage account.

Click on the CopySalesCSV activity on the canvas.
In the bottom panel, click the Source tab.
Click + New next to the Source dataset field to create a new dataset.
In the "New dataset" panel, search for Azure Blob Storage and select it.
Choose DelimitedText as the format (this is the CSV format).
Name the dataset ds_input_csv.

Now you need to create a Linked Service. This is the connection between ADF and your storage account.

Click New next to the Linked service dropdown.
Name the linked service ls_blob_storage.
Select your Azure subscription from the dropdown.
Select the storage account you created earlier (e.g., stpipelinedatajd01).
Click Test connection at the bottom to verify ADF can reach your storage.
Click Create.

Connection test failed?

Make sure you selected the correct subscription and storage account from the dropdowns. If the test still fails, check that your storage account is in the same region as your Data Factory.

Help me troubleshoot my linked service connection in Azure Data Factory.

With the linked service created, finish configuring the dataset.

In the file path fields, browse to the input container.
Select your sales_data.csv file.
Check the First row as header checkbox.
Click OK.

What are Linked Services and Datasets?

Think of a Linked Service as the login credentials for a data store. It knows the address and how to authenticate. A Dataset is like a pointer to a specific file or table within that data store.

You can reuse one Linked Service across many Datasets. This is exactly what you'll do in the next substep when you create the output dataset.

Your Source tab should now show ds_input_csv as the source dataset.

Configure the sink (where data goes)

The sink is the destination for your data. You'll point it at the output container so the copied file lands in a separate location from the original.

With the CopySalesCSV activity still selected, click the Sink tab in the bottom panel.
Click + New to create a new sink dataset.
Search for Azure Blob Storage and select it.
Choose DelimitedText as the format.
Name the dataset ds_output_csv.
Select the existing linked service ls_blob_storage from the dropdown (the one you just created).
Browse to the output container.
Enter sales_output.csv as the file name.
Check the First row as header checkbox.
Click OK.

Why reuse the same linked service?

Both your source and sink live in the same storage account. The linked service is the connection to that account, so you only need one. The datasets are what differ. One points to the input container and the other points to the output container.

Your Sink tab should now show ds_output_csv as the sink dataset. Your pipeline is fully configured and ready to run.

Don't see your container when browsing?

Make sure you selected the correct linked service (ls_blob_storage) from the dropdown. If the browse dialog is empty, the linked service might be pointing to the wrong storage account. Go back to the Manage section to check your linked service settings.

Help me fix my dataset configuration in Azure Data Factory.

Your pipeline is assembled and ready to go. Next up, you'll run it for the first time and watch your data move from the source container to the destination.

Run and Monitor Your Pipeline

Your pipeline is fully configured with a source dataset, a sink dataset, and a Copy Data activity connecting them. Now it is time to see it in action.

In this step, you will run your pipeline in test mode, confirm the data landed in the correct container, and publish your work to the live Azure Data Factory service.

In this step, get ready to:

Validate and debug your pipeline to copy data between containers.
Verify the copied file appeared in the output container.
Publish the pipeline and explore the Monitor view.

Validate and debug the pipeline

Before running a pipeline, you should check it for configuration errors. ADF Studio has a built-in validation feature that catches issues before any data moves.

Make sure your pipeline pl_copy_sales_data is open on the canvas in the Author section.
Click the Validate button in the pipeline toolbar (above the canvas).
Review the output panel at the bottom for any errors or warnings.
Fix any issues that appear before continuing.

What does Validate do?

Validate checks your pipeline configuration without actually running it. It catches missing dataset references, broken linked service connections, and incomplete activity settings before you use any compute resources.

Click the Debug button in the pipeline toolbar to run the pipeline immediately in test mode.
Watch the Output panel at the bottom of the canvas.
Wait for the run to complete. A green checkmark next to your activity means success.

What does Debug do?

Debug actually runs your pipeline end-to-end. For a Copy activity, it physically copies data from source to destination.

The key benefit is that you do not need to publish first. This lets you test changes iteratively without affecting the live service.

Seeing a red X instead of a green checkmark?

Click the error icon in the Output panel to see the failure details.
Check that your linked service ls_blob_storage connection is still valid by going to the Source tab and clicking Test connection.
Confirm that the file sales_data.csv still exists in your input container.

Why is my Debug run failing in ADF?

Verify the data was copied

A successful debug run means data should now exist in your destination container. Let's confirm it arrived.

Switch to your Azure Portal browser tab (the one showing your storage account).
Navigate to your storage account and click Containers in the left menu under Data storage.
Click on the output container.
You should see a file named sales_output.csv listed.
Click on sales_output.csv to verify the data matches your original sales data.

Don't see the file in the output container?

Click the Refresh button at the top of the container view. The portal does not auto-refresh.
Go back to ADF Studio and confirm the Debug run shows a green checkmark (not still in progress).
Check that the sink dataset ds_output_csv points to the output container and not a different container.

Why is my output container still empty after a successful pipeline run?

Publish your pipeline and check the Monitor view

Debug runs are temporary tests. To save your pipeline permanently to the live Data Factory service, you need to publish it.

Switch back to your ADF Studio browser tab.
Click Publish all in the top toolbar.
Review the list of pending changes (your pipeline, datasets, and linked service).
Click Publish to deploy everything to the live service.

What does publishing do?

Publishing saves your pipeline, datasets, and linked services to the live Azure Data Factory service. Until you publish, your work only exists in the current browser session.

Once published, your pipeline can be triggered by schedules, events, or manual runs outside of debug mode.

Click the Monitor icon (screen icon) in the left sidebar.
You will see your debug run listed with its status, start time, and duration.
Click the pipeline name pl_copy_sales_data to drill into the activity-level details.
You should see CopySalesCSV listed with details including rows read and rows written.

What does the Monitor view show?

The Monitor view displays every pipeline run with its Pipeline Name, Run Start, Duration, and Status (Succeeded, Failed, In Progress, or Canceled).

Clicking into a specific run shows activity-level details. For a Copy activity, this includes how many rows were read from the source and written to the sink.

Your pipeline is published and confirmed working. Next up, you will add a scheduled trigger to automate this pipeline so it runs on its own every day.

Add a Scheduled Trigger

In the last step, you ran your pipeline and confirmed it copies data successfully. However, clicking Debug every time you want data moved is not practical for real-world data engineering.

In this step, you will automate your pipeline by adding a schedule trigger. Triggers tell Azure Data Factory when to run your pipeline without any manual intervention.

In this step, get ready to:

Create a daily schedule trigger for your pipeline.
Verify the trigger fires in the Monitor section.
Stop the trigger to preserve your free credit.

Create a schedule trigger

A trigger is what makes your pipeline run automatically based on a condition you define. In this case, you will set up a time-based trigger that runs once per day.

In ADF Studio, click the Author icon (pencil) in the left sidebar.
Open your pipeline pl_copy_sales_data by clicking on it in the pipeline list.
Click Add trigger in the top toolbar.
Select New/Edit from the dropdown.
In the trigger panel that appears, click + New in the trigger dropdown.
Enter trg_daily_sales as the trigger name.
Under Type, select Schedule.
Set the recurrence to every 1 day.
Set the Start Date to today.
Set the start time to a few minutes from now.
Set End Date to No End.
Click OK.

What about the trigger run parameters?

After clicking OK, a second dialog may appear asking about trigger run parameters. Your pipeline does not use parameters, so click OK again to confirm.

Why a schedule trigger?

In production, data lands in storage at predictable intervals. A schedule trigger ensures your pipeline picks up new data without anyone pressing a button. This is how data engineers automate daily loads into data warehouses and analytics systems.

Publish and verify the trigger

Your trigger exists in draft mode until you publish it. Publishing deploys the trigger to the live Data Factory service so it can actually fire on schedule.

Click Publish all at the top of the ADF Studio canvas.
Review the pending changes and click Publish to confirm.
Click the Monitor icon (screen) in the left sidebar.
Click Trigger runs in the Monitor section.

If the scheduled time you set has already passed, you should see a new pipeline run listed that was triggered automatically. If it has not passed yet, wait a few minutes and refresh the page.

Don't see a trigger run yet?

Make sure the start time you set is in the past. If you set a time a few minutes ahead, wait for it to pass, then click the refresh button in the Monitor toolbar.

Also confirm you clicked Publish all after creating the trigger. Unpublished triggers do not fire.

help me figure out why my ADF trigger has not fired

Stop the trigger to avoid unnecessary runs

Your trigger is now live and will run your pipeline once every day. For this learning project, you do not need it running indefinitely. Stopping it preserves your free credit while keeping the pipeline intact for future use.

Click the Manage icon (toolbox) in the left sidebar.
Click Triggers in the left menu.
Find your trg_daily_sales trigger in the list and click on it.
Toggle the status to Stopped to deactivate it.
Click Publish all to save the deactivated state.

What happens when a trigger is stopped?

Your pipeline and trigger configuration remain intact. The trigger simply stops firing on its schedule. You can reactivate it at any time by toggling the status back on and publishing again.

Can't find the Triggers section?

Make sure you clicked the Manage icon (the toolbox) in the left sidebar, not the Author icon (pencil). Triggers appears under the Manage section.

help me find where to stop my trigger in ADF Studio

Your trigger is now safely deactivated. The pipeline still exists and can be triggered manually with Debug at any time.

Secret mission

Add a Data Flow Transformation

Your pipeline copies data perfectly. But what if you need to clean the data before it lands in the destination? In this secret mission, you will add a Mapping Data Flow that filters out bad rows and enriches the data with a new category column.

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely in the Azure Portal. Your trigger has already been stopped, so no automated runs will occur. However, leaving resources deployed indefinitely may consume your free credit over time.

Cost warning

Azure Data Factory charges $0.80 per month for each inactive pipeline. Your storage account also uses a small portion of your Azure free tier allocation.

If you are not planning to continue experimenting, choose Pause or Delete below to avoid unnecessary charges against your $200 free credit.

Resources you used:

Resource Group (rg-data-pipeline-project).
Storage Account (contains your input and output blob containers).
Azure Data Factory instance (contains your pipeline, datasets, linked service, and trigger).

Keep everything running

No action needed. Choose this if you are still actively experimenting with your pipeline or planning to add more activities.

Your trigger is already stopped, so no automated runs will occur.
The inactive pipeline charge is $0.80 per month. Storage costs are minimal.
Your $200 Azure free credit covers these idle costs comfortably for the 30-day free trial period.

Pause - I'll come back to this later

You have already stopped the trigger in Step 5. No further charges will accrue from automated pipeline runs.

Confirm your trigger is stopped by opening Azure Data Factory Studio, clicking the Manage icon (toolbox) in the left sidebar, and selecting Triggers.
Verify that trg_daily_sales shows a status of Stopped.
Your resources will remain in place. The inactive pipeline fee of $0.80 per month will still apply, but no data movement charges will occur.

Delete - I don't want to use this again

Remove all project resources and start fresh. Deleting the Resource Group removes everything inside it in one action.

Go to the Azure Portal at portal.azure.com.
Search for Resource groups in the top search bar.
Click on rg-data-pipeline-project.
Click Delete resource group at the top of the page.
Type the resource group name rg-data-pipeline-project to confirm.
Click Delete.

This removes the Resource Group, Storage Account, and Azure Data Factory instance. Your ongoing cost drops to zero.

Nice Work!

You just built a real data pipeline from scratch using Azure Data Factory. From creating cloud resources to automating data movement on a schedule, you now have hands-on experience with the foundational services every Azure data engineer uses daily.

You've learned how to:

Created Azure infrastructure from scratch including a Resource Group, Storage Account, and Blob Containers for input and output data.
Built a working pipeline with Linked Services, Datasets, and a Copy Activity that moves data between storage locations.
Automated your pipeline with a scheduled trigger and monitored pipeline runs in ADF Studio.
Added a Data Flow transformation to filter and enrich data before loading it to the destination.

Ready to quiz yourself?