Set Up a Production-Grade dbt Platform

Build a dbt CI/CD pipeline with BigQuery, automated tests, and Slim CI.

Introduction

30 Second Summary

Every data team dreads the morning a stakeholder messages about wrong numbers in a dashboard. The root cause is usually a code change that looked fine in development but quietly broke something in production.

In this project, you will set up a production-grade dbt platform connected to BigQuery. You will build models, layer on automated tests, deploy to production, and set up a pipeline that validates every code change before it reaches production data.

What You'll Build

You will open a pull request in GitHub and watch dbt Cloud automatically build and test only the models you changed.

By the end of this project, you'll have:

  • A dbt project where you run dbt build and watch raw CSV data flow through staging models into a clean, tested customers table in BigQuery.
  • Automated data tests that catch a real data quality bug before it reaches production, plus generated documentation with a lineage graph showing how every model connects.
  • A production deployment with a scheduled job and a CI pipeline where opening a pull request automatically triggers dbt Cloud to validate only the models you changed.
  • An optional Secret Mission to push your skills further.

Do I need any experience before starting?

You will need basic SQL knowledge (SELECT, JOIN, GROUP BY), a GitHub account, and a Google account. You will create your dbt Cloud account and BigQuery project during Step 1.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Committing to a clear goal now keeps the rest of the project focused.

Set Up BigQuery, dbt Cloud, and GitHub

Before you can write any dbt models, you need the three pillars of your platform connected. BigQuery stores your data. dbt Cloud transforms it. GitHub keeps your code under version control and powers CI/CD.

In this step, you will wire all three together so they can talk to each other. By the end, you will have a working development environment where you can write and run dbt code.

In this step, get ready to:
  • Create a Google Cloud project with a BigQuery service account.
  • Connect dbt Cloud to BigQuery and test the connection.
  • Link GitHub, initialize your dbt project, and push your first commit.
Set up BigQuery and create a service account

dbt needs a way to authenticate with your data warehouse. In Google Cloud, that means creating a project, then creating a service account with the right permissions and downloading its credentials as a JSON key file.

  • Open the Google Cloud Console and sign in with your Google account.
  • Click the project selector dropdown at the top of the page.
  • Click New Project.
  • Enter dbt-learn as the project name.
  • For Organization, select No organization if you are not part of one.
  • Click Create.

What is a GCP project?

A GCP project is a container that holds all your Google Cloud resources. Everything you create in BigQuery lives inside this project. The BigQuery API is automatically enabled for new projects.

Now create the service account that dbt Cloud will use to connect to BigQuery.

  • Open the GCP credentials wizard. Make sure dbt-learn is selected in the project dropdown at the top.
  • From the Select an API dropdown, choose BigQuery API.
  • Select Application data for the type of data you will be accessing.
  • Click Next to create a new service account.
  • Type dbt-user as the Service account name.
  • From the Select a role dropdown, choose BigQuery Job User.
  • Click Add another role and choose BigQuery Data Editor.
  • Click Continue.
  • Leave the Grant users access to this service account fields blank.
  • Click Done.

Why two roles?

BigQuery Job User lets dbt run queries (create jobs) in BigQuery. BigQuery Data Editor lets dbt create and write to tables and views. Together, they give dbt exactly the permissions it needs to build your models.

Next, create and download the JSON key file for this service account.

  • Open the Service accounts page in the Google Cloud Console.
  • Click on the dbt-user service account you just created.
  • Click the Keys tab.
  • Click Add Key, then select Create new key.
  • Select JSON as the key type and click Create.
  • A JSON file downloads to your computer. Save it somewhere you can find it easily.

Protect this file

This JSON key file works like a password. Do not commit it to version control or share it publicly. You will upload it to dbt Cloud in the next substep, and after that you will not need to share it anywhere else.

Create a dbt Cloud account and connect BigQuery

With your BigQuery credentials ready, you can now create your dbt Cloud account and connect it to your warehouse.

  • Go to cloud.getdbt.com and sign up for a free account. Select the Developer plan (free, no credit card required).
  • Click your account name in the left side menu.
  • Navigate to Account settings and click + New project.
  • Enter a project name (e.g. Jaffle Shop) and click Continue.
  • For the warehouse, click BigQuery, then click Next.
  • Click Upload a Service Account JSON File.
  • Select the JSON file you downloaded earlier. dbt Cloud fills in all the connection fields automatically.

What did uploading the JSON do?

The JSON file contains the service account's email, private key, and project ID. dbt Cloud reads these values and uses them to authenticate with BigQuery on your behalf. You do not need to enter any of these values manually.

Now set up your personal development credentials so dbt knows where to build your models during development.

  • Navigate to Your profile and select Credentials.
  • Select the project that uses the BigQuery connection.
  • Click Edit.
  • For Authentication Method, select Service Account JSON.
  • For Dataset, enter dbt_ followed by your first name (e.g. dbt_emma). This is the BigQuery dataset where your development models will be built.
  • Leave Target name and Threads as the defaults.
  • Click Test Connection.

✔️ Connection test succeeded

Your BigQuery connection is working. Click Next to continue.

ⓧ Connection test failed

Go back to the BigQuery credentials wizard and confirm you assigned both roles to the service account. Then create a new JSON key and re-upload it in dbt Cloud.

If the issue persists, help me debug my BigQuery connection in dbt Cloud.

Why a personal development dataset?

Each developer on a dbt team gets their own dataset (like dbt_emma). This keeps your in-progress work separate from production data and from other developers' work. You will set up a shared production dataset later in Step 4.

Connect GitHub and initialize the project

Your dbt project needs a Git repository so you can track changes, collaborate, and trigger CI/CD pipelines. You will create a GitHub repository, link it to dbt Cloud, and initialize the project scaffold.

  • Open GitHub in a new tab and sign in with your existing account.
  • Click the + icon in the top right and select New repository.
  • Name the repository (e.g. dbt-learn-jaffle-shop).
  • Leave it as Public and click Create repository. Do not add a README or .gitignore yet. dbt Cloud will initialize the repo for you.

Now link your GitHub account to dbt Cloud so it can access your repository.

  • Switch back to dbt Cloud.
  • Click your account name in the left side menu and select Account settings.
  • Select Personal profile under the Your profile section.
  • Scroll down to Linked accounts.
  • Click Link to the right of GitHub. This redirects you to GitHub.
  • On the GitHub page, select the repositories dbt Cloud should access (or grant access to all repositories).
  • Click Install & Authorize. You are redirected back to dbt Cloud with a linked account confirmation.

Seeing an 'account in use' error?

This means your GitHub account is already linked to another dbt Cloud account. You can only link one GitHub account to one dbt Cloud account at a time. Help me fix my GitHub linking issue in dbt Cloud.

Now connect your GitHub repository to the dbt Cloud project and initialize the project scaffold.

  • In dbt Cloud, navigate to Account settings and select your project.
  • Under Repository, click Configure Repository.
  • Select GitHub and choose the repository you just created.
  • Click Start developing in the Studio IDE. It may take a moment for the IDE to spin up as it connects to your Git repo and tests the warehouse connection.
  • Above the file tree on the left, click Initialize dbt project. This creates the folder structure with example models.
  • Click Commit and sync.
  • Enter the commit message initial commit and click Commit.

What did initializing create?

The Initialize dbt project button scaffolds your project with the standard dbt folder structure: models/, seeds/, tests/, macros/, and a dbt_project.yml configuration file. It also includes example models you will replace in the next step.

Before you move on, verify that dbt can reach BigQuery from the Studio IDE.

What do you think will happen when you run dbt debug in the command bar? Do you expect any errors, or a clean pass?

  • Type the following command into the command bar at the bottom of the Studio IDE and press Enter:
dbt debug

You should see output ending with Connection test: OK connection ok. This confirms your project is connected to BigQuery and ready to build models.

Connection test failed?

Double check that your development credentials dataset is set correctly under Your profile > Credentials. The dataset name should start with dbt_ followed by your name.

Also confirm the JSON key you uploaded matches the dbt-user service account in the dbt-learn GCP project. Help me debug my dbt debug connection failure.

Your platform foundation is in place. BigQuery, dbt Cloud, and GitHub are connected and talking to each other. Next up, you will build your first models and see data flow through the pipeline.

Build and Run Your First Models

Your dbt Cloud project is connected to BigQuery and GitHub, and the Studio IDE is ready to go.

An empty dbt project proves nothing to an interviewer. You need real models that transform data. In this step, you will build a small analytics pipeline that follows the same staging-to-marts pattern used in production dbt projects.

In this step, get ready to:
  • Load raw sample data into BigQuery using dbt seeds.
  • Create staging models that clean and rename raw columns.
  • Build a customers mart that joins staging models into one table.
Delete the example models and add seed data

The initialized dbt project includes a models/example/ directory with sample models you do not need. Removing it keeps the project clean before you add your own models.

  • In the Studio IDE File explorer on the left, right-click the models/example/ directory.
  • Select Delete to remove the entire directory.

The models/example/ folder should disappear from the file tree.

  • Open dbt_project.yml from the File explorer.
  • Scroll to the bottom and find the models: section. It currently looks like this:
models:
  jaffle_shop:
    example:
      +materialized: view
  • Delete the two lines example: and +materialized: view so the section looks like this:
models:
  jaffle_shop:
  • Save dbt_project.yml by pressing Cmd+S (macOS) or Ctrl+S (Windows).

Now you will create three CSV seed files that contain sample data for the Jaffle Shop. dbt loads these CSVs directly into your warehouse as tables.

  • Create a new file at seeds/raw_customers.csv and paste the following data:
id,first_name,last_name
1,Michael,P.
2,Shawn,M.
3,Kathleen,P.
4,Jimmy,C.
5,Katherine,R.
  • Save raw_customers.csv.
  • Create a new file at seeds/raw_orders.csv and paste the following data:
id,user_id,order_date,status
1,1,2018-01-01,returned
2,3,2018-01-02,completed
3,4,2018-01-04,completed
4,1,2018-01-05,completed
5,5,2018-01-10,completed
6,3,2018-01-11,completed
7,4,2018-01-12,completed
8,1,2018-02-01,returned
9,2,2018-02-10,completed
  • Save raw_orders.csv. You should see both CSV files in the seeds/ folder in the File explorer.
  • Create a new file at seeds/raw_payments.csv and paste the following data:
id,order_id,payment_method,amount
1,1,credit_card,1000
2,2,credit_card,2000
3,3,coupon,100
4,4,coupon,2500
5,5,bank_transfer,1700
6,6,credit_card,500
7,7,credit_card,1000
8,7,credit_card,200
9,8,credit_card,300
10,9,bank_transfer,1500
11,9,credit_card,500
12,1,bank_transfer,500
  • Save raw_payments.csv.

Load the seed data into BigQuery by running this command in the command bar at the bottom of the Studio IDE:

dbt seed

You should see output showing all three seeds loaded successfully, with row counts for each (5 customers, 9 orders, 12 payments).

What are seeds?

Seeds are CSV files in your dbt project that get loaded into your data warehouse as tables. They are useful for small, static reference data like lookup tables or sample data. The dbt seed command reads each CSV and creates a corresponding table in your development dataset.

Seeds failing to load?

Make sure you saved all three CSV files before running dbt seed. Unsaved files will not be included in the dbt invocation.

Check that each CSV file is inside the seeds/ directory, not the project root. The file path should be seeds/raw_customers.csv, not raw_customers.csv.

Help me debug my dbt seed failure.

Create staging models

Staging models sit between raw data and business logic. Each one selects from a single seed, renames columns to clean names, and uses the ref function so dbt knows the dependency.

  • Create a new folder models/staging/ and add a file stg_customers.sql inside it with the following code:
select
    id as customer_id,
    first_name,
    last_name
from {{ ref('raw_customers') }}
  • Save stg_customers.sql.
  • Create models/staging/stg_orders.sql with this code:
select
    id as order_id,
    user_id as customer_id,
    order_date,
    status
from {{ ref('raw_orders') }}
  • Save stg_orders.sql. You should see both files listed in the models/staging/ folder in the File explorer.
  • Create models/staging/stg_payments.sql with this code:
select
    id as payment_id,
    order_id,
    payment_method,
    amount / 100 as amount
from {{ ref('raw_payments') }}
  • Save stg_payments.sql.

What does ref() do?

The {{ ref('raw_customers') }} function tells dbt that this model depends on the raw_customers seed. dbt uses these references to build a DAG (directed acyclic graph) that determines the correct order to build models. It also means dbt automatically resolves the full table name, so your SQL never contains hardcoded schema or project names.

Build the staging models by running this command in the command bar:

dbt run

You should see all three staging models build successfully as views in your development dataset.

Models failing to compile?

Check that the filenames inside models/staging/ match exactly: stg_customers.sql, stg_orders.sql, stg_payments.sql. Typos in filenames cause "node not found" errors.

If you see a "relation does not exist" error, make sure you ran dbt seed successfully first. The staging models depend on the seed tables existing in BigQuery.

Help me fix my staging model errors.

Create a customers mart and run the project

A mart model combines multiple staging models into a table ready for analysis. Your customers mart will join customers, orders, and payments into a single flat table.

  • Create a new folder models/marts/ and add a file customers.sql inside it with the following code:
-- Joins customers with their orders and payments
select
    customers.customer_id,
    customers.first_name,
    customers.last_name,
    orders.order_date,
    orders.status,
    payments.amount
from {{ ref('stg_customers') }} as customers
left join {{ ref('stg_orders') }} as orders
    on customers.customer_id = orders.customer_id
left join {{ ref('stg_payments') }} as payments
    on orders.order_id = payments.order_id
  • Save customers.sql.

What does this model do?

This model creates a flat detail table by joining all three staging models together. Each row represents a customer-order-payment combination. The left join ensures that all customers appear even if they have no orders, and all orders appear even if they have no payments.

This flat join pattern is a common starting point. You will discover a problem with it in the next step when you add tests.

Now update the materialization settings. Staging models should be lightweight views, while mart models should be fully materialized tables for faster queries.

  • Return to dbt_project.yml and find the models: section at the bottom. It currently looks like this:
models:
  jaffle_shop:
  • Replace it with the following configuration that sets materializations for each folder:
models:
  jaffle_shop:
    staging:
      +materialized: view
    marts:
      +materialized: table

What are materializations?

A materialization tells dbt how to persist a model in your warehouse. A view is a saved query that runs every time it is queried. A table is a physical copy of the data that is rebuilt when you run dbt build.

Staging models are views because they are lightweight transformations. Mart models are tables because downstream consumers need fast, reliable reads.

  • Save dbt_project.yml.

✔️ Awesome, I've got everything!

Great. Make sure you saved dbt_project.yml before running the next command.

ⓧ I'd like to double check the full code

Your dbt_project.yml should look like this:

name: 'jaffle_shop'
version: '1.0.0'

profile: 'jaffle_shop'

model-paths: ["models"]
seed-paths: ["seeds"]
test-paths: ["tests"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
analysis-paths: ["analyses"]

clean-targets:
  - "target"
  - "dbt_packages"

models:
  jaffle_shop:
    staging:
      +materialized: view
    marts:
      +materialized: table

Before you run the full build, predict: what do you expect to happen when dbt encounters the customers model in the marts/ folder? Will it be a view or a table?

  • Run the full project by entering this command in the command bar:
dbt build

All seeds reload and all models build successfully. The customers model builds as a table (not a view) because of the +materialized: table setting you just configured.

  • Open models/marts/customers.sql in the editor.
  • Click the Preview button in the console section below the editor.

The Results tab displays rows with columns: customer_id, first_name, last_name, order_date, status, and amount. You have a working analytics pipeline.

Preview showing no results?

Make sure customers.sql is the active file open in the editor before clicking Preview. The Preview button runs the SQL from the currently open file.

If you see a compilation error, check that your staging model filenames match the ref() calls exactly: stg_customers, stg_orders, stg_payments.

Help me debug my customers model preview.

  • Commit your changes by clicking the Git actions button under Version Control in the left panel.
  • Enter the commit message add seeds and models and commit.

Committing in the Studio IDE

The Studio IDE protects the main branch. When you commit, it prompts you to commit to a new branch. Follow the prompts to commit and merge your changes back into main. This keeps your repository in sync for later steps.

Your models are building and you can see real data flowing through the pipeline. Next up, you will add tests to catch data quality issues and discover that this flat join has a hidden bug.

Add Tests and Fix Your First Data Quality Bug

In the previous step, your models built and ran without a single error. That clean build might feel like a green light, but all it proves is that the SQL executed. It says nothing about whether the data inside those models is correct.

Your customers mart currently joins customers, orders, and payments into a flat table. Some orders have multiple payments, so that join silently produces duplicate customer rows. In this step, you will add automated tests that catch this bug before a stakeholder does.

In this step, get ready to:
  • Add schema tests for staging and mart models.
  • Discover and fix a data quality bug in the customers mart.
  • Generate documentation with a lineage graph.
Add schema tests for staging models

dbt ships with four built-in data tests: unique, not_null, accepted_values, and relationships. You declare them in a YAML file alongside your models, and dbt runs them as SQL queries that look for failing rows.

  • Create a new file called schema.yml inside the models/staging/ directory.
  • Add the following code to models/staging/schema.yml:
version: 2

models:
  - name: stg_customers
    columns:
      - name: customer_id
        data_tests:
          - unique
          - not_null

  - name: stg_orders
    columns:
      - name: order_id
        data_tests:
          - unique
          - not_null
      - name: status
        data_tests:
          - accepted_values:
              values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']

What does this YAML do?

  • unique checks that no duplicate values exist in a column. If any value appears more than once, the test fails.
  • not_null checks that every row has a value in the column. No blanks allowed.
  • accepted_values checks that every value in the status column is one of the values in the list. If an order has a status like cancelled (not in the list), the test fails.
  • Save models/staging/schema.yml.
  • Run your staging tests by entering this command in the command bar at the bottom of the Studio IDE:
dbt test

You should see all 5 staging tests pass, with a summary showing PASS=5 WARN=0 ERROR=0.

Tests not passing?

  • Make sure you saved schema.yml before running the command. Unsaved changes in the IDE are not picked up by dbt.
  • Check that the file is inside models/staging/ and not in the project root.
  • Verify the YAML indentation is correct. YAML is whitespace-sensitive, so each nested level needs exactly two spaces.

Still stuck? Why are my dbt staging tests failing?

✔️ Awesome, I've got everything!

Great. Make sure you have saved models/staging/schema.yml.

ⓧ I'd like to double check the full code

version: 2

models:
  - name: stg_customers
    columns:
      - name: customer_id
        data_tests:
          - unique
          - not_null

  - name: stg_orders
    columns:
      - name: order_id
        data_tests:
          - unique
          - not_null
      - name: status
        data_tests:
          - accepted_values:
              values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']
Add schema tests for the customers mart and watch one fail

Your staging models are clean. Now add the same kind of guardrails to the customers mart. The customer_id column should be unique (one row per customer) and never null.

  • Create a new file called schema.yml inside the models/marts/ directory.
  • Add the following code to models/marts/schema.yml:
version: 2

models:
  - name: customers
    description: One record per customer with their order history and lifetime value.
    columns:
      - name: customer_id
        description: Primary key for customers.
        data_tests:
          - unique
          - not_null
      - name: first_order_date
        description: Date of the customer's first order.
      - name: most_recent_order_date
        description: Date of the customer's most recent order.
      - name: number_of_orders
        description: Count of orders placed by this customer.
      - name: lifetime_value
        description: Total amount spent by this customer across all orders.

What does this schema file add?

The unique and not_null tests on customer_id assert that the customers mart has exactly one row per customer with no gaps. The description fields document what each column means. dbt uses these descriptions to generate searchable documentation later.

  • Save models/marts/schema.yml.

Before you run the tests, think about what you expect to happen. Your staging tests all passed. Do you think the unique test on customer_id will pass too?

  • Run all tests by entering this command in the command bar:
dbt test

You will see an error. The unique test on customers.customer_id fails. The output shows something like Failure in test unique_customers_customer_id and reports ERROR=1 in the summary.

This failure is expected. It is exactly the bug you are here to find.

Why did the unique test fail?

The customers model currently joins stg_customers, stg_orders, and stg_payments in a flat table. When a single order has multiple payments, the join creates a separate row for each payment. That means a customer with three orders (one of which has two payments) ends up with four rows instead of one.

The model built successfully because the SQL is valid. But the data is wrong. Without the unique test, this silent duplication would have gone unnoticed until a stakeholder questioned why customer counts look inflated.

What is a fan-out join?

A fan-out join happens when you join two tables and one side has multiple rows matching a single row on the other side. Each match multiplies the result. In this project, order 7 has two payments (IDs 7 and 8), so the join produces two rows for that single order.

Fix the customers model and generate documentation

The fix is to aggregate before joining. Instead of a flat table that repeats customer rows, you will build separate CTEs that pre-calculate order counts and payment totals per customer. Then you join the summaries back to the customer list.

  • Open models/marts/customers.sql in the Studio IDE.
  • Delete all of the existing code in the file.
  • Paste the following code, which defines the source CTEs and the two aggregation CTEs:
with customers as (
    select * from {{ ref('stg_customers') }}
),

orders as (
    select * from {{ ref('stg_orders') }}
),

-- Aggregate order-level stats per customer
customer_orders as (
    select
        customer_id,
        min(order_date) as first_order_date,
        max(order_date) as most_recent_order_date,
        count(order_id) as number_of_orders
    from orders
    group by customer_id
),

-- Aggregate payment totals per customer
customer_payments as (
    select
        orders.customer_id,
        sum(payments.amount) as lifetime_value
    from orders
    left join {{ ref('stg_payments') }} as payments
        on orders.order_id = payments.order_id
    group by orders.customer_id
),

What do these CTEs do?

  • customer_orders groups orders by customer_id and calculates three things: the first order date, the most recent order date, and the total number of orders. The group by guarantees one row per customer.
  • customer_payments joins orders to payments and sums the amounts by customer_id. This handles the fan-out safely because the sum collapses multiple payment rows into a single total.
  • Add the final CTE and select statement directly below the code you just pasted:
-- Join the summaries back to the customer list
final as (
    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        coalesce(customer_orders.number_of_orders, 0) as number_of_orders,
        coalesce(customer_payments.lifetime_value, 0) as lifetime_value
    from customers
    left join customer_orders
        on customers.customer_id = customer_orders.customer_id
    left join customer_payments
        on customers.customer_id = customer_payments.customer_id
)

select * from final

What does the final CTE do?

The final CTE joins customers to the two pre-aggregated CTEs. Because each aggregation CTE already has one row per customer, the join can no longer fan out. The coalesce function fills in 0 for customers who have no orders or payments.

  • Save models/marts/customers.sql.

Now think about what will happen when you re-run the tests. The unique test failed before because of duplicate customer rows. With the aggregation fix, do you think it will pass this time?

  • Run the full build (seeds, models, and tests) by entering this command in the command bar:
dbt build

All tests pass. The summary shows ERROR=0 with every test marked PASS. The customers model now has exactly one row per customer.

Still seeing a test failure?

  • Make sure you deleted all of the old code in customers.sql before pasting the new version. Leftover code from the flat join can cause syntax errors or unexpected results.
  • Check that the trailing comma after the customer_payments CTE is present. Missing it causes a SQL syntax error.
  • Verify that group by customer_id appears in both the customer_orders and customer_payments CTEs.

Still stuck? Help me fix my refactored customers.sql model so all dbt tests pass.

✔️ Awesome, I've got everything!

Great. Make sure you have saved models/marts/customers.sql.

ⓧ I'd like to double check the full code

Scroll up and confirm your models/marts/customers.sql contains both chunks in order:

  • The first chunk starting with with customers as ( and ending with group by orders.customer_id followed by ), (trailing comma).
  • The second chunk starting with final as ( and ending with select * from final.

Make sure there are no leftover lines from the old flat join version.

Now generate the project documentation so you can see how all your models connect.

  • Generate documentation by entering this command in the command bar:
dbt docs generate
  • After the command completes, click the book icon above the file tree in the Studio IDE to open the documentation site in a new browser tab.
  • Click the green button in the bottom-right corner of the documentation page to expand the lineage graph.

You should see your three seed files (raw_customers, raw_orders, raw_payments) flowing into the staging models, which then feed into the customers mart. This is your project's DAG.

What is a lineage graph?

A lineage graph (also called a DAG) shows how data flows through your project from raw sources on the left to final outputs on the right. Every {{ ref() }} call in your SQL creates a connection in this graph. In an interview, walking someone through this lineage is a quick way to demonstrate you understand how your models fit together.

  • Commit your changes with the message add tests, fix customers model, generate docs.

Your models are tested, documented, and producing clean data. Next up, you will deploy everything to a production environment so downstream consumers can access it.

Deploy to Production

In the last step, you caught a real data quality bug and fixed it with automated tests. Your models are clean and reliable, but they live in your personal development dataset in BigQuery.

Nobody else can see or query your models yet. In this step, you'll create a Production environment in dbt Cloud and deploy your models to a shared analytics dataset that downstream consumers can query.

In this step, get ready to:
  • Create a Production deployment environment in dbt Cloud.
  • Create and run a deploy job that builds all models in the production dataset.
Create a Production deployment environment

In dbt Cloud, environments separate where your models are built. Your development environment writes to a personal dataset for testing, but a deployment environment writes to a shared dataset that other tools and team members can query.

  • In dbt Cloud, click Orchestration in the left sidebar.
  • Select Environments.
  • Click Create Environment.
  • Enter Production in the Name field.
  • Set the Environment type to Deployment.
  • Check the box to designate this environment as Production.
  • Under Deployment connection, enter analytics in the Dataset field.
  • Click Save.

You should see the Production environment listed on the Environments page with a Deployment badge.

Why a separate dataset for production?

Your development dataset (dbt_<your_name>) is your personal sandbox for experimenting. The analytics dataset is a shared production target that dashboards, reports, and other team members can rely on.

Keeping them separate means you can break things in development without affecting anyone downstream.

Don't see the Deployment connection section?

Make sure you selected Deployment as the environment type. Development environments do not show a deployment connection section.

Still stuck? Help me troubleshoot why I can't see the Deployment connection section when creating a dbt Cloud environment.

Create and run a deploy job

A deploy job tells dbt Cloud what to run and when. You'll create one that builds all your models in the Production environment so the analytics dataset stays up to date.

  • On the Production environment page, click Create job.
  • Select Deploy job.
  • Enter Daily production run in the Name field.
  • Confirm the Command is set to dbt build.
  • Check the box for Generate docs on run.
  • Click Save.

What does Generate docs on run do?

When enabled, the deploy job regenerates your project documentation every time it runs. This keeps your docs and lineage graph in sync with whatever is currently in production.

  • Click Run now to trigger the job manually.

The job runs each seed, model, and test in sequence. Watch the run progress in the dbt Cloud interface.

All steps should complete with green checkmarks, confirming that your seeds loaded, models built, and tests passed in the production environment.

Seeing a failed run?

Click the failed step in the run details to see the error message. Common causes include a misconfigured dataset name in the Production environment or missing BigQuery permissions on your service account.

Double check that the Dataset field under Deployment connection is set to analytics and that your service account has the BigQuery Data Editor role.

Help me debug a failed dbt Cloud deploy job run.

Before you check BigQuery, what do you expect to find in the analytics dataset? Think about how many objects should be there and whether each one is a view or a table.

  • Switch to the BigQuery Console in your browser.
  • In the Explorer panel on the left, expand your dbt-learn project.
  • Look for the analytics dataset.

You should see four objects inside the analytics dataset: stg_customers, stg_orders, and stg_payments as views, and customers as a table. The staging models are views because your dbt_project.yml sets +materialized: view for staging, and +materialized: table for marts.

Your models are now running in production. Next up, you'll set up continuous integration so every pull request is automatically validated before it can touch production data.

Set Up Continuous Integration

Your production environment is live. The daily deploy job builds tested models on a schedule into the analytics dataset.

But what happens when someone pushes broken SQL to the main branch? That broken code gets built into production on the next scheduled run. Continuous Integration solves this by validating every code change before it reaches production. In this step, you'll set up a CI pipeline in dbt Cloud that automatically builds and tests only the models you changed whenever you open a pull request.

In this step, get ready to:
  • Create a CI environment and CI job in dbt Cloud.
  • Make a code change on a new branch.
  • Open a pull request and watch dbt Cloud validate the change automatically.
Create a CI environment and job

CI runs need their own deployment environment so they stay isolated from production builds. CI runs write to temporary schemas that are automatically cleaned up when the pull request is merged.

  • In dbt Cloud, navigate to Orchestration > Environments.
  • Click Create Environment.
  • Enter CI as the environment name.
  • Set the environment type to Deployment.
  • Leave the Set as Production environment checkbox unchecked. This environment is for CI only.
  • Click Save.

You should see the CI environment appear in your Environments list.

Now create the CI job inside this environment.

  • On the CI environment page, click Create job > Continuous integration job.
  • Enter CI Check as the job name.
  • The default command dbt build --select state:modified+ is already filled in. Leave it as is.
  • Under Compare changes against, confirm that the Production environment is selected.
  • Click Save.

What does state:modified+ do?

This selector tells dbt to build only the models that have changed compared to the last successful production build, plus their downstream dependents. The + means "and everything downstream."

This approach is called Slim CI because it avoids rebuilding the entire project on every pull request. In a large project with hundreds of models, this saves significant time and compute.

Don't see Continuous integration job?

Make sure you're on the CI environment page, not the Production environment. Navigate to Orchestration > Environments, click CI, then click Create job.

Help me find the CI job creation option in dbt Cloud.

Make a code change on a new branch

To trigger the CI pipeline, you need a code change on a branch with a pull request. You'll add a new column to the customers mart that flags repeat customers.

  • Switch back to the Studio IDE in dbt Cloud.
  • In the Version Control section on the left sidebar, click Create branch.
  • Enter update-customers-model as the branch name and create the branch.
  • Open models/marts/customers.sql from the file tree on the left.
  • In customers.sql, scroll down to the final CTE.
  • Find this line:
        coalesce(customer_payments.lifetime_value, 0) as lifetime_value
  • Add a comma after lifetime_value, then add the following lines directly below it so the section looks like this:
        coalesce(customer_payments.lifetime_value, 0) as lifetime_value,
        case
            when coalesce(customer_orders.number_of_orders, 0) > 1 then true
            else false
        end as is_repeat_customer

What does this column do?

The is_repeat_customer column uses a case expression to check if a customer has placed more than one order. If number_of_orders is greater than 1, it returns true. Otherwise, it returns false.

  • Save customers.sql.
  • Verify the change works by running this command in the Studio IDE command bar at the bottom of the screen:
dbt build --select customers

You should see the customers model build successfully and its tests pass.

  • In the Version Control section on the left sidebar, click Commit and sync.
  • Enter a commit message like Add is_repeat_customer column.
  • Click Commit Changes.

Build failing after your edit?

Check that you added a comma after lifetime_value on the line above the case statement. A missing comma is the most common cause of SQL syntax errors here.

Help me fix my customers.sql after adding the is_repeat_customer column.

Open a pull request and watch CI run

With your code committed on the update-customers-model branch, opening a pull request in GitHub triggers the CI job automatically.

  • Go to your GitHub repository in a new browser tab.
  • You should see a banner suggesting you create a pull request for the update-customers-model branch. Click Compare & pull request.
  • Enter a title for the pull request, such as Add repeat customer flag.
  • Click Create pull request.

Before you check the result, think about this: will the CI job build all four of your models, or just one? Why?

  • Wait a minute or two for dbt Cloud to detect the pull request and trigger the CI job.
  • Look at the Checks section at the bottom of your pull request page in GitHub.

You should see a dbt Cloud CI check appear. It runs dbt build --select state:modified+, which means it only builds the customers model (the one you changed) and runs its tests. It does not rebuild the staging models or reload seeds because those were not modified.

Why didn't CI build everything?

The state:modified+ selector compares your branch against the last successful production build. Only customers.sql changed, so only that model and its downstream dependents get built.

This is what makes Slim CI fast. In a project with hundreds of models, rebuilding everything on every pull request would be wasteful. state:modified+ narrows the scope to just the models that matter for this change.

Where does CI run write its results?

CI runs write to temporary schemas named dbt_cloud_pr_<job_id>_<pr_id>. These schemas are automatically cleaned up when the pull request is merged. CI runs also do not consume your account's run slots, so they never block production jobs.

CI check not appearing in GitHub?

The CI job can take a minute or two to start after you create the pull request. Refresh the GitHub pull request page. If the check still does not appear, return to dbt Cloud and navigate to the CI Check job to see if a run was triggered.

Help me get the dbt Cloud CI check to appear in my GitHub pull request.

Your CI pipeline is live. Every pull request now gets automatically validated before it can reach production. Ready for an optional challenge to push your platform even further?

Secret mission

Add Continuous Deployment with a Merge Job

Your CI pipeline validates changes before they reach production. But after merging a PR, someone still has to manually trigger a production build. Can you close this gap with a merge job that automatically deploys only the changed models the moment a PR is merged?

Clean Up Your Resources

Clean Up Your Resources

This project uses entirely free-tier services, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • dbt Cloud project on the Developer plan.
  • GCP project dbt-learn with BigQuery datasets and a dbt-user service account.
  • GitHub repository connected to the dbt Cloud project.
  • Service account JSON key file stored on your computer.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project further.

  • All three services (dbt Cloud, BigQuery, GitHub) are on free tiers. No charges will accrue for this project.
  • Your Daily production run deploy job will continue running on its set schedule (if you configured one).
  • CI and merge jobs will continue triggering automatically when you open or merge pull requests.
  • The service account JSON key file remains on your computer. Store it securely or delete it if you no longer need local access.

Pause - I'll come back to this later

Disable automated jobs so nothing runs while you are away. Your project, data, and configurations remain intact.

  • In dbt Cloud, go to Orchestration and open the Daily production run deploy job.
  • Click Edit and remove or disable the schedule. Save the job.
  • Your CI and merge jobs only trigger on pull request and merge events. They will stay idle until you open a new PR.
  • Your dbt Cloud project, BigQuery datasets, and GitHub repository remain fully intact and ready to resume.

Delete - I don't want to use this again

Remove all project resources permanently. This cannot be undone.

  • In dbt Cloud, go to Account Settings and select Projects. Delete your dbt Cloud project. This removes all environments, jobs, and run history.
  • In the Google Cloud Console, open the dbt-learn project. Go to IAM & Admin, then Settings, and shut down the project. This removes all BigQuery datasets and the dbt-user service account.
  • In your GitHub repository, go to Settings. Scroll to the bottom of the page and delete the repository.
  • Delete the service account JSON key file from your computer. This file contains credentials that are no longer needed once the GCP project is deleted.

Nice Work!

Nice Work!

You just built a production-grade dbt platform from scratch, complete with automated testing, documentation, deployment, and CI/CD. This is the same infrastructure that data teams rely on every day to keep production data trustworthy.

You've learned how to:

  • Build a dbt project with staging and mart models, using the ref function to connect models and materializations to control how they are built in BigQuery.
  • Catch a real data quality bug (duplicate rows from a fan-out join) using dbt's built-in testing framework, then generate documentation with a lineage graph showing how your models connect.
  • Deploy models to a production environment with a scheduled job and set up a CI pipeline with Slim CI that automatically validates every pull request using state:modified+.
  • Completed an optional Secret Mission to push your CI/CD pipeline even further.

Ready to quiz yourself?