CI/CD for a dbt + BigQuery Data Pipeline with GitHub Actions

Build and test your dbt models against BigQuery on every pull request.

A dbt pipeline on BigQuery compiles models into SQL and materializes them in the warehouse. This recipe authenticates to GCP, builds models into an isolated CI dataset, and gates on dbt tests.

What the pipeline does

• install dbt-bigquery
• authenticate to GCP with Workload Identity Federation
• compile and build models into a CI dataset with dbt build
• run dbt tests
• fail the PR on any failed model or test

The workflow

google-github-actions/auth with Workload Identity Federation avoids long-lived keys. dbt build runs models and their tests together into a PR-scoped dataset so CI never touches production tables.

name: CI
on:
  pull_request:
permissions:
  contents: read
  id-token: write
jobs:
  dbt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
      - run: pip install dbt-bigquery
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_SA_EMAIL }}
      - run: dbt deps
      - run: dbt build --target ci --vars '{dataset: ci_pr_${{ github.event.number }}}'

Caching and speed

cache: pip restores the dbt install, and dbt state comparison (--defer with a stored manifest) can build only changed models. Warehouse round-trips dominate wall-clock time; cheaper managed runners such as Latchkey keep frequent PR runs affordable and auto-retry transient BigQuery API blips.

Deploying

On merge to main, run dbt build against the production target on a schedule or trigger. Use slim CI (state:modified+ with --defer) to build only changed models, and drop the per-PR CI dataset after the run to control warehouse cost.

Key takeaways

• Build into a per-PR dataset so CI never touches production.
• Authenticate with Workload Identity Federation, not static keys.
• Use dbt state/--defer for slim CI on large projects.

Related guides