How to Deploy a Dataflow Job With GitHub Actions

gcloud dataflow flex-template run launches a Dataflow pipeline from a built template, parameterized at runtime.

Authenticate via WIF, then run gcloud dataflow flex-template run pointing at the template spec in GCS, with --parameters for inputs and outputs.

Steps

• Authenticate via WIF and set up gcloud.
• Reference a flex-template spec in GCS.
• Run flex-template run with --parameters.

Workflow

permissions:
  id-token: write
  contents: read
jobs:
  launch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.DEPLOY_SA }}
      - uses: google-github-actions/setup-gcloud@v2
      - run: |
          gcloud dataflow flex-template run "etl-${{ github.run_id }}" \
            --template-file-gcs-location gs://my-templates/etl.json \
            --region us-central1 \
            --parameters inputTable=ds.in,outputTable=ds.out

Gotchas

• The worker service account needs Dataflow Worker plus access to the source and sink.
• Job names must be unique; suffix with github.run_id to avoid "job already exists".

Related guides