CI/CD for a Python Data Pipeline with GitHub Actions

Lint, test, and smoke-run your ETL so a broken transform never reaches production data.

A Python data pipeline -- extraction, transformation, loading -- needs CI that catches both code bugs and broken transforms. The recipe lints with ruff, runs pytest, and does a smoke run against sample data so a malformed transform fails in CI rather than in production. Dependency caching keeps installs quick.

What the pipeline does

• Installs dependencies with pip caching enabled.
• Lints the code with ruff.
• Runs the pytest suite, including data-transform tests.
• Smoke-runs the pipeline against a small sample dataset.

The workflow

setup-python provides pip caching keyed on your requirements file so dependency installs are fast on repeat runs.

name: Data Pipeline CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
      - run: pip install -r requirements.txt
      - name: Lint
        run: ruff check .
      - name: Test
        run: pytest -q
      - name: Smoke run on sample data
        run: python -m pipeline.run --input tests/fixtures/sample.csv --dry-run

Notes for this platform

Keep a small committed sample dataset so the smoke run validates the real transform logic without touching production sources. Cache pip to avoid reinstalling heavy data libraries (pandas, pyarrow) every run. This is Linux-native work, so use your cheapest, fastest runner class; managed runners auto-retry transient PyPI install failures so a flaky package index does not red-build a clean pipeline change.

Key takeaways

• Smoke-run the pipeline on a committed sample so transforms are validated in CI.
• Cache pip to skip reinstalling heavy data libraries every run.
• Lint with ruff and test with pytest before any pipeline run step.

Related guides