Skip to content
Latchkey
Start Free

BigQuery Schema Generator CI workflow (bxparks/bigquery-schema-generator)

The BigQuery Schema Generator CI workflow from bxparks/bigquery-schema-generator, explained and optimized by Latchkey.

D

CI health: D - needs work

Point runs-on at Latchkey and get caching, run de-duplication, job timeouts, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →
Source: bxparks/bigquery-schema-generator.github/workflows/pythonpackage.ymlLicense Apache-2.0View source

What it does

This is the BigQuery Schema Generator CI workflow from the bxparks/bigquery-schema-generator repository, a real project running GitHub Actions. It is shown here with attribution under its Apache-2.0 license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: BigQuery Schema Generator CI 

on:
  push:
    branches: [ develop ]
  pull_request:
    branches: [ develop ]

jobs:
  build:

    runs-on: ubuntu-22.04
    strategy: 
      matrix:
        # Supported Python versions under GitHub Actions is listed in
        # https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
        #
        # * 3.5 does not support f-strings so no longer supported by
        #   bigquery-schema-generator.
        # * 3.6 should support bigquery-schema-generator, but is not
        #   supported by ubuntu-22.04 image.
        # * 3.11 is the latest under ubuntu-22.04.
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8
        # pip install -r requirements.txt

    - name: Lint with flake8
      run: |
        # Stop the build for most python errors.
        # W503 and W504 are both enabled by default and are mutual
        # contradictory, so we have to suppress one of them
        # E501 uses 79 columns by default, but 80 is the default line wrap in
        # vim, so change the line-length.
        flake8 . --count --ignore W503 --show-source --statistics \
            --max-line-length=80

        # Exit-zero treats all errors as warnings.
        # The complexity warning is not useful... in fact the whole thing is
        # not useful, so turn it off.
        # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=80
        # --statistics

    - name: Test with unittest
      run: |
        python -m unittest

The same workflow, on Latchkey

Estimated ~20% faster on cache hits, plus fewer wasted runs and a safer supply chain. Added and changed lines are highlighted.

# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
name: BigQuery Schema Generator CI 
 
on:
  push:
    branches: [ develop ]
  pull_request:
    branches: [ develop ]
 
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
 
jobs:
  build:
    timeout-minutes: 30
 
    runs-on: latchkey-small
    strategy: 
      matrix:
        # Supported Python versions under GitHub Actions is listed in
        # https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
        #
        # * 3.5 does not support f-strings so no longer supported by
        #   bigquery-schema-generator.
        # * 3.6 should support bigquery-schema-generator, but is not
        #   supported by ubuntu-22.04 image.
        # * 3.11 is the latest under ubuntu-22.04.
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
    steps:
    - uses: actions/checkout@v3
 
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        cache: 'pip'
        python-version: ${{ matrix.python-version }}
 
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8
        # pip install -r requirements.txt
 
    - name: Lint with flake8
      run: |
        # Stop the build for most python errors.
        # W503 and W504 are both enabled by default and are mutual
        # contradictory, so we have to suppress one of them
        # E501 uses 79 columns by default, but 80 is the default line wrap in
        # vim, so change the line-length.
        flake8 . --count --ignore W503 --show-source --statistics \
            --max-line-length=80
 
        # Exit-zero treats all errors as warnings.
        # The complexity warning is not useful... in fact the whole thing is
        # not useful, so turn it off.
        # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=80
        # --statistics
 
    - name: Test with unittest
      run: |
        python -m unittest

What changed

What Latchkey heals here

This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:

This workflow runs 1 job (5 with the matrix expanded) per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow

actions/checkout actions/setup-python