CI workflow (ProjectQ-Framework/ProjectQ)
The CI workflow from ProjectQ-Framework/ProjectQ, explained and optimized by Latchkey.
CI health: A - excellent
Point runs-on at Latchkey and get job timeouts, self-healing for flaky steps, and up to 58% lower cost, applied automatically.
What it does
This is the CI workflow from the ProjectQ-Framework/ProjectQ repository, a real project running GitHub Actions. It is shown here with attribution under its Apache-2.0 license.
Below, Latchkey shows a faster, safer version produced by its optimization engine.
The workflow
---
name: CI
on:
workflow_dispatch:
merge_group:
pull_request:
push:
branches:
- master
- develop
- v*
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
standard:
strategy:
fail-fast: false
matrix:
runs-on: [ubuntu-latest, windows-latest, macos-latest]
python:
- 3.8
- 3.9
- '3.10'
- '3.11'
- '3.12'
name: "Python ${{ matrix.python }} • ${{ matrix.runs-on }} • x64 ${{ matrix.args }}"
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@v3
- name: Get history and tags for SCM versioning to work
if: ${{ !env.ACT }}
run: |
git fetch --prune --unshallow
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
- name: Setup Python ${{ matrix.python }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
architecture: 'x64'
cache: 'pip'
cache-dependency-path: |
setup.cfg
pyproject.toml
- name: Prepare env
run: |
python -m pip install -U "setuptools>=61.0.0"
- name: Generate requirement file (Unix)
if: runner.os != 'Windows'
run: |
python setup.py gen_reqfile --include-extras=test,azure-quantum,braket,revkit
- name: Generate requirement file (Windows)
if: runner.os == 'Windows'
run: |
python setup.py gen_reqfile --include-extras=test,azure-quantum,braket
- name: Prepare env
run: |
python -m pip install -U pip setuptools wheel pybind11
cat requirements.txt
python -m pip install -r requirements.txt --prefer-binary
python -m pip install coveralls
- name: Setup annotations on Linux
if: runner.os == 'Linux'
run: python -m pip install pytest-github-actions-annotate-failures
- name: Build and install package (Unix)
if: runner.os != 'Windows'
run: python -m pip install -ve .[azure-quantum,braket,revkit,test]
- name: Build and install package (Windows)
if: runner.os == 'Windows'
run: python -m pip install -ve .[azure-quantum,braket,test]
- name: Pytest
run: |
echo 'backend: Agg' > matplotlibrc
python -m pytest -p no:warnings --cov=projectq
- name: Coveralls.io
run: coveralls --service=github
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COVERALLS_FLAG_NAME: python-${{ matrix.python }}-${{ matrix.runs-on }}-x64
COVERALLS_PARALLEL: true
finish:
needs: standard
runs-on: ubuntu-latest
container: python:3-slim
steps:
- name: Coveralls Finished
run: |
pip3 install --upgrade coveralls
coveralls --finish
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
clang:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
clang:
- 11
- latest
env:
CC: clang
CXX: clang++
PROJECTQ_CLEANUP_COMPILER_FLAGS: ${{ (matrix.clang <= 10) && 1 || 0 }}
name: "Python 3 • Clang ${{ matrix.clang }} • x64"
container: "silkeh/clang:${{ matrix.clang }}"
steps:
- name: Install Git
run: |
apt-get update && apt-get install -y git --no-install-recommends
# Work-around for https://github.com/actions/runner-images/issues/6775
- name: Change Owner of Container Working Directory
run: chown root:root .
- uses: actions/checkout@v3
- name: Get history and tags for SCM versioning to work
if: ${{ !env.ACT }}
run: |
git fetch --prune --unshallow
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
- name: Prepare env
run: >
apt-get update && apt-get install -y python3-dev python3-pip python3-setuptools python3-wheel
python3-numpy python3-scipy python3-matplotlib python3-requests python3-networkx
python3-pytest python3-pytest-cov python3-flaky python3-venv
--no-install-recommends
- name: Prepare Python env
run: |
python3 -m venv venv
./venv/bin/python3 -m pip install -U pip setuptools wheel
./venv/bin/python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket
cat requirements.txt
./venv/bin/python3 -m pip install -r requirements.txt --prefer-binary
- name: Upgrade pybind11 and flaky
run: ./venv/bin/python3 -m pip install --upgrade pybind11 flaky --prefer-binary
- name: Build and install package
run: ./venv/bin/python3 -m pip install -ve .[azure-quantum,braket,test]
- name: Pytest
run: |
echo 'backend: Agg' > matplotlibrc
./venv/bin/python3 -m pytest -p no:warnings
gcc:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
gcc:
- 9
- latest
name: "Python 3 • GCC ${{ matrix.gcc }} • x64"
container: "gcc:${{ matrix.gcc }}"
steps:
- uses: actions/checkout@v3
# Work-around for https://github.com/actions/runner-images/issues/6775
- name: Change Owner of Container Working Directory
run: chown root:root .
- name: Get history and tags for SCM versioning to work
if: ${{ !env.ACT }}
run: |
git fetch --prune --unshallow
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
- name: Prepare env
run: >
apt-get update && apt-get install -y python3-dev python3-pip python3-setuptools python3-wheel
python3-numpy python3-scipy python3-matplotlib python3-requests python3-networkx
python3-pytest python3-pytest-cov python3-flaky python3-venv
--no-install-recommends
- name: Prepare Python env
run: |
python3 -m venv venv
./venv/bin/python3 -m pip install -U pip setuptools wheel
./venv/bin/python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket
cat requirements.txt
./venv/bin/python3 -m pip install -r requirements.txt --prefer-binary
- name: Upgrade pybind11 and flaky
run: ./venv/bin/python3 -m pip install --upgrade pybind11 flaky --prefer-binary
- name: Build and install package
run: ./venv/bin/python3 -m pip install -ve .[azure-quantum,braket,test]
- name: Pytest
run: |
echo 'backend: Agg' > matplotlibrc
./venv/bin/python3 -m pytest -p no:warnings
# Testing on CentOS (manylinux uses a centos base, and this is an easy way
# to get GCC 4.8, which is the manylinux1 compiler).
centos:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
centos:
- 7 # GCC 4.8
- 8
include:
- centos: 7
python_pkg: rh-python38-python-pip rh-python38-python-devel
enable_repo: --enablerepo=centos-sclo-rh
- centos: 8
python_pkg: python38-devel
name: "Python 3 • CentOS ${{ matrix.centos }} • x64"
container: "centos:${{ matrix.centos }}"
steps:
- name: Enable cache for yum
run: echo 'keepcache=1' >> /etc/yum.conf
- name: Setup yum cache
uses: actions/cache@v3
with:
path: |
/var/cache/yum/
/var/cache/dnf/
key: ${{ runner.os }}-centos${{ matrix.centos }}-yum-${{ secrets.yum_cache }}
- name: Fix repository URLs (CentOS 8 only)
if: matrix.centos == 8
run: |
sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
- name: Update YUM/DNF
run: yum update -y
- name: Enable extra repositories && modify PATH (CentOS 7)
if: matrix.centos == 7
run: |
yum install -y centos-release-scl-rh
yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
echo '/opt/rh/rh-python38/root/usr/bin' >> $GITHUB_PATH
- name: Add Python 3 and other dependencies
run: yum install -y ${{ matrix.enable_repo }} ${{ matrix.python_pkg }} gcc-c++ make
- name: Install Git > 2.18
run: |
yum install -y git
git config --global --add safe.directory /__w/ProjectQ/ProjectQ
# Work-around for https://github.com/actions/runner-images/issues/6775
- name: Change Owner of Container Working Directory
run: chown root:root .
- uses: actions/checkout@v3
- name: Get history and tags for SCM versioning to work
if: ${{ !env.ACT }}
run: |
git fetch --prune --unshallow
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
- name: Create pip cache dir
run: mkdir -p ~/.cache/pip
- name: Cache wheels
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-centos${{ matrix.centos }}-pip-${{ hashFiles('**/setup.cfg', '**/pyproject.toml') }}
restore-keys: ${{ runner.os }}-centos-pip-
- name: Install dependencies
run: |
python3 -m pip install -U pip setuptools wheel
python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket
cat requirements.txt
python3 -m pip install -r requirements.txt --prefer-binary
- name: Build and install package
run: python3 -m pip install -ve .[azure-quantum,braket,test]
- name: Pytest
run: |
echo 'backend: Agg' > matplotlibrc
python3 -m pytest -p no:warnings
documentation:
name: "Documentation build test"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Get history and tags for SCM versioning to work
if: ${{ !env.ACT }}
run: |
git fetch --prune --unshallow
git fetch --depth=1 origin +refs/tags/*:refs/tags/*
- uses: actions/setup-python@v4
with:
python-version: '3.x'
architecture: 'x64'
cache: 'pip'
cache-dependency-path: |
setup.cfg
pyproject.toml
- name: Install docs & setup requirements
run: |
python3 -m pip install .[docs]
- name: Build docs
run: python3 -m sphinx -b html docs docs/.build
# NB: disabling until setup.py is updated to remove any mention of distutils
# - name: Make SDist
# run: python3 setup.py sdist
The same workflow, on Latchkey
Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.
--- name: CI on: workflow_dispatch: merge_group: pull_request: push: branches: - master - develop - v* concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: standard: timeout-minutes: 30 strategy: fail-fast: false matrix: runs-on: [ubuntu-latest, windows-latest, macos-latest] python: - 3.8 - 3.9 - '3.10' - '3.11' - '3.12' name: "Python ${{ matrix.python }} • ${{ matrix.runs-on }} • x64 ${{ matrix.args }}" runs-on: ${{ matrix.runs-on }} steps: - uses: actions/checkout@v3 - name: Get history and tags for SCM versioning to work if: ${{ !env.ACT }} run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Setup Python ${{ matrix.python }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python }} architecture: 'x64' cache: 'pip' cache-dependency-path: | setup.cfg pyproject.toml - name: Prepare env run: | python -m pip install -U "setuptools>=61.0.0" - name: Generate requirement file (Unix) if: runner.os != 'Windows' run: | python setup.py gen_reqfile --include-extras=test,azure-quantum,braket,revkit - name: Generate requirement file (Windows) if: runner.os == 'Windows' run: | python setup.py gen_reqfile --include-extras=test,azure-quantum,braket - name: Prepare env run: | python -m pip install -U pip setuptools wheel pybind11 cat requirements.txt python -m pip install -r requirements.txt --prefer-binary python -m pip install coveralls - name: Setup annotations on Linux if: runner.os == 'Linux' run: python -m pip install pytest-github-actions-annotate-failures - name: Build and install package (Unix) if: runner.os != 'Windows' run: python -m pip install -ve .[azure-quantum,braket,revkit,test] - name: Build and install package (Windows) if: runner.os == 'Windows' run: python -m pip install -ve .[azure-quantum,braket,test] - name: Pytest run: | echo 'backend: Agg' > matplotlibrc python -m pytest -p no:warnings --cov=projectq - name: Coveralls.io run: coveralls --service=github env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} COVERALLS_FLAG_NAME: python-${{ matrix.python }}-${{ matrix.runs-on }}-x64 COVERALLS_PARALLEL: true finish: timeout-minutes: 30 needs: standard runs-on: latchkey-small container: python:3-slim steps: - name: Coveralls Finished run: | pip3 install --upgrade coveralls coveralls --finish env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} clang: timeout-minutes: 30 runs-on: latchkey-small strategy: fail-fast: false matrix: clang: - 11 - latest env: CC: clang CXX: clang++ PROJECTQ_CLEANUP_COMPILER_FLAGS: ${{ (matrix.clang <= 10) && 1 || 0 }} name: "Python 3 • Clang ${{ matrix.clang }} • x64" container: "silkeh/clang:${{ matrix.clang }}" steps: - name: Install Git run: | apt-get update && apt-get install -y git --no-install-recommends # Work-around for https://github.com/actions/runner-images/issues/6775 - name: Change Owner of Container Working Directory run: chown root:root . - uses: actions/checkout@v3 - name: Get history and tags for SCM versioning to work if: ${{ !env.ACT }} run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Prepare env run: > apt-get update && apt-get install -y python3-dev python3-pip python3-setuptools python3-wheel python3-numpy python3-scipy python3-matplotlib python3-requests python3-networkx python3-pytest python3-pytest-cov python3-flaky python3-venv --no-install-recommends - name: Prepare Python env run: | python3 -m venv venv ./venv/bin/python3 -m pip install -U pip setuptools wheel ./venv/bin/python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket cat requirements.txt ./venv/bin/python3 -m pip install -r requirements.txt --prefer-binary - name: Upgrade pybind11 and flaky run: ./venv/bin/python3 -m pip install --upgrade pybind11 flaky --prefer-binary - name: Build and install package run: ./venv/bin/python3 -m pip install -ve .[azure-quantum,braket,test] - name: Pytest run: | echo 'backend: Agg' > matplotlibrc ./venv/bin/python3 -m pytest -p no:warnings gcc: timeout-minutes: 30 runs-on: latchkey-small strategy: fail-fast: false matrix: gcc: - 9 - latest name: "Python 3 • GCC ${{ matrix.gcc }} • x64" container: "gcc:${{ matrix.gcc }}" steps: - uses: actions/checkout@v3 # Work-around for https://github.com/actions/runner-images/issues/6775 - name: Change Owner of Container Working Directory run: chown root:root . - name: Get history and tags for SCM versioning to work if: ${{ !env.ACT }} run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Prepare env run: > apt-get update && apt-get install -y python3-dev python3-pip python3-setuptools python3-wheel python3-numpy python3-scipy python3-matplotlib python3-requests python3-networkx python3-pytest python3-pytest-cov python3-flaky python3-venv --no-install-recommends - name: Prepare Python env run: | python3 -m venv venv ./venv/bin/python3 -m pip install -U pip setuptools wheel ./venv/bin/python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket cat requirements.txt ./venv/bin/python3 -m pip install -r requirements.txt --prefer-binary - name: Upgrade pybind11 and flaky run: ./venv/bin/python3 -m pip install --upgrade pybind11 flaky --prefer-binary - name: Build and install package run: ./venv/bin/python3 -m pip install -ve .[azure-quantum,braket,test] - name: Pytest run: | echo 'backend: Agg' > matplotlibrc ./venv/bin/python3 -m pytest -p no:warnings # Testing on CentOS (manylinux uses a centos base, and this is an easy way # to get GCC 4.8, which is the manylinux1 compiler). centos: timeout-minutes: 30 runs-on: latchkey-small strategy: fail-fast: false matrix: centos: - 7 # GCC 4.8 - 8 include: - centos: 7 python_pkg: rh-python38-python-pip rh-python38-python-devel enable_repo: --enablerepo=centos-sclo-rh - centos: 8 python_pkg: python38-devel name: "Python 3 • CentOS ${{ matrix.centos }} • x64" container: "centos:${{ matrix.centos }}" steps: - name: Enable cache for yum run: echo 'keepcache=1' >> /etc/yum.conf - name: Setup yum cache uses: actions/cache@v3 with: path: | /var/cache/yum/ /var/cache/dnf/ key: ${{ runner.os }}-centos${{ matrix.centos }}-yum-${{ secrets.yum_cache }} - name: Fix repository URLs (CentOS 8 only) if: matrix.centos == 8 run: | sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* - name: Update YUM/DNF run: yum update -y - name: Enable extra repositories && modify PATH (CentOS 7) if: matrix.centos == 7 run: | yum install -y centos-release-scl-rh yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm echo '/opt/rh/rh-python38/root/usr/bin' >> $GITHUB_PATH - name: Add Python 3 and other dependencies run: yum install -y ${{ matrix.enable_repo }} ${{ matrix.python_pkg }} gcc-c++ make - name: Install Git > 2.18 run: | yum install -y git git config --global --add safe.directory /__w/ProjectQ/ProjectQ # Work-around for https://github.com/actions/runner-images/issues/6775 - name: Change Owner of Container Working Directory run: chown root:root . - uses: actions/checkout@v3 - name: Get history and tags for SCM versioning to work if: ${{ !env.ACT }} run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - name: Create pip cache dir run: mkdir -p ~/.cache/pip - name: Cache wheels uses: actions/cache@v3 with: path: ~/.cache/pip key: ${{ runner.os }}-centos${{ matrix.centos }}-pip-${{ hashFiles('**/setup.cfg', '**/pyproject.toml') }} restore-keys: ${{ runner.os }}-centos-pip- - name: Install dependencies run: | python3 -m pip install -U pip setuptools wheel python3 setup.py gen_reqfile --include-extras=test,azure-quantum,braket cat requirements.txt python3 -m pip install -r requirements.txt --prefer-binary - name: Build and install package run: python3 -m pip install -ve .[azure-quantum,braket,test] - name: Pytest run: | echo 'backend: Agg' > matplotlibrc python3 -m pytest -p no:warnings documentation: timeout-minutes: 30 name: "Documentation build test" runs-on: latchkey-small steps: - uses: actions/checkout@v3 - name: Get history and tags for SCM versioning to work if: ${{ !env.ACT }} run: | git fetch --prune --unshallow git fetch --depth=1 origin +refs/tags/*:refs/tags/* - uses: actions/setup-python@v4 with: python-version: '3.x' architecture: 'x64' cache: 'pip' cache-dependency-path: | setup.cfg pyproject.toml - name: Install docs & setup requirements run: | python3 -m pip install .[docs] - name: Build docs run: python3 -m sphinx -b html docs docs/.build # NB: disabling until setup.py is updated to remove any mention of distutils # - name: Make SDist # run: python3 setup.py sdist
What changed
- Run on Latchkey managed runners with one line (
runs-on), which apply the fixes below automatically and self-heal transient failures. This example useslatchkey-small; pick the runner size that fits the job. - Add a job timeout so a hung step cannot burn hours of runner time.
What Latchkey heals here
This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:
- Dependency installs
This workflow runs 6 jobs (23 with the matrix expanded) per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.