build workflow (hyz-xmaster/swa_object_detection)
The build workflow from hyz-xmaster/swa_object_detection, explained and optimized by Latchkey.
CI health: F - at risk
Point runs-on at Latchkey and get caching, run de-duplication, job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.
What it does
This is the build workflow from the hyz-xmaster/swa_object_detection repository, a real project running GitHub Actions. It is shown here with attribution under its Apache-2.0 license.
Below, Latchkey shows a faster, safer version produced by its optimization engine.
The workflow
name: build
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.7
uses: actions/setup-python@v2
with:
python-version: 3.7
- name: Install pre-commit hook
run: |
pip install pre-commit
pre-commit install
- name: Linting
run: pre-commit run --all-files
- name: Check docstring coverage
run: |
pip install interrogate
interrogate -v --ignore-init-method --ignore-module --ignore-nested-functions --ignore-regex "__repr__" --fail-under 80 mmdet
build_cpu:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]
torch: [1.3.1, 1.5.1, 1.6.0]
include:
- torch: 1.3.1
torchvision: 0.4.2
mmcv: "latest+torch1.3.0+cpu"
- torch: 1.5.1
torchvision: 0.6.1
mmcv: "latest+torch1.5.0+cpu"
- torch: 1.6.0
torchvision: 0.7.0
mmcv: "latest+torch1.6.0+cpu"
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install Pillow
run: pip install Pillow==6.2.2
if: ${{matrix.torchvision == '0.4.2'}}
- name: Install PyTorch
run: pip install torch==${{matrix.torch}}+cpu torchvision==${{matrix.torchvision}}+cpu -f https://download.pytorch.org/whl/torch_stable.html
- name: Install MMCV
run: |
pip install mmcv-full==${{matrix.mmcv}} -f https://download.openmmlab.com/mmcv/dist/index.html --use-deprecated=legacy-resolver
python -c 'import mmcv; print(mmcv.__version__)'
- name: Install unittest dependencies
run: pip install -r requirements/tests.txt -r requirements/optional.txt
- name: Build and install
run: rm -rf .eggs && pip install -e .
- name: Run unittests and generate coverage report
run: |
coverage run --branch --source mmdet -m pytest tests/
coverage xml
coverage report -m
build_cuda:
runs-on: ubuntu-latest
env:
CUDA: 10.1.105-1
CUDA_SHORT: 10.1
UBUNTU_VERSION: ubuntu1804
strategy:
matrix:
python-version: [3.7]
torch: [1.3.1, 1.5.1+cu101, 1.6.0+cu101]
include:
- torch: 1.3.1
torchvision: 0.4.2
mmcv: "latest+torch1.3.0+cu101"
- torch: 1.5.1+cu101
torchvision: 0.6.1+cu101
mmcv: "latest+torch1.5.0+cu101"
- torch: 1.6.0+cu101
torchvision: 0.7.0+cu101
mmcv: "latest+torch1.6.0+cu101"
- torch: 1.6.0+cu101
torchvision: 0.7.0+cu101
mmcv: "latest+torch1.6.0+cu101"
python-version: 3.6
- torch: 1.6.0+cu101
torchvision: 0.7.0+cu101
mmcv: "latest+torch1.6.0+cu101"
python-version: 3.8
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install CUDA
run: |
export INSTALLER=cuda-repo-${UBUNTU_VERSION}_${CUDA}_amd64.deb
wget http://developer.download.nvidia.com/compute/cuda/repos/${UBUNTU_VERSION}/x86_64/${INSTALLER}
sudo dpkg -i ${INSTALLER}
wget https://developer.download.nvidia.com/compute/cuda/repos/${UBUNTU_VERSION}/x86_64/7fa2af80.pub
sudo apt-key add 7fa2af80.pub
sudo apt update -qq
sudo apt install -y cuda-${CUDA_SHORT/./-} cuda-cufft-dev-${CUDA_SHORT/./-}
sudo apt clean
export CUDA_HOME=/usr/local/cuda-${CUDA_SHORT}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${CUDA_HOME}/include:${LD_LIBRARY_PATH}
export PATH=${CUDA_HOME}/bin:${PATH}
- name: Install Pillow
run: pip install Pillow==6.2.2
if: ${{matrix.torchvision < 0.5}}
- name: Install PyTorch
run: pip install torch==${{matrix.torch}} torchvision==${{matrix.torchvision}} -f https://download.pytorch.org/whl/torch_stable.html
- name: Install mmdet dependencies
run: |
pip install mmcv-full==${{matrix.mmcv}} -f https://download.openmmlab.com/mmcv/dist/index.html --use-deprecated=legacy-resolver
pip install -r requirements.txt
python -c 'import mmcv; print(mmcv.__version__)'
- name: Build and install
run: |
rm -rf .eggs
python setup.py check -m -s
TORCH_CUDA_ARCH_LIST=7.0 pip install .
- name: Run unittests and generate coverage report
run: |
coverage run --branch --source mmdet -m pytest tests/
coverage xml
coverage report -m
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1.0.10
with:
file: ./coverage.xml
flags: unittests
env_vars: OS,PYTHON
name: codecov-umbrella
fail_ci_if_error: false
The same workflow, on Latchkey
Estimated ~20% faster on cache hits, plus fewer wasted runs and a safer supply chain. Added and changed lines are highlighted.
name: build on: [push, pull_request] concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: lint: timeout-minutes: 30 runs-on: latchkey-small steps: - uses: actions/checkout@v2 - name: Set up Python 3.7 uses: actions/setup-python@v2 with: cache: 'pip' python-version: 3.7 - name: Install pre-commit hook run: | pip install pre-commit pre-commit install - name: Linting run: pre-commit run --all-files - name: Check docstring coverage run: | pip install interrogate interrogate -v --ignore-init-method --ignore-module --ignore-nested-functions --ignore-regex "__repr__" --fail-under 80 mmdet build_cpu: timeout-minutes: 30 runs-on: latchkey-small strategy: matrix: python-version: [3.7] torch: [1.3.1, 1.5.1, 1.6.0] include: - torch: 1.3.1 torchvision: 0.4.2 mmcv: "latest+torch1.3.0+cpu" - torch: 1.5.1 torchvision: 0.6.1 mmcv: "latest+torch1.5.0+cpu" - torch: 1.6.0 torchvision: 0.7.0 mmcv: "latest+torch1.6.0+cpu" steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: cache: 'pip' python-version: ${{ matrix.python-version }} - name: Install Pillow run: pip install Pillow==6.2.2 if: ${{matrix.torchvision == '0.4.2'}} - name: Install PyTorch run: pip install torch==${{matrix.torch}}+cpu torchvision==${{matrix.torchvision}}+cpu -f https://download.pytorch.org/whl/torch_stable.html - name: Install MMCV run: | pip install mmcv-full==${{matrix.mmcv}} -f https://download.openmmlab.com/mmcv/dist/index.html --use-deprecated=legacy-resolver python -c 'import mmcv; print(mmcv.__version__)' - name: Install unittest dependencies run: pip install -r requirements/tests.txt -r requirements/optional.txt - name: Build and install run: rm -rf .eggs && pip install -e . - name: Run unittests and generate coverage report run: | coverage run --branch --source mmdet -m pytest tests/ coverage xml coverage report -m build_cuda: timeout-minutes: 30 runs-on: latchkey-small env: CUDA: 10.1.105-1 CUDA_SHORT: 10.1 UBUNTU_VERSION: ubuntu1804 strategy: matrix: python-version: [3.7] torch: [1.3.1, 1.5.1+cu101, 1.6.0+cu101] include: - torch: 1.3.1 torchvision: 0.4.2 mmcv: "latest+torch1.3.0+cu101" - torch: 1.5.1+cu101 torchvision: 0.6.1+cu101 mmcv: "latest+torch1.5.0+cu101" - torch: 1.6.0+cu101 torchvision: 0.7.0+cu101 mmcv: "latest+torch1.6.0+cu101" - torch: 1.6.0+cu101 torchvision: 0.7.0+cu101 mmcv: "latest+torch1.6.0+cu101" python-version: 3.6 - torch: 1.6.0+cu101 torchvision: 0.7.0+cu101 mmcv: "latest+torch1.6.0+cu101" python-version: 3.8 steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: cache: 'pip' python-version: ${{ matrix.python-version }} - name: Install CUDA run: | export INSTALLER=cuda-repo-${UBUNTU_VERSION}_${CUDA}_amd64.deb wget http://developer.download.nvidia.com/compute/cuda/repos/${UBUNTU_VERSION}/x86_64/${INSTALLER} sudo dpkg -i ${INSTALLER} wget https://developer.download.nvidia.com/compute/cuda/repos/${UBUNTU_VERSION}/x86_64/7fa2af80.pub sudo apt-key add 7fa2af80.pub sudo apt update -qq sudo apt install -y cuda-${CUDA_SHORT/./-} cuda-cufft-dev-${CUDA_SHORT/./-} sudo apt clean export CUDA_HOME=/usr/local/cuda-${CUDA_SHORT} export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${CUDA_HOME}/include:${LD_LIBRARY_PATH} export PATH=${CUDA_HOME}/bin:${PATH} - name: Install Pillow run: pip install Pillow==6.2.2 if: ${{matrix.torchvision < 0.5}} - name: Install PyTorch run: pip install torch==${{matrix.torch}} torchvision==${{matrix.torchvision}} -f https://download.pytorch.org/whl/torch_stable.html - name: Install mmdet dependencies run: | pip install mmcv-full==${{matrix.mmcv}} -f https://download.openmmlab.com/mmcv/dist/index.html --use-deprecated=legacy-resolver pip install -r requirements.txt python -c 'import mmcv; print(mmcv.__version__)' - name: Build and install run: | rm -rf .eggs python setup.py check -m -s TORCH_CUDA_ARCH_LIST=7.0 pip install . - name: Run unittests and generate coverage report run: | coverage run --branch --source mmdet -m pytest tests/ coverage xml coverage report -m - name: Upload coverage to Codecov uses: codecov/codecov-action@v1.0.10 with: file: ./coverage.xml flags: unittests env_vars: OS,PYTHON name: codecov-umbrella fail_ci_if_error: false
What changed
- Run on Latchkey managed runners with one line (
runs-on), which apply the fixes below automatically and self-heal transient failures. This example useslatchkey-small; pick the runner size that fits the job. - Cancel superseded runs when a branch or PR gets a newer push.
- Cache dependency installs on the setup step so they are served from cache.
- Add a job timeout so a hung step cannot burn hours of runner time.
1 third-party action is referenced by a movable tag. Pin it to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.
What Latchkey heals here
This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:
- Dependency installs
- Network fetches
This workflow runs 3 jobs (7 with the matrix expanded) per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.