Benchmarks workflow (napari/napari)

The Benchmarks workflow from napari/napari, explained and optimized by Latchkey.

CI health: C - fair

Point runs-on at Latchkey and get caching, job timeouts, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →

Source: napari/napari.github/workflows/benchmarks.ymlLicense BSD-3-ClauseView source

What it does

This is the Benchmarks workflow from the napari/napari repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)

# This CI configuration for relative benchmarks is based on the research done
# for scikit-image's implementation available here:
# https://github.com/scikit-image/scikit-image/blob/9bdd010a8/.github/workflows/benchmarks.yml#L1
# Blog post with the rationale: https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/

name: Benchmarks

on:
  workflow_call:
  schedule:
    - cron: "6 6 * * 0" # every sunday
  workflow_dispatch:
    inputs:
      base_ref:
        description: "Baseline commit or git reference"
        required: true
      contender_ref:
        description: "Contender commit or git reference"
        required: true

# This is the main configuration section that needs to be fine tuned to napari's needs
# All the *_THREADS options is just to make the benchmarks more robust by not using parallelism
env:
  OPENBLAS_NUM_THREADS: "1"
  MKL_NUM_THREADS: "1"
  OMP_NUM_THREADS: "1"
  ASV_OPTIONS: "--split --show-stderr --factor 1.5 --attribute timeout=900"
  # --split -> split final reports in tables
  # --show-stderr -> print tracebacks if errors occur
  # --factor 1.5 -> report anomaly if tested timings are beyond 1.5x base timings
  # --attribute timeout=300 -> override timeout attribute (default=60s) to allow slow tests to run
  # see https://asv.readthedocs.io/en/stable/commands.html#asv-continuous for more details!

jobs:
  benchmark:
    name: ${{ matrix.benchmark-name }}
    runs-on: ${{ matrix.runs-on }}
    env:
      PYTHON_VERSION: "3.12"
    permissions:
      contents: read
      issues: write
    strategy:
      fail-fast: false
      matrix:
        include:
          - benchmark-name: Qt
            asv-command: continuous
            selection-regex: "^benchmark_qt_.*"
            runs-on: macos-15
            # Qt tests run on macOS to avoid using Xvfb business
            # xvfb makes everything run, but some tests segfault :shrug:
            # Fortunately, macOS graphics stack does not need xvfb!
          - benchmark-name: non-Qt
            asv-command: continuous
            selection-regex: "^benchmark_(?!qt_).*"
            runs-on: ubuntu-latest

    steps:
      # We need the full repo to avoid this issue
      # https://github.com/actions/checkout/issues/23
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          fetch-depth: 0
          persist-credentials: false

      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
        name: Install Python
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache-dependency-path: pyproject.toml

      - name: Setup headless display
        uses: pyvista/setup-headless-display-action@5bc8de3bc71fcda7a96439571287a554901541a0 # v4.3
        with:
          qt: true

      - name: Download benchmark data
        run: |
          wget https://data.napari.dev/triangulation-benchmark-shapes.zip
          unzip triangulation-benchmark-shapes.zip -d src/napari/benchmarks/
          rm triangulation-benchmark-shapes.zip

      - name: Setup asv
        run: python -m pip install "asv[virtualenv]"
        env:
          PIP_CONSTRAINT: resources/constraints/benchmark.txt

      - uses: octokit/request-action@45eef13998ff762839b0a0de9bbcf41970fd056d # v2.x
        id: latest_release
        with:
          route: GET /repos/{owner}/{repo}/releases/latest
          owner: napari
          repo: napari
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Run ${{ matrix.benchmark-name }} benchmarks
        id: run_benchmark
        env:
          # asv will checkout commits, which might contain LFS artifacts; ignore those errors since
          # they are probably just documentation PNGs not needed here anyway
          GIT_LFS_SKIP_SMUDGE: 1
          HEAD_LABEL: ${{ github.event.pull_request.head.label }}
          PIP_CONSTRAINT: ${{ github.workspace }}/resources/constraints/benchmark.txt
          TMPDIR: ${{ github.workspace }}/tmp
        run: |
          set -euxo pipefail
          mkdir -p "${TMPDIR}"
          touch "${TMPDIR}/empty"
          read -ra cmd_options <<< "$ASV_OPTIONS"

          # ID this runner
          asv machine --yes

          if [[ $GITHUB_EVENT_NAME == pull_request ]]; then
            EVENT_NAME="PR #${{ github.event.pull_request.number }}"
            BASE_REF=${{ github.event.pull_request.base.sha }}
            CONTENDER_REF=${GITHUB_SHA}
            echo "Baseline:  ${BASE_REF} (${{ github.event.pull_request.base.label }})"
            echo "Contender: ${CONTENDER_REF} ($HEAD_LABEL)"
          elif [[ $GITHUB_EVENT_NAME == schedule ]]; then
            EVENT_NAME="cronjob"
            BASE_REF="${{ fromJSON(steps.latest_release.outputs.data).target_commitish }}"
            CONTENDER_REF="${GITHUB_SHA}"
            echo "Baseline:  ${BASE_REF} (${{ fromJSON(steps.latest_release.outputs.data).tag_name }})"
            echo "Contender: ${CONTENDER_REF} (current main)"
          elif [[ $GITHUB_EVENT_NAME == workflow_dispatch ]]; then
            EVENT_NAME="manual trigger"
            BASE_REF="${{ github.event.inputs.base_ref }}"
            CONTENDER_REF="${{ github.event.inputs.contender_ref }}"
            echo "Baseline:  ${BASE_REF} (workflow input)"
            echo "Contender: ${CONTENDER_REF} (workflow input)"
          fi

          echo "EVENT_NAME=$EVENT_NAME" >> "$GITHUB_ENV"
          echo "BASE_REF=$BASE_REF" >> "$GITHUB_ENV"
          echo "CONTENDER_REF=$CONTENDER_REF" >> "$GITHUB_ENV"

          # Run benchmarks for current commit against base
          asv continuous "${cmd_options[@]}" -b "${{ matrix.selection-regex }}" "${BASE_REF}" "${CONTENDER_REF}" \
          | sed -E "/Traceback | failed$|PERFORMANCE DECREASED/ s/^/::error:: /" \
          | tee asv_continuous.log

          # Report and export results for subsequent steps
          if grep "Traceback \|failed\|PERFORMANCE DECREASED" asv_continuous.log > /dev/null ; then
              exit 1
          fi

      - name: Report Failures as Issue
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && failure() }}
        uses: JasonEtco/create-an-issue@1b14a70e4d8dc185e5cc76d3bec9eab20257b2c5 # v2.9.2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PLATFORM: ${{ matrix.runs-on }}
          PYTHON: ${{ env.PYTHON_VERSION }}
          BACKEND: ${{ matrix.benchmark-name }}
          RUN_ID: ${{ github.run_id }}
          TITLE: "[test-bot] Benchmark tests failing"
        with:
          filename: .github/TEST_FAIL_TEMPLATE.md
          update_existing: true

      - name: Upload additional information about failure
#         upload the tmp directory to the artifact
#         this is where our code stores the dumped
#         structures that crash triangulation
        if: failure()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          path: ${{ github.workspace }}/tmp
          include-hidden-files: true
          name: tmp-${{ matrix.benchmark-name }}

      - name: Add more info to artifact
        if: always()
        run: |
          # Copy the full `asv continuous` log
          cp asv_continuous.log .asv/results/asv_continuous_${{ matrix.benchmark-name }}.log
          # ensure that even if this isn't a PR, the benchmark_report workflow can run without error
          touch .asv/results/message_${{ matrix.benchmark-name }}.txt

          # Add the message that might be posted as a comment on the PR
          # We delegate the actual comment to `benchmarks_report.yml` due to
          # potential token permissions issues
          if [[ $GITHUB_EVENT_NAME == pull_request ]]; then

          echo "${{ github.event.pull_request.number }}" > .asv/results/pr_number
          echo \
          "The ${{ matrix.benchmark-name }} benchmark run requested by $EVENT_NAME ($CONTENDER_REF vs $BASE_REF) has" \
          "finished with status '${{ steps.run_benchmark.outcome }}'. See the" \
          "[CI logs and artifacts](||BENCHMARK_CI_LOGS_URL||) for further details." \
          > .asv/results/message_${{ matrix.benchmark-name }}.txt

          awk  '/Benchmark.*Parameter/,/SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY/' asv_continuous.log \
          >> .asv/results/message_${{ matrix.benchmark-name }}.txt

          fi

      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: asv-benchmark-results-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.benchmark-name }}
          path: .asv/results

  combine-artifacts:
    runs-on: ubuntu-latest
    needs: benchmark
    if: always() && needs.benchmark.result != 'skipped'
    steps:
      - name: Download artifact
        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          pattern: asv-benchmark-results*
          path: asv_result
          merge-multiple: true
      - name: Upload artifact
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: asv-benchmark-results-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}
          path: asv_result

The same workflow, on Latchkey

Estimated ~20% faster on cache hits, plus fewer wasted runs and a safer supply chain. Added and changed lines are highlighted.

# This CI configuration for relative benchmarks is based on the research done
# for scikit-image's implementation available here:
# https://github.com/scikit-image/scikit-image/blob/9bdd010a8/.github/workflows/benchmarks.yml#L1
# Blog post with the rationale: https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/
 
name: Benchmarks
 
on:
  workflow_call:
  schedule:
    - cron: "6 6 * * 0" # every sunday
  workflow_dispatch:
    inputs:
      base_ref:
        description: "Baseline commit or git reference"
        required: true
      contender_ref:
        description: "Contender commit or git reference"
        required: true
 
# This is the main configuration section that needs to be fine tuned to napari's needs
# All the *_THREADS options is just to make the benchmarks more robust by not using parallelism
env:
  OPENBLAS_NUM_THREADS: "1"
  MKL_NUM_THREADS: "1"
  OMP_NUM_THREADS: "1"
  ASV_OPTIONS: "--split --show-stderr --factor 1.5 --attribute timeout=900"
  # --split -> split final reports in tables
  # --show-stderr -> print tracebacks if errors occur
  # --factor 1.5 -> report anomaly if tested timings are beyond 1.5x base timings
  # --attribute timeout=300 -> override timeout attribute (default=60s) to allow slow tests to run
  # see https://asv.readthedocs.io/en/stable/commands.html#asv-continuous for more details!
 
jobs:
  benchmark:
    timeout-minutes: 30
    name: ${{ matrix.benchmark-name }}
    runs-on: ${{ matrix.runs-on }}
    env:
      PYTHON_VERSION: "3.12"
    permissions:
      contents: read
      issues: write
    strategy:
      fail-fast: false
      matrix:
        include:
          - benchmark-name: Qt
            asv-command: continuous
            selection-regex: "^benchmark_qt_.*"
            runs-on: macos-15
            # Qt tests run on macOS to avoid using Xvfb business
            # xvfb makes everything run, but some tests segfault :shrug:
            # Fortunately, macOS graphics stack does not need xvfb!
          - benchmark-name: non-Qt
            asv-command: continuous
            selection-regex: "^benchmark_(?!qt_).*"
            runs-on: latchkey-small
 
    steps:
      # We need the full repo to avoid this issue
      # https://github.com/actions/checkout/issues/23
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          fetch-depth: 0
          persist-credentials: false
 
      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
        name: Install Python
        with:
          cache: 'pip'
          python-version: ${{ env.PYTHON_VERSION }}
          cache-dependency-path: pyproject.toml
 
      - name: Setup headless display
        uses: pyvista/setup-headless-display-action@5bc8de3bc71fcda7a96439571287a554901541a0 # v4.3
        with:
          qt: true
 
      - name: Download benchmark data
        run: |
          wget https://data.napari.dev/triangulation-benchmark-shapes.zip
          unzip triangulation-benchmark-shapes.zip -d src/napari/benchmarks/
          rm triangulation-benchmark-shapes.zip
 
      - name: Setup asv
        run: python -m pip install "asv[virtualenv]"
        env:
          PIP_CONSTRAINT: resources/constraints/benchmark.txt
 
      - uses: octokit/request-action@45eef13998ff762839b0a0de9bbcf41970fd056d # v2.x
        id: latest_release
        with:
          route: GET /repos/{owner}/{repo}/releases/latest
          owner: napari
          repo: napari
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
      - name: Run ${{ matrix.benchmark-name }} benchmarks
        id: run_benchmark
        env:
          # asv will checkout commits, which might contain LFS artifacts; ignore those errors since
          # they are probably just documentation PNGs not needed here anyway
          GIT_LFS_SKIP_SMUDGE: 1
          HEAD_LABEL: ${{ github.event.pull_request.head.label }}
          PIP_CONSTRAINT: ${{ github.workspace }}/resources/constraints/benchmark.txt
          TMPDIR: ${{ github.workspace }}/tmp
        run: |
          set -euxo pipefail
          mkdir -p "${TMPDIR}"
          touch "${TMPDIR}/empty"
          read -ra cmd_options <<< "$ASV_OPTIONS"
 
          # ID this runner
          asv machine --yes
 
          if [[ $GITHUB_EVENT_NAME == pull_request ]]; then
            EVENT_NAME="PR #${{ github.event.pull_request.number }}"
            BASE_REF=${{ github.event.pull_request.base.sha }}
            CONTENDER_REF=${GITHUB_SHA}
            echo "Baseline:  ${BASE_REF} (${{ github.event.pull_request.base.label }})"
            echo "Contender: ${CONTENDER_REF} ($HEAD_LABEL)"
          elif [[ $GITHUB_EVENT_NAME == schedule ]]; then
            EVENT_NAME="cronjob"
            BASE_REF="${{ fromJSON(steps.latest_release.outputs.data).target_commitish }}"
            CONTENDER_REF="${GITHUB_SHA}"
            echo "Baseline:  ${BASE_REF} (${{ fromJSON(steps.latest_release.outputs.data).tag_name }})"
            echo "Contender: ${CONTENDER_REF} (current main)"
          elif [[ $GITHUB_EVENT_NAME == workflow_dispatch ]]; then
            EVENT_NAME="manual trigger"
            BASE_REF="${{ github.event.inputs.base_ref }}"
            CONTENDER_REF="${{ github.event.inputs.contender_ref }}"
            echo "Baseline:  ${BASE_REF} (workflow input)"
            echo "Contender: ${CONTENDER_REF} (workflow input)"
          fi
 
          echo "EVENT_NAME=$EVENT_NAME" >> "$GITHUB_ENV"
          echo "BASE_REF=$BASE_REF" >> "$GITHUB_ENV"
          echo "CONTENDER_REF=$CONTENDER_REF" >> "$GITHUB_ENV"
 
          # Run benchmarks for current commit against base
          asv continuous "${cmd_options[@]}" -b "${{ matrix.selection-regex }}" "${BASE_REF}" "${CONTENDER_REF}" \
          | sed -E "/Traceback | failed$|PERFORMANCE DECREASED/ s/^/::error:: /" \
          | tee asv_continuous.log
 
          # Report and export results for subsequent steps
          if grep "Traceback \|failed\|PERFORMANCE DECREASED" asv_continuous.log > /dev/null ; then
              exit 1
          fi
 
      - name: Report Failures as Issue
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && failure() }}
        uses: JasonEtco/create-an-issue@1b14a70e4d8dc185e5cc76d3bec9eab20257b2c5 # v2.9.2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PLATFORM: ${{ matrix.runs-on }}
          PYTHON: ${{ env.PYTHON_VERSION }}
          BACKEND: ${{ matrix.benchmark-name }}
          RUN_ID: ${{ github.run_id }}
          TITLE: "[test-bot] Benchmark tests failing"
        with:
          filename: .github/TEST_FAIL_TEMPLATE.md
          update_existing: true
 
      - name: Upload additional information about failure
#         upload the tmp directory to the artifact
#         this is where our code stores the dumped
#         structures that crash triangulation
        if: failure()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          path: ${{ github.workspace }}/tmp
          include-hidden-files: true
          name: tmp-${{ matrix.benchmark-name }}
 
      - name: Add more info to artifact
        if: always()
        run: |
          # Copy the full `asv continuous` log
          cp asv_continuous.log .asv/results/asv_continuous_${{ matrix.benchmark-name }}.log
          # ensure that even if this isn't a PR, the benchmark_report workflow can run without error
          touch .asv/results/message_${{ matrix.benchmark-name }}.txt
 
          # Add the message that might be posted as a comment on the PR
          # We delegate the actual comment to `benchmarks_report.yml` due to
          # potential token permissions issues
          if [[ $GITHUB_EVENT_NAME == pull_request ]]; then
 
          echo "${{ github.event.pull_request.number }}" > .asv/results/pr_number
          echo \
          "The ${{ matrix.benchmark-name }} benchmark run requested by $EVENT_NAME ($CONTENDER_REF vs $BASE_REF) has" \
          "finished with status '${{ steps.run_benchmark.outcome }}'. See the" \
          "[CI logs and artifacts](||BENCHMARK_CI_LOGS_URL||) for further details." \
          > .asv/results/message_${{ matrix.benchmark-name }}.txt
 
          awk  '/Benchmark.*Parameter/,/SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY/' asv_continuous.log \
          >> .asv/results/message_${{ matrix.benchmark-name }}.txt
 
          fi
 
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: asv-benchmark-results-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.benchmark-name }}
          path: .asv/results
 
  combine-artifacts:
    timeout-minutes: 30
    runs-on: latchkey-small
    needs: benchmark
    if: always() && needs.benchmark.result != 'skipped'
    steps:
      - name: Download artifact
        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          pattern: asv-benchmark-results*
          path: asv_result
          merge-multiple: true
      - name: Upload artifact
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: asv-benchmark-results-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}
          path: asv_result

What changed

Run on Latchkey managed runners with one line (runs-on), which apply the fixes below automatically and self-heal transient failures. This example uses latchkey-small; pick the runner size that fits the job.
Cache dependency installs on the setup step so they are served from cache.
Add a job timeout so a hung step cannot burn hours of runner time.

What Latchkey heals here

This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:

Dependency installs
Network fetches

This workflow runs 2 jobs per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow

actions/checkout actions/setup-python actions/upload-artifact actions/download-artifact