Performance Benchmark workflow (cocotb/cocotb)

The Performance Benchmark workflow from cocotb/cocotb, explained and optimized by Latchkey.

CI health: C - fair

Point runs-on at Latchkey and get caching, job timeouts, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →

Source: cocotb/cocotb.github/workflows/benchmark.ymlLicense BSD-3-ClauseView source

What it does

This is the Performance Benchmark workflow from the cocotb/cocotb repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)

name: Performance Benchmark
# adapted from https://github.com/benchmark-action/github-action-benchmark#charts-on-github-pages-1

concurrency:
  group: ${{ github.workflow }}-${{ github.event.number }}-${{ github.event.ref }}
  cancel-in-progress: ${{ !(contains(github.ref, 'release/') || contains(github.ref, 'master')) }}

on:
  push:
    branches:
    - master
  pull_request:
    branches:
      - master

jobs:

  tests:
    if: github.repository == 'cocotb/cocotb'
    name: Python ${{matrix.python-version}}
    runs-on: ubuntu-22.04

    strategy:
      fail-fast: false
      matrix:
        include:
          - python-version: 3.9

    steps:
    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
    - name: Set up Python ${{matrix.python-version}}
      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1  # v6.3.0
      with:
        python-version: ${{matrix.python-version}}

    - name: Install Icarus Verilog
      run: |
        sudo apt-get install -y --no-install-recommends iverilog

    - name: Set up NVC (Ubuntu)
      run: |
        sudo apt-get install -y --no-install-recommends llvm-dev libdw-dev flex libzstd-dev pkg-config
        git clone https://github.com/nickg/nvc.git
        cd nvc
        git reset --hard r1.16.0
        ./autogen.sh
        mkdir build
        cd build
        ../configure
        make -j $(nproc)
        sudo make install

    - name: Run benchmark
      run: |
        pip install pytest pytest-benchmark
        pip install .
        pytest -c /dev/null tests/benchmarks --benchmark-json output.json

    # Pushing the benchmark requires elevated permissions to the
    # cocotb/cocotb-benchmark-results repository, which we only grant for
    # master builds, not for PR builds.
    - name: Generate a token to access cocotb/cocotb-benchmark-results
      if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
      id: generate_token
      uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1  # v3.2.0
      with:
        app-id: ${{ secrets.COCOTB_CI_REPOACCESS_APP_ID }}
        private-key: ${{ secrets.COCOTB_CI_REPOACCESS_APP_PRIVATE_KEY }}
        owner: ${{ github.repository_owner }}
        repositories: cocotb-benchmark-results

    - name: Store benchmark result
      if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
      uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba  # v1.22.1
      continue-on-error: true
      with:
        tool: 'pytest'
        output-file-path: output.json
        alert-threshold: '120%'
        fail-on-alert: true
        github-token: ${{ steps.generate_token.outputs.token }}
        auto-push: true
        gh-repository: 'github.com/cocotb/cocotb-benchmark-results'

The same workflow, on Latchkey

Estimated ~20% faster on cache hits, plus fewer wasted runs and a safer supply chain. Added and changed lines are highlighted.

name: Performance Benchmark
# adapted from https://github.com/benchmark-action/github-action-benchmark#charts-on-github-pages-1
 
concurrency:
  group: ${{ github.workflow }}-${{ github.event.number }}-${{ github.event.ref }}
  cancel-in-progress: ${{ !(contains(github.ref, 'release/') || contains(github.ref, 'master')) }}
 
on:
  push:
    branches:
    - master
  pull_request:
    branches:
      - master
 
jobs:
 
  tests:
    timeout-minutes: 30
    if: github.repository == 'cocotb/cocotb'
    name: Python ${{matrix.python-version}}
    runs-on: latchkey-small
 
    strategy:
      fail-fast: false
      matrix:
        include:
          - python-version: 3.9
 
    steps:
    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
    - name: Set up Python ${{matrix.python-version}}
      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1  # v6.3.0
      with:
        cache: 'pip'
        python-version: ${{matrix.python-version}}
 
    - name: Install Icarus Verilog
      run: |
        sudo apt-get install -y --no-install-recommends iverilog
 
    - name: Set up NVC (Ubuntu)
      run: |
        sudo apt-get install -y --no-install-recommends llvm-dev libdw-dev flex libzstd-dev pkg-config
        git clone https://github.com/nickg/nvc.git
        cd nvc
        git reset --hard r1.16.0
        ./autogen.sh
        mkdir build
        cd build
        ../configure
        make -j $(nproc)
        sudo make install
 
    - name: Run benchmark
      run: |
        pip install pytest pytest-benchmark
        pip install .
        pytest -c /dev/null tests/benchmarks --benchmark-json output.json
 
    # Pushing the benchmark requires elevated permissions to the
    # cocotb/cocotb-benchmark-results repository, which we only grant for
    # master builds, not for PR builds.
    - name: Generate a token to access cocotb/cocotb-benchmark-results
      if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
      id: generate_token
      uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1  # v3.2.0
      with:
        app-id: ${{ secrets.COCOTB_CI_REPOACCESS_APP_ID }}
        private-key: ${{ secrets.COCOTB_CI_REPOACCESS_APP_PRIVATE_KEY }}
        owner: ${{ github.repository_owner }}
        repositories: cocotb-benchmark-results
 
    - name: Store benchmark result
      if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
      uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba  # v1.22.1
      continue-on-error: true
      with:
        tool: 'pytest'
        output-file-path: output.json
        alert-threshold: '120%'
        fail-on-alert: true
        github-token: ${{ steps.generate_token.outputs.token }}
        auto-push: true
        gh-repository: 'github.com/cocotb/cocotb-benchmark-results'

What changed

Run on Latchkey managed runners with one line (runs-on), which apply the fixes below automatically and self-heal transient failures. This example uses latchkey-small; pick the runner size that fits the job.
Cache dependency installs on the setup step so they are served from cache.
Add a job timeout so a hung step cannot burn hours of runner time.

What Latchkey heals here

This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:

Dependency installs

This workflow runs 1 job per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow

actions/checkout actions/setup-python actions/create-github-app-token