Build Dynamic Embedding Wheels workflow (meta-pytorch/torchrec)

The Build Dynamic Embedding Wheels workflow from meta-pytorch/torchrec, explained and optimized by Latchkey.

CI health: C - fair

Point runs-on at Latchkey and get run de-duplication, job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →

Source: meta-pytorch/torchrec.github/workflows/build_dynamic_embedding_wheels.ymlLicense BSD-3-ClauseView source

What it does

This is the Build Dynamic Embedding Wheels workflow from the meta-pytorch/torchrec repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)

name: Build Dynamic Embedding Wheels

on:
  workflow_dispatch:
  pull_request:
    paths:
      - "contrib/*"
    branches:
      - main
  push:
    branches:
      - main
  release:
    types:
      - published

jobs:
  build_wheels:
    name: Wheels on ${{ matrix.os }}/${{ matrix.pyver }}/cu${{ matrix.cuver }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ ubuntu-latest ]
        pyver: [ cp39, cp310, cp311, cp312 ]
        cuver: [ "12.1", "12.4"]

    steps:
      -
        name: Check disk space
        run: df . -h

      - name: Remove unnecessary files
        run: |
          sudo rm -rf /usr/share/dotnet
          sudo rm -rf "$AGENT_TOOLSDIRECTORY"

      -
        name: Check disk space
        run: df . -h

      - uses: actions/checkout@v4
        with:
          submodules: recursive

      - uses: pypa/cibuildwheel@v2.20.0
        with:
          package-dir: contrib/dynamic_embedding
        env:
          CIBW_BEFORE_BUILD: "env CUDA_VERSION=${{ matrix.cuver }} contrib/dynamic_embedding/tools/before_linux_build.sh"
          CIBW_BUILD: "${{ matrix.pyver }}-manylinux_x86_64"
          CIBW_REPAIR_WHEEL_COMMAND: "env CUDA_VERSION=${{ matrix.cuver }} contrib/dynamic_embedding/tools/repair_wheel.sh {wheel} {dest_dir}"
          CIBW_MANYLINUX_X86_64_IMAGE: "manylinux_2_28"

      - name: Verify clean directory
        run: git diff --exit-code
        shell: bash

      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: artifact-${{ matrix.os }}-${{ matrix.pyver }}-cu${{ matrix.cuver }}
          path: wheelhouse/*.whl

  merge:
    runs-on: ubuntu-latest
    needs: build_wheels
    steps:
      - name: Merge Artifacts
        uses: actions/upload-artifact/merge@v4
        with:
          name: artifact
          pattern: artifact-*

The same workflow, on Latchkey

Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.

name: Build Dynamic Embedding Wheels
 
on:
  workflow_dispatch:
  pull_request:
    paths:
      - "contrib/*"
    branches:
      - main
  push:
    branches:
      - main
  release:
    types:
      - published
 
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
 
jobs:
  build_wheels:
    timeout-minutes: 30
    name: Wheels on ${{ matrix.os }}/${{ matrix.pyver }}/cu${{ matrix.cuver }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ ubuntu-latest ]
        pyver: [ cp39, cp310, cp311, cp312 ]
        cuver: [ "12.1", "12.4"]
 
    steps:
      -
        name: Check disk space
        run: df . -h
 
      - name: Remove unnecessary files
        run: |
          sudo rm -rf /usr/share/dotnet
          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
 
      -
        name: Check disk space
        run: df . -h
 
      - uses: actions/checkout@v4
        with:
          submodules: recursive
 
      - uses: pypa/cibuildwheel@v2.20.0
        with:
          package-dir: contrib/dynamic_embedding
        env:
          CIBW_BEFORE_BUILD: "env CUDA_VERSION=${{ matrix.cuver }} contrib/dynamic_embedding/tools/before_linux_build.sh"
          CIBW_BUILD: "${{ matrix.pyver }}-manylinux_x86_64"
          CIBW_REPAIR_WHEEL_COMMAND: "env CUDA_VERSION=${{ matrix.cuver }} contrib/dynamic_embedding/tools/repair_wheel.sh {wheel} {dest_dir}"
          CIBW_MANYLINUX_X86_64_IMAGE: "manylinux_2_28"
 
      - name: Verify clean directory
        run: git diff --exit-code
        shell: bash
 
      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: artifact-${{ matrix.os }}-${{ matrix.pyver }}-cu${{ matrix.cuver }}
          path: wheelhouse/*.whl
 
  merge:
    timeout-minutes: 30
    runs-on: latchkey-small
    needs: build_wheels
    steps:
      - name: Merge Artifacts
        uses: actions/upload-artifact/merge@v4
        with:
          name: artifact
          pattern: artifact-*

What changed

Run on Latchkey managed runners with one line (runs-on), which apply the fixes below automatically and self-heal transient failures. This example uses latchkey-small; pick the runner size that fits the job.
Cancel superseded runs when a branch or PR gets a newer push.
Add a job timeout so a hung step cannot burn hours of runner time.

1 third-party action is referenced by a movable tag. Pin it to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.

This workflow runs 2 jobs (9 with the matrix expanded) per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow

actions/checkout actions/upload-artifact