Skip to content
Latchkey

Upstream workflow (dask/dask-ml)

The Upstream workflow from dask/dask-ml, explained and optimized by Latchkey.

C

CI health: C - fair

Point runs-on at Latchkey and get run de-duplication, job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →
Source: dask/dask-ml.github/workflows/upstream.ymlLicense BSD-3-ClauseView source

What it does

This is the Upstream workflow from the dask/dask-ml repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)
name: Upstream

on:
  schedule:
    - cron: "0 1 * * *"
  push:
  pull_request:

jobs:

  check:
    runs-on: ubuntu-latest
    if: github.event_name == 'push' || github.event_name == 'pull_request'
    outputs:
      test-upstream: ${{ steps.detect-trigger.outputs.trigger-found }}
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 2
      - uses: xarray-contrib/ci-trigger@v1
        id: detect-trigger
        with:
          keyword: "test-upstream"

  build:
    needs: check
    runs-on: ubuntu-latest
    if: |
      always()
      && (
          needs.check.outputs.test-upstream == 'true'
          || (github.repository == 'dask/dask-ml' && github.event_name != 'pull_request')
      )

    env:
      COVERAGE: "true"

    steps:
      - name: Checkout source
        uses: actions/checkout@v2

      - name: Setup Conda Environment
        uses: conda-incubator/setup-miniconda@v3
        with:
          miniforge-version: latest
          use-mamba: true
          channel-priority: strict
          python-version: "3.12"
          environment-file: ci/environment-3.12.yaml
          activate-environment: test-environment
          auto-activate-base: false

      - name: Install
        shell: bash -l {0}
        env:
          UPSTREAM_DEV: 1
        run: source ci/install.sh

      - name: Run tests
        shell: bash -l {0}
        run: pytest -v

  report:
    name: report
    needs: build
    if: |
      always()
      && github.event_name != 'pull_request'
      && github.repository == 'dask/dask-ml'
      && needs.build.result == 'failure'
    runs-on: ubuntu-latest
    defaults:
      run:
        shell: bash
    steps:
      - uses: actions/checkout@v2
      - name: Report failures
        uses: actions/github-script@v3
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const title = "⚠️ Upstream CI failed ⚠️"
            const workflow_url = `https://github.com/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
            const issue_body = `[Workflow Run URL](${workflow_url})`
            // Run GraphQL query against GitHub API to find the most recent open issue used for reporting failures
            const query = `query($owner:String!, $name:String!, $creator:String!, $label:String!){
              repository(owner: $owner, name: $name) {
                issues(first: 1, states: OPEN, filterBy: {createdBy: $creator, labels: [$label]}, orderBy: {field: CREATED_AT, direction: DESC}) {
                  edges {
                    node {
                      body
                      id
                      number
                    }
                  }
                }
              }
            }`;
            const variables = {
                owner: context.repo.owner,
                name: context.repo.repo,
                label: 'upstream',
                creator: "github-actions[bot]"
            }
            const result = await github.graphql(query, variables)
            // If no issue is open, create a new issue,
            // else update the body of the existing issue.
            if (result.repository.issues.edges.length === 0) {
                github.issues.create({
                    owner: variables.owner,
                    repo: variables.name,
                    body: issue_body,
                    title: title,
                    labels: [variables.label]
                })
            } else {
                github.issues.update({
                    owner: variables.owner,
                    repo: variables.name,
                    issue_number: result.repository.issues.edges[0].node.number,
                    body: issue_body
                })
            }

The same workflow, on Latchkey

Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.

name: Upstream
 
on:
  schedule:
    - cron: "0 1 * * *"
  push:
  pull_request:
 
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
 
jobs:
 
  check:
    timeout-minutes: 30
    runs-on: latchkey-small
    if: github.event_name == 'push' || github.event_name == 'pull_request'
    outputs:
      test-upstream: ${{ steps.detect-trigger.outputs.trigger-found }}
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 2
      - uses: xarray-contrib/ci-trigger@v1
        id: detect-trigger
        with:
          keyword: "test-upstream"
 
  build:
    timeout-minutes: 30
    needs: check
    runs-on: latchkey-small
    if: |
      always()
      && (
          needs.check.outputs.test-upstream == 'true'
          || (github.repository == 'dask/dask-ml' && github.event_name != 'pull_request')
      )
 
    env:
      COVERAGE: "true"
 
    steps:
      - name: Checkout source
        uses: actions/checkout@v2
 
      - name: Setup Conda Environment
        uses: conda-incubator/setup-miniconda@v3
        with:
          miniforge-version: latest
          use-mamba: true
          channel-priority: strict
          python-version: "3.12"
          environment-file: ci/environment-3.12.yaml
          activate-environment: test-environment
          auto-activate-base: false
 
      - name: Install
        shell: bash -l {0}
        env:
          UPSTREAM_DEV: 1
        run: source ci/install.sh
 
      - name: Run tests
        shell: bash -l {0}
        run: pytest -v
 
  report:
    timeout-minutes: 30
    name: report
    needs: build
    if: |
      always()
      && github.event_name != 'pull_request'
      && github.repository == 'dask/dask-ml'
      && needs.build.result == 'failure'
    runs-on: latchkey-small
    defaults:
      run:
        shell: bash
    steps:
      - uses: actions/checkout@v2
      - name: Report failures
        uses: actions/github-script@v3
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const title = "⚠️ Upstream CI failed ⚠️"
            const workflow_url = `https://github.com/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
            const issue_body = `[Workflow Run URL](${workflow_url})`
            // Run GraphQL query against GitHub API to find the most recent open issue used for reporting failures
            const query = `query($owner:String!, $name:String!, $creator:String!, $label:String!){
              repository(owner: $owner, name: $name) {
                issues(first: 1, states: OPEN, filterBy: {createdBy: $creator, labels: [$label]}, orderBy: {field: CREATED_AT, direction: DESC}) {
                  edges {
                    node {
                      body
                      id
                      number
                    }
                  }
                }
              }
            }`;
            const variables = {
                owner: context.repo.owner,
                name: context.repo.repo,
                label: 'upstream',
                creator: "github-actions[bot]"
            }
            const result = await github.graphql(query, variables)
            // If no issue is open, create a new issue,
            // else update the body of the existing issue.
            if (result.repository.issues.edges.length === 0) {
                github.issues.create({
                    owner: variables.owner,
                    repo: variables.name,
                    body: issue_body,
                    title: title,
                    labels: [variables.label]
                })
            } else {
                github.issues.update({
                    owner: variables.owner,
                    repo: variables.name,
                    issue_number: result.repository.issues.edges[0].node.number,
                    body: issue_body
                })
            }
 

What changed

2 third-party actions are referenced by a movable tag. Pin them to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.

This workflow runs 3 jobs per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow