Upstream workflow (dask/dask-ml)
The Upstream workflow from dask/dask-ml, explained and optimized by Latchkey.
CI health: C - fair
Point runs-on at Latchkey and get run de-duplication, job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.
What it does
This is the Upstream workflow from the dask/dask-ml repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.
Below, Latchkey shows a faster, safer version produced by its optimization engine.
The workflow
name: Upstream
on:
schedule:
- cron: "0 1 * * *"
push:
pull_request:
jobs:
check:
runs-on: ubuntu-latest
if: github.event_name == 'push' || github.event_name == 'pull_request'
outputs:
test-upstream: ${{ steps.detect-trigger.outputs.trigger-found }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 2
- uses: xarray-contrib/ci-trigger@v1
id: detect-trigger
with:
keyword: "test-upstream"
build:
needs: check
runs-on: ubuntu-latest
if: |
always()
&& (
needs.check.outputs.test-upstream == 'true'
|| (github.repository == 'dask/dask-ml' && github.event_name != 'pull_request')
)
env:
COVERAGE: "true"
steps:
- name: Checkout source
uses: actions/checkout@v2
- name: Setup Conda Environment
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest
use-mamba: true
channel-priority: strict
python-version: "3.12"
environment-file: ci/environment-3.12.yaml
activate-environment: test-environment
auto-activate-base: false
- name: Install
shell: bash -l {0}
env:
UPSTREAM_DEV: 1
run: source ci/install.sh
- name: Run tests
shell: bash -l {0}
run: pytest -v
report:
name: report
needs: build
if: |
always()
&& github.event_name != 'pull_request'
&& github.repository == 'dask/dask-ml'
&& needs.build.result == 'failure'
runs-on: ubuntu-latest
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v2
- name: Report failures
uses: actions/github-script@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const title = "⚠️ Upstream CI failed ⚠️"
const workflow_url = `https://github.com/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
const issue_body = `[Workflow Run URL](${workflow_url})`
// Run GraphQL query against GitHub API to find the most recent open issue used for reporting failures
const query = `query($owner:String!, $name:String!, $creator:String!, $label:String!){
repository(owner: $owner, name: $name) {
issues(first: 1, states: OPEN, filterBy: {createdBy: $creator, labels: [$label]}, orderBy: {field: CREATED_AT, direction: DESC}) {
edges {
node {
body
id
number
}
}
}
}
}`;
const variables = {
owner: context.repo.owner,
name: context.repo.repo,
label: 'upstream',
creator: "github-actions[bot]"
}
const result = await github.graphql(query, variables)
// If no issue is open, create a new issue,
// else update the body of the existing issue.
if (result.repository.issues.edges.length === 0) {
github.issues.create({
owner: variables.owner,
repo: variables.name,
body: issue_body,
title: title,
labels: [variables.label]
})
} else {
github.issues.update({
owner: variables.owner,
repo: variables.name,
issue_number: result.repository.issues.edges[0].node.number,
body: issue_body
})
}
The same workflow, on Latchkey
Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.
name: Upstream on: schedule: - cron: "0 1 * * *" push: pull_request: concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: check: timeout-minutes: 30 runs-on: latchkey-small if: github.event_name == 'push' || github.event_name == 'pull_request' outputs: test-upstream: ${{ steps.detect-trigger.outputs.trigger-found }} steps: - uses: actions/checkout@v2 with: fetch-depth: 2 - uses: xarray-contrib/ci-trigger@v1 id: detect-trigger with: keyword: "test-upstream" build: timeout-minutes: 30 needs: check runs-on: latchkey-small if: | always() && ( needs.check.outputs.test-upstream == 'true' || (github.repository == 'dask/dask-ml' && github.event_name != 'pull_request') ) env: COVERAGE: "true" steps: - name: Checkout source uses: actions/checkout@v2 - name: Setup Conda Environment uses: conda-incubator/setup-miniconda@v3 with: miniforge-version: latest use-mamba: true channel-priority: strict python-version: "3.12" environment-file: ci/environment-3.12.yaml activate-environment: test-environment auto-activate-base: false - name: Install shell: bash -l {0} env: UPSTREAM_DEV: 1 run: source ci/install.sh - name: Run tests shell: bash -l {0} run: pytest -v report: timeout-minutes: 30 name: report needs: build if: | always() && github.event_name != 'pull_request' && github.repository == 'dask/dask-ml' && needs.build.result == 'failure' runs-on: latchkey-small defaults: run: shell: bash steps: - uses: actions/checkout@v2 - name: Report failures uses: actions/github-script@v3 with: github-token: ${{ secrets.GITHUB_TOKEN }} script: | const title = "⚠️ Upstream CI failed ⚠️" const workflow_url = `https://github.com/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}` const issue_body = `[Workflow Run URL](${workflow_url})` // Run GraphQL query against GitHub API to find the most recent open issue used for reporting failures const query = `query($owner:String!, $name:String!, $creator:String!, $label:String!){ repository(owner: $owner, name: $name) { issues(first: 1, states: OPEN, filterBy: {createdBy: $creator, labels: [$label]}, orderBy: {field: CREATED_AT, direction: DESC}) { edges { node { body id number } } } } }`; const variables = { owner: context.repo.owner, name: context.repo.repo, label: 'upstream', creator: "github-actions[bot]" } const result = await github.graphql(query, variables) // If no issue is open, create a new issue, // else update the body of the existing issue. if (result.repository.issues.edges.length === 0) { github.issues.create({ owner: variables.owner, repo: variables.name, body: issue_body, title: title, labels: [variables.label] }) } else { github.issues.update({ owner: variables.owner, repo: variables.name, issue_number: result.repository.issues.edges[0].node.number, body: issue_body }) }
What changed
- Run on Latchkey managed runners with one line (
runs-on), which apply the fixes below automatically and self-heal transient failures. This example useslatchkey-small; pick the runner size that fits the job. - Cancel superseded runs when a branch or PR gets a newer push.
- Add a job timeout so a hung step cannot burn hours of runner time.
2 third-party actions are referenced by a movable tag. Pin them to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.
This workflow runs 3 jobs per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.