Build Data workflow (2020PB/police-brutality)
The Build Data workflow from 2020PB/police-brutality, explained and optimized by Latchkey.
CI health: F - at risk
Point runs-on at Latchkey and get caching, run de-duplication, job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.
What it does
This is the Build Data workflow from the 2020PB/police-brutality repository, a real project running GitHub Actions. It is shown here with attribution under its MIT license.
Below, Latchkey shows a faster, safer version produced by its optimization engine.
The workflow
# This is a basic workflow to help you get started with Actions
name: Build Data
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the main branch
on:
push:
branches: [ main ]
# Pausing the build on pull_request because it seems these cause a duplicate build which can fail.
# I added this event because it seemed sometimes `push` was not causing a build.
# So I'm leaving these comments until it's verified that `push` alone is good.
#pull_request:
# types: [ closed ]
# branches: [ main ]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks the branch out under $GITHUB_WORKSPACE, so build_data can access it
- uses: actions/checkout@v2
- name: Set up Python 3.7
uses: actions/setup-python@v2
with:
# Semantic version range syntax or exact version of a Python version
python-version: '3.7'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r tools/requirements.txt
python -c "import nltk; nltk.download('wordnet')"
- name: Run unit tests with pytest
run: |
cd tools
pytest
cd ..
- name: Create json csv md files
run: python tools/data_builder.py
# Deploy data to local repo
- name: Deploy
uses: s0/git-publish-subdir-action@master
env:
REPO: self
BRANCH: data_build
FOLDER: tools/data_build
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The same workflow, on Latchkey
Estimated ~20% faster on cache hits, plus fewer wasted runs and a safer supply chain. Added and changed lines are highlighted.
# This is a basic workflow to help you get started with Actions name: Build Data # Controls when the action will run. Triggers the workflow on push or pull request # events but only for the main branch on: push: branches: [ main ] # Pausing the build on pull_request because it seems these cause a duplicate build which can fail. # I added this event because it seemed sometimes `push` was not causing a build. # So I'm leaving these comments until it's verified that `push` alone is good. #pull_request: # types: [ closed ] # branches: [ main ] # A workflow run is made up of one or more jobs that can run sequentially or in parallel concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: # This workflow contains a single job called "build" build: timeout-minutes: 30 # The type of runner that the job will run on runs-on: latchkey-small # Steps represent a sequence of tasks that will be executed as part of the job steps: # Checks the branch out under $GITHUB_WORKSPACE, so build_data can access it - uses: actions/checkout@v2 - name: Set up Python 3.7 uses: actions/setup-python@v2 with: cache: 'pip' # Semantic version range syntax or exact version of a Python version python-version: '3.7' - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r tools/requirements.txt python -c "import nltk; nltk.download('wordnet')" - name: Run unit tests with pytest run: | cd tools pytest cd .. - name: Create json csv md files run: python tools/data_builder.py # Deploy data to local repo - name: Deploy uses: s0/git-publish-subdir-action@master env: REPO: self BRANCH: data_build FOLDER: tools/data_build GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
What changed
- Run on Latchkey managed runners with one line (
runs-on), which apply the fixes below automatically and self-heal transient failures. This example useslatchkey-small; pick the runner size that fits the job. - Cancel superseded runs when a branch or PR gets a newer push.
- Cache dependency installs on the setup step so they are served from cache.
- Add a job timeout so a hung step cannot burn hours of runner time.
1 third-party action is referenced by a movable tag. Pin it to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.
What Latchkey heals here
This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:
- Dependency installs
This workflow runs 1 job per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.