Deploy workflow (leaningtech/webvm)
The Deploy workflow from leaningtech/webvm, explained and optimized by Latchkey.
CI health: B - good
Point runs-on at Latchkey and get job timeouts, SHA-pinned actions, self-healing for flaky steps, and up to 58% lower cost, applied automatically.
What it does
This is the Deploy workflow from the leaningtech/webvm repository, a real project running GitHub Actions. It is shown here with attribution under its Apache-2.0 license.
Below, Latchkey shows a faster, safer version produced by its optimization engine.
The workflow
name: Deploy
# Define when the workflow should run
on:
# Allow manual triggering of the workflow from the Actions tab
workflow_dispatch:
# Allow inputs to be passed when manually triggering the workflow from the Actions tab
inputs:
DOCKERFILE_PATH:
type: string
description: 'Path to the Dockerfile'
required: true
default: 'dockerfiles/debian_mini'
IMAGE_SIZE:
type: string
description: 'Image size, 950M max'
required: true
default: '750M'
DEPLOY_TO_GITHUB_PAGES:
type: boolean
description: 'Deploy to Github pages'
required: true
default: true
GITHUB_RELEASE:
type: boolean
description: 'Upload GitHub release'
required: true
default: false
jobs:
guard_clause:
runs-on: ubuntu-latest
env:
GH_TOKEN: ${{ github.token }} # As required by the GitHub-CLI
permissions:
actions: 'write' # Required in order to terminate the workflow run.
steps:
- uses: actions/checkout@v4
# Guard clause that cancels the workflow in case of an invalid DOCKERFILE_PATH and/or incorrectly configured Github Pages.
# The main reason for choosing this workaround for aborting the workflow is the fact that it does not display the workflow as successful, which can set false expectations.
- name: DOCKERFILE_PATH.
shell: bash
run: |
# We check whether the Dockerfile_path is valid.
if [ ! -f ${{ github.event.inputs.DOCKERFILE_PATH }} ]; then
echo "::error title=Invalid Dockerfile path::No file found at ${{ github.event.inputs.DOCKERFILE_PATH }}"
echo "terminate=true" >> $GITHUB_ENV
fi
- name: Github Pages config guard clause
if: ${{ github.event.inputs.DEPLOY_TO_GITHUB_PAGES == 'true' }}
run: |
# We use the Github Rest api to get information regarding pages for the Github Repository and store it into a temporary file named "pages_response".
set +e
gh api \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
/repos/${{ github.repository_owner }}/$(basename ${{ github.repository }})/pages > pages_response
# We make sure Github Pages has been enabled for this repository.
if [ "$?" -ne 0 ]; then
echo "::error title=Potential pages configuration error.::Please make sure you have enabled Github pages for the ${{ github.repository }} repository. If already enabled then Github pages might be down"
echo "terminate=true" >> $GITHUB_ENV
fi
set -e
# We make sure the Github pages build & deployment source is set to "workflow" (Github Actions). Instead of a "legacy" (branch).
if [[ "$(jq --compact-output --raw-output .build_type pages_response)" != "workflow" ]]; then
echo "Undefined behaviour, Make sure the Github Pages source is correctly configured in the Github Pages settings."
echo "::error title=Pages configuration error.::Please make sure you have correctly picked \"Github Actions\" as the build and deployment source for the Github Pages."
echo "terminate=true" >> $GITHUB_ENV
fi
rm pages_response
- name: Terminate run if error occurred.
run: |
if [[ $terminate == "true" ]]; then
gh run cancel ${{ github.run_id }}
gh run watch ${{ github.run_id }}
fi
build:
needs: guard_clause # Dependency
runs-on: ubuntu-latest # Image to run the worker on.
env:
TAG: "ext2-webvm-base-image" # Tag of docker image.
IMAGE_SIZE: '${{ github.event.inputs.IMAGE_SIZE }}'
DEPLOY_DIR: /webvm_deploy/ # Path to directory where we host the final image from.
permissions: # Permissions to grant the GITHUB_TOKEN.
contents: write # Required permission to make a github release.
steps:
# Checks-out our repository under $GITHUB_WORKSPACE, so our job can access it
- uses: actions/checkout@v4
# Setting the IMAGE_NAME variable in GITHUB_ENV to <Dockerfile name>_<date>_<run_id>.ext2.
- name: Generate the image_name.
id: image_name_gen
run: |
echo "IMAGE_NAME=$(basename ${{ github.event.inputs.DOCKERFILE_PATH }})_$(date +%Y%m%d)_${{ github.run_id }}.ext2" >> $GITHUB_ENV
# Create directory to host the image from.
- run: sudo mkdir -p $DEPLOY_DIR
# Build the i386 Dockerfile image.
- run: docker build . --tag $TAG --file ${{ github.event.inputs.DOCKERFILE_PATH }} --platform=i386
# Run the docker image so that we can export the container.
# Run the Docker container with the Google Public DNS nameservers: 8.8.8.8, 8.8.4.4
- run: |
docker run --dns 8.8.8.8 --dns 8.8.4.4 -d $TAG
echo "CONTAINER_ID=$(sudo docker ps -aq)" >> $GITHUB_ENV
# We extract the CMD, we first need to figure whether the Dockerfile uses CMD or an Entrypoint.
- name: Extracting CMD / Entrypoint and args
shell: bash
run: |
cmd=$(sudo docker inspect --format='{{json .Config.Cmd}}' $CONTAINER_ID)
entrypoint=$(sudo docker inspect --format='{{json .Config.Entrypoint}}' $CONTAINER_ID)
if [[ $entrypoint != "null" && $cmd != "null" ]]; then
echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint' )" >> $GITHUB_ENV
echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd' )" >> $GITHUB_ENV
elif [[ $cmd != "null" ]]; then
echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd[:1]' )" >> $GITHUB_ENV
echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd[1:]' )" >> $GITHUB_ENV
else
echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint[:1]' )" >> $GITHUB_ENV
echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint[1:]' )" >> $GITHUB_ENV
fi
# We extract the ENV, CMD/Entrypoint and cwd from the Docker container with docker inspect.
- name: Extracting env, args and cwd.
shell: bash
run: |
echo "ENV=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Env' )" >> $GITHUB_ENV
echo "CWD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.WorkingDir' )" >> $GITHUB_ENV
# We create and mount the base ext2 image to extract the Docker container's filesystem its contents into.
- name: Create ext2 image.
run: |
# Preallocate space for the ext2 image
sudo fallocate -l $IMAGE_SIZE ${IMAGE_NAME}
# Format to ext2 linux kernel revision 0
sudo mkfs.ext2 -r 0 ${IMAGE_NAME}
# Mount the ext2 image to modify it
sudo mount -o loop -t ext2 ${IMAGE_NAME} /mnt/
# We opt for 'docker cp --archive' over 'docker save' since our focus is solely on the end product rather than individual layers and metadata.
# However, it's important to note that despite being specified in the documentation, the '--archive' flag does not currently preserve uid/gid information when copying files from the container to the host machine.
# Another compelling reason to use 'docker cp' is that it preserves resolv.conf.
- name: Export and unpack container filesystem contents into mounted ext2 image.
run: |
sudo docker cp -a ${CONTAINER_ID}:/ /mnt/
sudo umount /mnt/
# Result is an ext2 image for webvm.
# The .txt suffix enabled HTTP compression for free
- name: Generate image split chunks and .meta file
run: |
sudo split ${{ env.IMAGE_NAME }} ${{ env.DEPLOY_DIR }}/${{ env.IMAGE_NAME }}.c -a 6 -b 128k -x --additional-suffix=.txt
sudo bash -c "stat -c%s ${{ env.IMAGE_NAME }} > ${{ env.DEPLOY_DIR }}/${{ env.IMAGE_NAME }}.meta"
# This step updates the default config_github_terminal.js file by performing the following actions:
# 1. Replaces all occurrences of IMAGE_URL with the URL to the image.
# 2. Replace CMD with the Dockerfile entry command.
# 3. Replace args with the Dockerfile CMD / Entrypoint args.
# 4. Replace ENV with the container's environment values.
# 5. Replace CWD with the container's current working directory.
- name: Adjust config_github_terminal.js
run: |
sed -i 's#IMAGE_URL#"${{ env.IMAGE_NAME }}"#g' config_github_terminal.js
sed -i 's#CMD#${{ env.CMD }}#g' config_github_terminal.js
sed -i 's#ARGS#${{ env.ARGS }}#g' config_github_terminal.js
sed -i 's#ENV#${{ env.ENV }}#g' config_github_terminal.js
sed -i 's#CWD#${{ env.CWD }}#g' config_github_terminal.js
- name: Build NPM package
run: |
npm install
WEBVM_MODE=github npm run build
# Move required files for gh-pages deployment to the deployment directory $DEPLOY_DIR.
- name: Copy build
run: |
rm build/alpine.html
sudo mv build/* $DEPLOY_DIR/
# We generate index.list files for our httpfs to function properly.
- name: make index.list
shell: bash
run: |
find $DEPLOY_DIR -type d | while read -r dir;
do
index_list="$dir/index.list";
sudo rm -f "$index_list";
sudo ls "$dir" | sudo tee "$index_list" > /dev/null;
sudo chmod +rw "$index_list";
sudo echo "created $index_list";
done
# Create a gh-pages artifact in order to deploy to gh-pages.
- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@v3
with:
# Path of the directory containing the static assets for our gh pages deployment.
path: ${{ env.DEPLOY_DIR }} # optional, default is _site/
- name: github release # To upload our final ext2 image as a github release.
if: ${{ github.event.inputs.GITHUB_RELEASE == 'true' }}
uses: softprops/action-gh-release@v2
with:
target_commitish: ${{ github.sha }} # Last commit on the GITHUB_REF branch or tag
tag_name: ext2_image
fail_on_unmatched_files: 'true' # Fail in case of no matches with the file(s) glob(s).
files: | # Assets to upload as release.
${{ env.IMAGE_NAME }}
deploy_to_github_pages: # Job that deploys the github-pages artifact to github-pages.
if: ${{ github.event.inputs.DEPLOY_TO_GITHUB_PAGES == 'true' }}
needs: build
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
# Grant GITHUB_TOKEN the permissions required to make a Pages deployment
permissions:
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source
runs-on: ubuntu-latest
steps:
# Deployment to github pages
- name: Deploy GitHub Pages site
id: deployment
uses: actions/deploy-pages@v4
The same workflow, on Latchkey
Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.
name: Deploy # Define when the workflow should run on: # Allow manual triggering of the workflow from the Actions tab workflow_dispatch: # Allow inputs to be passed when manually triggering the workflow from the Actions tab inputs: DOCKERFILE_PATH: type: string description: 'Path to the Dockerfile' required: true default: 'dockerfiles/debian_mini' IMAGE_SIZE: type: string description: 'Image size, 950M max' required: true default: '750M' DEPLOY_TO_GITHUB_PAGES: type: boolean description: 'Deploy to Github pages' required: true default: true GITHUB_RELEASE: type: boolean description: 'Upload GitHub release' required: true default: false jobs: guard_clause: timeout-minutes: 30 runs-on: latchkey-small env: GH_TOKEN: ${{ github.token }} # As required by the GitHub-CLI permissions: actions: 'write' # Required in order to terminate the workflow run. steps: - uses: actions/checkout@v4 # Guard clause that cancels the workflow in case of an invalid DOCKERFILE_PATH and/or incorrectly configured Github Pages. # The main reason for choosing this workaround for aborting the workflow is the fact that it does not display the workflow as successful, which can set false expectations. - name: DOCKERFILE_PATH. shell: bash run: | # We check whether the Dockerfile_path is valid. if [ ! -f ${{ github.event.inputs.DOCKERFILE_PATH }} ]; then echo "::error title=Invalid Dockerfile path::No file found at ${{ github.event.inputs.DOCKERFILE_PATH }}" echo "terminate=true" >> $GITHUB_ENV fi - name: Github Pages config guard clause if: ${{ github.event.inputs.DEPLOY_TO_GITHUB_PAGES == 'true' }} run: | # We use the Github Rest api to get information regarding pages for the Github Repository and store it into a temporary file named "pages_response". set +e gh api \ -H "Accept: application/vnd.github+json" \ -H "X-GitHub-Api-Version: 2022-11-28" \ /repos/${{ github.repository_owner }}/$(basename ${{ github.repository }})/pages > pages_response # We make sure Github Pages has been enabled for this repository. if [ "$?" -ne 0 ]; then echo "::error title=Potential pages configuration error.::Please make sure you have enabled Github pages for the ${{ github.repository }} repository. If already enabled then Github pages might be down" echo "terminate=true" >> $GITHUB_ENV fi set -e # We make sure the Github pages build & deployment source is set to "workflow" (Github Actions). Instead of a "legacy" (branch). if [[ "$(jq --compact-output --raw-output .build_type pages_response)" != "workflow" ]]; then echo "Undefined behaviour, Make sure the Github Pages source is correctly configured in the Github Pages settings." echo "::error title=Pages configuration error.::Please make sure you have correctly picked \"Github Actions\" as the build and deployment source for the Github Pages." echo "terminate=true" >> $GITHUB_ENV fi rm pages_response - name: Terminate run if error occurred. run: | if [[ $terminate == "true" ]]; then gh run cancel ${{ github.run_id }} gh run watch ${{ github.run_id }} fi build: timeout-minutes: 30 needs: guard_clause # Dependency runs-on: latchkey-small # Image to run the worker on. env: TAG: "ext2-webvm-base-image" # Tag of docker image. IMAGE_SIZE: '${{ github.event.inputs.IMAGE_SIZE }}' DEPLOY_DIR: /webvm_deploy/ # Path to directory where we host the final image from. permissions: # Permissions to grant the GITHUB_TOKEN. contents: write # Required permission to make a github release. steps: # Checks-out our repository under $GITHUB_WORKSPACE, so our job can access it - uses: actions/checkout@v4 # Setting the IMAGE_NAME variable in GITHUB_ENV to <Dockerfile name>_<date>_<run_id>.ext2. - name: Generate the image_name. id: image_name_gen run: | echo "IMAGE_NAME=$(basename ${{ github.event.inputs.DOCKERFILE_PATH }})_$(date +%Y%m%d)_${{ github.run_id }}.ext2" >> $GITHUB_ENV # Create directory to host the image from. - run: sudo mkdir -p $DEPLOY_DIR # Build the i386 Dockerfile image. - run: docker build . --tag $TAG --file ${{ github.event.inputs.DOCKERFILE_PATH }} --platform=i386 # Run the docker image so that we can export the container. # Run the Docker container with the Google Public DNS nameservers: 8.8.8.8, 8.8.4.4 - run: | docker run --dns 8.8.8.8 --dns 8.8.4.4 -d $TAG echo "CONTAINER_ID=$(sudo docker ps -aq)" >> $GITHUB_ENV # We extract the CMD, we first need to figure whether the Dockerfile uses CMD or an Entrypoint. - name: Extracting CMD / Entrypoint and args shell: bash run: | cmd=$(sudo docker inspect --format='{{json .Config.Cmd}}' $CONTAINER_ID) entrypoint=$(sudo docker inspect --format='{{json .Config.Entrypoint}}' $CONTAINER_ID) if [[ $entrypoint != "null" && $cmd != "null" ]]; then echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint' )" >> $GITHUB_ENV echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd' )" >> $GITHUB_ENV elif [[ $cmd != "null" ]]; then echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd[:1]' )" >> $GITHUB_ENV echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Cmd[1:]' )" >> $GITHUB_ENV else echo "CMD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint[:1]' )" >> $GITHUB_ENV echo "ARGS=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Entrypoint[1:]' )" >> $GITHUB_ENV fi # We extract the ENV, CMD/Entrypoint and cwd from the Docker container with docker inspect. - name: Extracting env, args and cwd. shell: bash run: | echo "ENV=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.Env' )" >> $GITHUB_ENV echo "CWD=$( sudo docker inspect $CONTAINER_ID | jq --compact-output '.[0].Config.WorkingDir' )" >> $GITHUB_ENV # We create and mount the base ext2 image to extract the Docker container's filesystem its contents into. - name: Create ext2 image. run: | # Preallocate space for the ext2 image sudo fallocate -l $IMAGE_SIZE ${IMAGE_NAME} # Format to ext2 linux kernel revision 0 sudo mkfs.ext2 -r 0 ${IMAGE_NAME} # Mount the ext2 image to modify it sudo mount -o loop -t ext2 ${IMAGE_NAME} /mnt/ # We opt for 'docker cp --archive' over 'docker save' since our focus is solely on the end product rather than individual layers and metadata. # However, it's important to note that despite being specified in the documentation, the '--archive' flag does not currently preserve uid/gid information when copying files from the container to the host machine. # Another compelling reason to use 'docker cp' is that it preserves resolv.conf. - name: Export and unpack container filesystem contents into mounted ext2 image. run: | sudo docker cp -a ${CONTAINER_ID}:/ /mnt/ sudo umount /mnt/ # Result is an ext2 image for webvm. # The .txt suffix enabled HTTP compression for free - name: Generate image split chunks and .meta file run: | sudo split ${{ env.IMAGE_NAME }} ${{ env.DEPLOY_DIR }}/${{ env.IMAGE_NAME }}.c -a 6 -b 128k -x --additional-suffix=.txt sudo bash -c "stat -c%s ${{ env.IMAGE_NAME }} > ${{ env.DEPLOY_DIR }}/${{ env.IMAGE_NAME }}.meta" # This step updates the default config_github_terminal.js file by performing the following actions: # 1. Replaces all occurrences of IMAGE_URL with the URL to the image. # 2. Replace CMD with the Dockerfile entry command. # 3. Replace args with the Dockerfile CMD / Entrypoint args. # 4. Replace ENV with the container's environment values. # 5. Replace CWD with the container's current working directory. - name: Adjust config_github_terminal.js run: | sed -i 's#IMAGE_URL#"${{ env.IMAGE_NAME }}"#g' config_github_terminal.js sed -i 's#CMD#${{ env.CMD }}#g' config_github_terminal.js sed -i 's#ARGS#${{ env.ARGS }}#g' config_github_terminal.js sed -i 's#ENV#${{ env.ENV }}#g' config_github_terminal.js sed -i 's#CWD#${{ env.CWD }}#g' config_github_terminal.js - name: Build NPM package run: | npm install WEBVM_MODE=github npm run build # Move required files for gh-pages deployment to the deployment directory $DEPLOY_DIR. - name: Copy build run: | rm build/alpine.html sudo mv build/* $DEPLOY_DIR/ # We generate index.list files for our httpfs to function properly. - name: make index.list shell: bash run: | find $DEPLOY_DIR -type d | while read -r dir; do index_list="$dir/index.list"; sudo rm -f "$index_list"; sudo ls "$dir" | sudo tee "$index_list" > /dev/null; sudo chmod +rw "$index_list"; sudo echo "created $index_list"; done # Create a gh-pages artifact in order to deploy to gh-pages. - name: Upload GitHub Pages artifact uses: actions/upload-pages-artifact@v3 with: # Path of the directory containing the static assets for our gh pages deployment. path: ${{ env.DEPLOY_DIR }} # optional, default is _site/ - name: github release # To upload our final ext2 image as a github release. if: ${{ github.event.inputs.GITHUB_RELEASE == 'true' }} uses: softprops/action-gh-release@v2 with: target_commitish: ${{ github.sha }} # Last commit on the GITHUB_REF branch or tag tag_name: ext2_image fail_on_unmatched_files: 'true' # Fail in case of no matches with the file(s) glob(s). files: | # Assets to upload as release. ${{ env.IMAGE_NAME }} deploy_to_github_pages: # Job that deploys the github-pages artifact to github-pages. timeout-minutes: 30 if: ${{ github.event.inputs.DEPLOY_TO_GITHUB_PAGES == 'true' }} needs: build environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} # Grant GITHUB_TOKEN the permissions required to make a Pages deployment permissions: pages: write # to deploy to Pages id-token: write # to verify the deployment originates from an appropriate source runs-on: latchkey-small steps: # Deployment to github pages - name: Deploy GitHub Pages site id: deployment uses: actions/deploy-pages@v4
What changed
- Run on Latchkey managed runners with one line (
runs-on), which apply the fixes below automatically and self-heal transient failures. This example useslatchkey-small; pick the runner size that fits the job. - Add a job timeout so a hung step cannot burn hours of runner time.
1 third-party action is referenced by a movable tag. Pin it to the commit SHA (Latchkey resolves and applies this automatically) so a repointed tag cannot change what runs.
What Latchkey heals here
This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:
- Dependency installs
- Container pulls and builds
This workflow runs 3 jobs per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.