Build and Publish Base Docker Images workflow (SpecterOps/Nemesis)

The Build and Publish Base Docker Images workflow from SpecterOps/Nemesis, explained and optimized by Latchkey.

CI health: C - fair

Point runs-on at Latchkey and get run de-duplication, job timeouts, self-healing for flaky steps, and up to 58% lower cost, applied automatically.

Grade your own workflow free or run it on Latchkey →

Source: SpecterOps/Nemesis.github/workflows/docker-build-base.ymlLicense BSD-3-ClauseView source

What it does

This is the Build and Publish Base Docker Images workflow from the SpecterOps/Nemesis repository, a real project running GitHub Actions. It is shown here with attribution under its BSD-3-Clause license.

Below, Latchkey shows a faster, safer version produced by its optimization engine.

The workflow

workflow (.yml)

name: Build and Publish Base Docker Images

on:
  push:
    branches: [ "main" ]
    paths:
      - 'infra/docker/python_base/**'
  workflow_dispatch:

env:
  REGISTRY: ghcr.io
  IMAGE_PREFIX: "specterops/nemesis" # ${{ github.repository }} causes issues as SpecterOps is not all lowercase

jobs:
  build-base-images:
    runs-on: ${{ matrix.runner }}
    permissions:
      contents: read
      packages: write
    strategy:
      matrix:
        include:
          - runner: ubuntu-22.04
            platform: linux/amd64
            arch: amd64
          - runner: ubuntu-22.04-arm
            platform: linux/arm64
            arch: arm64
    outputs:
      python-base-dev-tag: ${{ steps.meta-python-base-dev.outputs.version }}
      python-base-prod-tag: ${{ steps.meta-python-base-prod.outputs.version }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4

      - name: Log in to the Container registry
        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # Python Base Dev Image
      - name: Extract metadata for Python base dev image
        id: meta-python-base-dev
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-dev
          tags: |
            type=sha,format=short,suffix=-${{ matrix.arch }}
            type=ref,event=branch,suffix=-${{ matrix.arch }}
            type=raw,value=latest-${{ matrix.arch }},enable={{is_default_branch}}

      - name: Build and push Python base dev image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
        with:
          context: ./infra/docker/python_base
          file: ./infra/docker/python_base/dev.Dockerfile
          push: true
          platforms: ${{ matrix.platform }}
          tags: ${{ steps.meta-python-base-dev.outputs.tags }}
          labels: ${{ steps.meta-python-base-dev.outputs.labels }}
          cache-from: type=gha,scope=python-base-dev-${{ matrix.arch }}
          cache-to: type=gha,mode=max,scope=python-base-dev-${{ matrix.arch }}

      # Python Base Prod Image
      - name: Extract metadata for Python base prod image
        id: meta-python-base-prod
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-prod
          tags: |
            type=sha,format=short,suffix=-${{ matrix.arch }}
            type=ref,event=branch,suffix=-${{ matrix.arch }}
            type=raw,value=latest-${{ matrix.arch }},enable={{is_default_branch}}

      - name: Build and push Python base prod image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
        with:
          context: ./infra/docker/python_base
          file: ./infra/docker/python_base/prod.Dockerfile
          push: true
          platforms: ${{ matrix.platform }}
          tags: ${{ steps.meta-python-base-prod.outputs.tags }}
          labels: ${{ steps.meta-python-base-prod.outputs.labels }}
          cache-from: type=gha,scope=python-base-prod-${{ matrix.arch }}
          cache-to: type=gha,mode=max,scope=python-base-prod-${{ matrix.arch }}

  # Create multi-arch manifests
  create-manifests:
    needs: build-base-images
    runs-on: ubuntu-22.04
    permissions:
      contents: read
      packages: write

    steps:
      - name: Log in to the Container registry
        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # Python Base Dev Manifest
      - name: Extract metadata for Python base dev manifest
        id: meta-python-base-dev
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-dev
          tags: |
            type=sha,format=short
            type=ref,event=branch
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Create and push Python base dev multi-arch manifest
        run: |
          echo '${{ steps.meta-python-base-dev.outputs.tags }}' | while IFS= read -r tag; do
            if [ -n "$tag" ]; then
              echo "Creating manifest for: $tag"
              docker buildx imagetools create \
                --tag "$tag" \
                "${tag}-amd64" \
                "${tag}-arm64"
            fi
          done

      # Python Base Prod Manifest
      - name: Extract metadata for Python base prod manifest
        id: meta-python-base-prod
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-prod
          tags: |
            type=sha,format=short
            type=ref,event=branch
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Create and push Python base prod multi-arch manifest
        run: |
          echo '${{ steps.meta-python-base-prod.outputs.tags }}' | while IFS= read -r tag; do
            if [ -n "$tag" ]; then
              echo "Creating manifest for: $tag"
              docker buildx imagetools create \
                --tag "$tag" \
                "${tag}-amd64" \
                "${tag}-arm64"
            fi
          done

The same workflow, on Latchkey

Removes redundant runs and caps runaway jobs. Added and changed lines are highlighted.

name: Build and Publish Base Docker Images
 
on:
  push:
    branches: [ "main" ]
    paths:
      - 'infra/docker/python_base/**'
  workflow_dispatch:
 
env:
  REGISTRY: ghcr.io
  IMAGE_PREFIX: "specterops/nemesis" # ${{ github.repository }} causes issues as SpecterOps is not all lowercase
 
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
 
jobs:
  build-base-images:
    timeout-minutes: 30
    runs-on: ${{ matrix.runner }}
    permissions:
      contents: read
      packages: write
    strategy:
      matrix:
        include:
          - runner: ubuntu-22.04
            platform: linux/amd64
            arch: amd64
          - runner: ubuntu-22.04-arm
            platform: linux/arm64
            arch: arm64
    outputs:
      python-base-dev-tag: ${{ steps.meta-python-base-dev.outputs.version }}
      python-base-prod-tag: ${{ steps.meta-python-base-prod.outputs.version }}
 
    steps:
      - name: Checkout repository
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
 
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4
 
      - name: Log in to the Container registry
        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
 
      # Python Base Dev Image
      - name: Extract metadata for Python base dev image
        id: meta-python-base-dev
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-dev
          tags: |
            type=sha,format=short,suffix=-${{ matrix.arch }}
            type=ref,event=branch,suffix=-${{ matrix.arch }}
            type=raw,value=latest-${{ matrix.arch }},enable={{is_default_branch}}
 
      - name: Build and push Python base dev image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
        with:
          context: ./infra/docker/python_base
          file: ./infra/docker/python_base/dev.Dockerfile
          push: true
          platforms: ${{ matrix.platform }}
          tags: ${{ steps.meta-python-base-dev.outputs.tags }}
          labels: ${{ steps.meta-python-base-dev.outputs.labels }}
          cache-from: type=gha,scope=python-base-dev-${{ matrix.arch }}
          cache-to: type=gha,mode=max,scope=python-base-dev-${{ matrix.arch }}
 
      # Python Base Prod Image
      - name: Extract metadata for Python base prod image
        id: meta-python-base-prod
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-prod
          tags: |
            type=sha,format=short,suffix=-${{ matrix.arch }}
            type=ref,event=branch,suffix=-${{ matrix.arch }}
            type=raw,value=latest-${{ matrix.arch }},enable={{is_default_branch}}
 
      - name: Build and push Python base prod image
        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
        with:
          context: ./infra/docker/python_base
          file: ./infra/docker/python_base/prod.Dockerfile
          push: true
          platforms: ${{ matrix.platform }}
          tags: ${{ steps.meta-python-base-prod.outputs.tags }}
          labels: ${{ steps.meta-python-base-prod.outputs.labels }}
          cache-from: type=gha,scope=python-base-prod-${{ matrix.arch }}
          cache-to: type=gha,mode=max,scope=python-base-prod-${{ matrix.arch }}
 
  # Create multi-arch manifests
  create-manifests:
    timeout-minutes: 30
    needs: build-base-images
    runs-on: latchkey-small
    permissions:
      contents: read
      packages: write
 
    steps:
      - name: Log in to the Container registry
        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
 
      # Python Base Dev Manifest
      - name: Extract metadata for Python base dev manifest
        id: meta-python-base-dev
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-dev
          tags: |
            type=sha,format=short
            type=ref,event=branch
            type=raw,value=latest,enable={{is_default_branch}}
 
      - name: Create and push Python base dev multi-arch manifest
        run: |
          echo '${{ steps.meta-python-base-dev.outputs.tags }}' | while IFS= read -r tag; do
            if [ -n "$tag" ]; then
              echo "Creating manifest for: $tag"
              docker buildx imagetools create \
                --tag "$tag" \
                "${tag}-amd64" \
                "${tag}-arm64"
            fi
          done
 
      # Python Base Prod Manifest
      - name: Extract metadata for Python base prod manifest
        id: meta-python-base-prod
        uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}/python-base-prod
          tags: |
            type=sha,format=short
            type=ref,event=branch
            type=raw,value=latest,enable={{is_default_branch}}
 
      - name: Create and push Python base prod multi-arch manifest
        run: |
          echo '${{ steps.meta-python-base-prod.outputs.tags }}' | while IFS= read -r tag; do
            if [ -n "$tag" ]; then
              echo "Creating manifest for: $tag"
              docker buildx imagetools create \
                --tag "$tag" \
                "${tag}-amd64" \
                "${tag}-arm64"
            fi
          done

What changed

Run on Latchkey managed runners with one line (runs-on), which apply the fixes below automatically and self-heal transient failures. This example uses latchkey-small; pick the runner size that fits the job.
Cancel superseded runs when a branch or PR gets a newer push.
Add a job timeout so a hung step cannot burn hours of runner time.

What Latchkey heals here

This workflow has steps that commonly fail on transient issues (network, registries, flaky browsers). On Latchkey managed runners they are detected, retried, and self-healed instead of failing your build:

Container pulls and builds

This workflow runs 2 jobs per trigger. On Latchkey the same minutes cost up to 58% less than GitHub-hosted, with zero queue time.

Actions used in this workflow

actions/checkout docker/setup-buildx-action docker/login-action docker/metadata-action docker/build-push-action