How to Keep Full Training Out of CI to Control Cost in GitHub Actions

Run only fast smoke and evaluation steps in CI and dispatch long training to an external GPU platform so a workflow never holds an expensive GPU for hours.

CI is billed per minute, so a multi-hour training run on a GPU runner is costly. Keep CI to smoke tests and short evals, and trigger real training on a dedicated trainer (a batch job, cluster, or managed service) from the workflow.

Steps

• Run smoke tests and short evals inline in CI.
• Submit long training to an external trainer via API or CLI.
• Poll or webhook for results rather than blocking the runner.

Workflow

jobs:
  dispatch-training:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Submit training job
        run: |
          python submit_training.py --commit "${GITHUB_SHA}" --wait false
        env:
          TRAINER_TOKEN: ${{ secrets.TRAINER_TOKEN }}

Gotchas

• Blocking a GPU runner while training runs for hours is the most common source of runaway CI cost.
• Record the external job id in the run log so results can be traced back to the commit.

Related guides