How to Smoke-Test Distributed Training in GitHub Actions

Launch a two-process run with torchrun on tiny data so the process group init and a single DDP step are exercised without a full cluster.

Use torchrun --nproc-per-node to launch multiple processes on one runner and run one training step. This exercises process-group setup and gradient sync without needing multiple nodes.

Steps

• Wrap the model in DDP and read the local rank from the environment.
• Launch with torchrun --nproc-per-node.
• Run one step and assert it completes on all ranks.

Workflow

jobs:
  ddp-smoke:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .
      - name: Two-process DDP step
        run: torchrun --standalone --nproc-per-node 2 train_ddp.py --max-steps 1

Gotchas

• On a single GPU, use the gloo backend or share one device; NCCL wants one GPU per process.
• A hung process group usually means a rank crashed; set a step timeout so it does not run forever.

Related guides