Rollbacks and Automated Recovery

The fastest way to fix a bad deploy is usually to undo it, not to debug it live.

Every deployment strategy needs an answer to "what if this release is bad?" This lesson covers making deploys reversible, detecting failure automatically with health checks and metrics, and wiring up automated rollback so recovery is fast and predictable.

Roll back, then investigate

When a release degrades production, the priority is restoring service, not root-causing live. If your previous artifact is still available, redeploying it is the quickest recovery. Investigate the failure afterward from logs and traces, with users already back on a healthy version.

Make rollback cheap

Keep immutable, versioned artifacts so any prior version can be redeployed.
Prefer additive, backward-compatible migrations so the schema does not block rollback.
Tag releases so "deploy the previous version" is a single command.
Practice rollback regularly so it is not first attempted during an incident.

Detect bad releases automatically

Tie rollback to signals: post-deploy smoke tests, health endpoints, and error-rate or latency thresholds. Progressive delivery controllers can watch these during the ramp and abort automatically. A manual rollback path should always exist as a fallback.

A scripted rollback step

Expose rollback as a workflow you can trigger on demand to redeploy a known-good tag:

.github/workflows/rollback.yml

on:
  workflow_dispatch:
    inputs:
      version: { description: "Tag to roll back to", required: true }
jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: ./deploy.sh ${{ inputs.version }}

Key takeaways

Restore service by rolling back first; debug the failure afterward.
Immutable artifacts and backward-compatible migrations keep rollback cheap.
Wire health checks and metric thresholds to trigger automatic rollback.