Self-Healing CI: Recovering an ECR Token Expiry Mid-Push
An ECR push that turns unauthorized partway through hit a short-lived token expiring mid-transfer, not bad credentials -- re-authenticating and retrying clears it.
The problem
A docker push to ECR fails with unauthorized partway through because the ECR auth token expired during a long push of large layers. The IAM credentials are valid; the short-lived token aged out mid-transfer. A human re-authenticates to ECR and re-runs the push and it completes unchanged.
denied: Your authorization token has expired. Reauthenticate and try again.
error parsing HTTP 401 response body (pushing layer sha256:...)Why it happens
ECR issues a short-lived authorization token from a login step, and a push of large layers can take long enough that the token expires before the transfer finishes, producing a 401 mid-push purely because of timing.
It is a token-lifetime issue, not a permissions problem: the IAM credentials still grant access, and re-authenticating to get a fresh token lets the same push complete (resuming already-pushed layers).
The manual fix
The manual fix is to re-authenticate and retry:
- Re-run
aws ecr get-login-password | docker loginimmediately before (or again during) the push. - Move the ECR login as close as possible to the push step on long jobs.
- Re-run the push so a fresh token is issued -- already-pushed layers are reused.
aws ecr get-login-password --region "${REGION}" | docker login --username AWS --password-stdin "${ECR_HOST}"
docker push "${IMAGE}"How this gets automated
A token expiry mid-push has a recognizable signature -- a 401 on credentials that are valid -- and a well-defined remedy: refresh the token and retry. A self-healing CI pipeline detects the auth-expiry condition, re-authenticates, retries the push (resuming completed layers), and only escalates if the credentials are genuinely rejected, distinguishing an expired token from a real permissions problem.