Self-Healing CI: Recovering a Retracted GCS Resumable Upload
A GCS resumable upload that the server retracts or restarts mid-flight hit a transient blip, not a build error -- re-initiating and resuming the upload completes it.
The problem
An upload to Google Cloud Storage fails because a resumable session was retracted or had to restart -- a 503/5xx or a connection reset asked the client to re-initiate. The object data is fine; the transfer hit a transient backend or network problem. A human re-runs and the upload completes unchanged.
ServiceException: 503 Backend Error (resumable upload); upload must be restarted
ResumableUploadAbortException: Upload complete with ... but server reported a different stateWhy it happens
A resumable upload negotiates a session and streams data, and a backend 5xx or a dropped connection can force that session to be retracted and re-initiated, so a single transfer can fail mid-flight even though the data and destination are valid.
It is a transient backend/transport blip, not a build problem: re-initiating or resuming the session uploads the same object successfully.
The manual fix
Manual mitigations for a GCS upload retraction:
- Re-run the job to re-initiate the upload.
- Rely on the client library’s resumable-upload retry/backoff for large objects.
- Reduce concurrency and add jitter when uploading many objects at once.
How this gets automated
A retracted resumable upload has a recognizable transient signature -- a 5xx or session restart rather than a permissions or data error -- and the safe response is to re-initiate or resume and retry. A self-healing CI pipeline detects the upload failure, retries the transfer with backoff, and only escalates if the upload keeps failing, distinguishing a backend blip from a genuine storage problem.