bundler could not fetch specs rubygems ci: which fetcher failed

By Daniel Zoghalchali·Latchkey

A bundler could not fetch specs rubygems ci failure is raised by Bundler's oldest and slowest index fetcher, which is the third one it tries, so the message proves that two faster paths already failed and were dropped without a word. Reading it as a plain outage report is how teams end up retrying a source that is answering everything except the request Bundler actually made.

What this error means

The job stops inside bundle install or bundle lock, usually after a row of dots and one or more Retrying fetcher due to error (n/4) lines. Two different sentences end the run and they come from different places. Could not fetch specs from <uri> due to underlying error <...> is raised in Bundler::Fetcher::Index, the legacy full-index fetcher, which sits last in Bundler's list. Could not reach host <host>. Check your network connection and try again. is raised much earlier, in the downloader itself, and it is the only one of the two that means no TCP conversation happened at all. A third shape, Network error while fetching <uri> (<exception>), comes from the same downloader for the errors it considers worth retrying. Which sentence you have decides whether the thing to fix is the source, the path to the source, or the fact that your source does not speak the modern API.

Fetching gem metadata from https://rubygems.org/.......
Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://rubygems.org/ due to underlying error <bad response Service Unavailable 503 (https://rubygems.org/specs.4.8.gz)>
Bundler::HTTPError: Could not fetch specs from https://rubygems.org/ due to underlying error <bad response Service Unavailable 503 (https://rubygems.org/specs.4.8.gz)>

Three fetchers, and only one of them says this

Bundler::Fetcher#available_fetchers returns [CompactIndex, Dependency, Index] for any source that is not a file: URL and has not had the endpoint disabled. fetch_specs walks that list and drops a fetcher as soon as it returns nothing usable. So the compact index API is tried first, the dependency API second, and the full specs.4.8.gz index last.

What makes the ordering invisible is the wrapper on the first one. CompactIndex.compact_index_request catches HTTPError and returns nothing, with only a trace-level note, so an ordinary 500 or 503 on the compact index does not end the run and does not print. It has two exceptions: a NetworkDownError or a checksum mismatch is re-raised as an HTTPError, and a 401 is re-raised as is. Everything else falls through quietly to the next fetcher.

That is why Could not fetch specs from deserves a different reading than the one it usually gets. By the time it appears, Bundler has already given up on two endpoints. On rubygems.org during a real incident that is simply an outage with extra steps. Against a private mirror it usually means the mirror does not implement the compact index API at all, and every one of your builds has been quietly falling back to the slowest possible path and only fails when that path also hiccups.

Could not reach host rubygems.org. Check your network connection and try again.

Common causes

The source is genuinely unwell, and all three fetchers hit it

A rubygems.org incident or a CDN problem in front of it fails the compact index, the dependency API and the full index in turn, and the last one is the one that speaks. The tell is that the angle brackets carry a 5xx and that the whole ladder ran in a few seconds. This is the version a retry actually fixes, and it is the minority of the reports we see.

Your mirror does not implement the compact index API

A source pointed at an internal proxy, an artifact server or an old Gemstash can answer the full index and nothing else. Bundler falls back silently, so nothing in a green build tells you this is happening, and the first symptom is a failure on the slowest path under load. In our experience this is the most common cause of the message inside a company network, and it is also the one that gets misfiled as a rubygems.org outage.

The runner cannot resolve or reach the host at all

A DNS failure or a blocked egress raises SocketError or Errno::EHOSTUNREACH, which are in HTTP_NON_RETRYABLE_ERRORS, so the downloader raises NetworkDownError with the Could not reach host wording instead. That error is one of the two the compact index wrapper re-raises rather than swallows, so it ends the run on the first fetcher and you never see the specs message.

A proxy in the path is closing connections mid-index

The full index is a single large gzipped file, so it is the request most likely to be cut short by a middlebox with an idle or size limit. That surfaces as Gem::Timeout::Error, EOFError or Errno::ECONNRESET in the angle brackets, all of which are in HTTP_RETRYABLE_ERRORS, which is why you see the retry lines before the failure rather than an immediate stop.

How to fix it

Find out which fetcher is actually serving you

1. Run the install once with --verbose in a scratch job. The debug output logs each HTTP GET, so a source that is serving specs.4.8.gz rather than versions and info/<gem> is answering from the full index.
2. If it is the full index, fix the source rather than the timeout. Every build on that path is downloading the entire index instead of the few gems that changed.
3. Keep the finding in the repository. A comment beside the source line saying which API the mirror supports is worth more than the next three incident write-ups.

bundle install --verbose 2>&1 | grep -E "HTTP GET|Retrying"

Cache the bundle so most jobs never ask the index anything

The cheapest index request is the one that does not happen. ruby/setup-ruby installs gems and caches them when bundler-cache is on, keyed on the lockfile and the Ruby version, so a job with an unchanged Gemfile.lock does no index work at all and a flaky source stops being able to fail your build.

- uses: ruby/setup-ruby@v1
  with:
    ruby-version: .ruby-version
    bundler-cache: true

Raise the retry count and the timeout, and know what each buys

The retry count covers a blip and the timeout covers a slow transfer, and they fail differently. Retries do nothing for a credentials error because those are in FAIL_ERRORS. The timeout is applied as both the open and the read timeout, so raising it also makes an unreachable host take longer to fail, which is the cost you are paying for the benefit.

- run: |
    bundle config set --global retry 5
    bundle config set --global timeout 30
    bundle install --jobs 4

Stop the message being ambiguous in your own logs

Since three different sentences mean three different things, match on them separately wherever you alert. A rule that fires on the word fetch alone will page somebody about a certificate problem and a rate limit with the same text, and the on-call person will start by checking a status page that is green.

grep -E "Could not reach host|Could not fetch specs from|Could not verify the SSL certificate" build.log

What the retry counter is counting

The Retrying fetcher due to error (2/4) line is written by Bundler::Retry#fail_attempt, and the word before due to is the name the retry was created with. Bundler::Fetcher#specs_with_retry creates it as "fetcher", the dependency API path creates one called "dependency api", and gem downloads create one named after the redacted URL, so the noun tells you which layer is retrying.

The denominator is retries + 1, and retries is Bundler.settings[:retry], whose default in DEFAULT_CONFIG is 3. That is where the familiar (2/4) comes from: four runs in total, three of them retries. The backoff is exponential from a base of one second, doubling, capped at sixty, with up to half a second of jitter. Two other defaults from the same table matter here: BUNDLE_TIMEOUT is 10 seconds and is applied as both the open and the read timeout on the connection, and BUNDLE_REDIRECT is 5.

A set of exceptions skips the retry ladder entirely. FAIL_ERRORS holds the three authentication errors, the fallback error used for a 404, SecurityError, and a list of Gem::Net::HTTP* response classes including forbidden, unauthorized and not found. The reasoning in the source is blunt and correct: if your password did not work the first time, it will not work the third. So a credentials problem fails fast and an outage fails slowly, and the elapsed time of the failing step is itself a signal.

bundle config set --global retry 5
bundle config set --global timeout 30
bundle install --jobs 4

Why this page carries no recorded run

To produce this message honestly you need a source that answers the compact index request with something Bundler will swallow, then fails the full index request. That is a two-behaviour server, not an outage, and standing one up on a runner would prove that a server we wrote returns what we told it to. The branch that picks each sentence is the thing worth knowing, and it is readable in Bundler at a named version.

So the blocks here are assembled from fetcher/index.rb, fetcher/downloader.rb and retry.rb at Bundler 4.0.21 and labelled as reconstructions. The text inside the angle brackets in the first block is whatever the underlying error object stringified to, which varies by Ruby and by source, so treat the shape as the quotable part and your own bracket contents as the detail. This page makes no claim that Latchkey repairs the failure: content/heal-evidence.mjs has no record for this slug, and without one the claim is not available to make.

How to prevent it

• Turn on bundler caching in setup-ruby so the index is consulted only when the lockfile moves.
• Confirm once, in writing, which index API every source in your Gemfile actually serves.
• Set retry and timeout globally in the job rather than on one install command, so every bundler call in the job agrees.
• Alert on the three sentences separately. Only one of them is worth waking somebody for.

Frequently asked questions

Related guides

References

• rubygems/rubygems: Fetcher::Index#specs, where the specs message is raised
• rubygems/rubygems: Fetcher::Downloader and its retryable and non-retryable error lists
• rubygems/rubygems: Bundler::Retry, the counter and the backoff
• ruby/setup-ruby: bundler-cache and what it keys on
• Bundler documentation
• GitHub Actions documentation