Hugging Face "Can't load tokenizer for X" in CI

AutoTokenizer.from_pretrained could not assemble a tokenizer: the model repo or local folder lacks the expected tokenizer files, or the tokenizers fast backend is not installed for a model that needs it.

What this error means

A load fails with "OSError: Can't load tokenizer for 'X'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name."

OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it
from 'https://huggingface.co/models', make sure you don't have a local directory with
the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a
directory containing all relevant files for a BertTokenizerFast tokenizer.

Common causes

Tokenizer files are missing from the source

A local folder or partial download lacks tokenizer.json/vocab.txt/tokenizer_config.json, so transformers cannot build the tokenizer.

A name collision or missing fast backend

A local directory shares the model name and shadows the Hub repo, or the tokenizers package is absent for a fast-only tokenizer.

How to fix it

Ensure the tokenizer files and backend exist

1. Confirm the source has the tokenizer files (or re-download the full repo).
2. Install the fast-tokenizer backend if the model requires it.
3. Rename any local directory that collides with the Hub repo name.

pip install "transformers[sentencepiece]" tokenizers

Pin a revision and cache it

Download a specific revision in a setup step so all tokenizer files are present and cached for the test job.

from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("bert-base-uncased", revision="main")

How to prevent it

• Cache the full model+tokenizer repo, not just weights.
• Avoid local directories that shadow Hub repo names.
• Install the tokenizer backend the model needs.

Related guides