Hugging Face "Can't load tokenizer for <repo>" in CI

AutoTokenizer.from_pretrained could not assemble the tokenizer for the repo. Either the tokenizer files are absent at that revision, the private repo needs a token, or a backend like sentencepiece is not installed on the runner.

What this error means

The load fails with "OSError: Can't load tokenizer for '<repo>'. If you were trying to load it from ... make sure ... a directory containing all relevant files for a tokenizer."

transformers

OSError: Can't load tokenizer for 'org/model'. If you were trying to load it from
'https://huggingface.co/models', make sure you don't have a local directory with the same name.
Otherwise, make sure 'org/model' is the correct path to a directory containing all relevant files
for a tokenizer.

Common causes

Missing tokenizer files or missing backend

The repo lacks tokenizer.json/vocab files at that revision, or a needed backend (sentencepiece, tokenizers) is not installed.

A private repo without a token

An unauthenticated request cannot fetch the tokenizer files of a private model.

How to fix it

Install the tokenizer backends and set a token

Install the backends the tokenizer needs, and provide HF_TOKEN for private repos.

Terminal

pip install "transformers[sentencepiece]" tokenizers
# and in the workflow: env: HF_TOKEN: ${{ secrets.HF_TOKEN }}

Point at a directory with the tokenizer files

Ensure the local path or revision actually contains the tokenizer artifacts, not only weights.

python

from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("org/model", revision="main")

How to prevent it

Install sentencepiece/tokenizers extras when models need them.
Set HF_TOKEN for private tokenizers.
Verify tokenizer files exist at the loaded revision.