Python "UnicodeDecodeError" in CI - Fix Locale & Encoding

Python read a file or stream as ASCII (or the wrong codec) because the runner has no UTF-8 locale set. The same code works locally where the locale is UTF-8.

What this error means

Reading a file with non-ASCII characters, or printing Unicode, crashes with a UnicodeDecodeError or UnicodeEncodeError only in CI. Locally it passes because your shell locale is UTF-8.

Python traceback

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 14:
ordinal not in range(128)

Common causes

No UTF-8 locale on the runner

Minimal images set LANG=C/POSIX, so Python’s default text encoding becomes ASCII. Any non-ASCII byte then fails to decode.

Opening files without an explicit encoding

Pre-3.15 open() uses the locale’s preferred encoding. On a non-UTF-8 runner that is not UTF-8, so reads of UTF-8 files break.

How to fix it

Force UTF-8 mode

Python’s UTF-8 mode makes the interpreter use UTF-8 regardless of locale.

Terminal

export PYTHONUTF8=1
# or per command
python -X utf8 app.py

Set a UTF-8 locale

Terminal

# Debian/Ubuntu
apt-get install -y locales
locale-gen en_US.UTF-8
export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

Specify encoding when opening files

Always pass encoding="utf-8" so the result does not depend on the runner locale.

Python

with open("data.txt", encoding="utf-8") as f:
    text = f.read()

How to prevent it

Set PYTHONUTF8=1 (or a UTF-8 LANG) in CI.
Always pass encoding="utf-8" to open().
Use a runner image that ships a UTF-8 locale.