What Is a Character Encoding? Explained

A character encoding is the scheme that maps characters to the bytes used to store and transmit text, and back again.

Computers store everything as bytes, but humans read characters. A character encoding is the agreement that connects the two: which bytes mean which letters. When the writer and reader of a file agree on the encoding, text round-trips perfectly; when they disagree, you get garbled output - one of the most common and confusing CI failures.

What a character encoding is

A character encoding is a mapping in both directions: from characters to bytes when writing, and from bytes back to characters when reading. To read a file correctly, a program must use the same encoding that was used to write it. Otherwise it interprets the bytes wrongly.

Character set versus encoding

A character set (like Unicode) is the inventory of characters and the number assigned to each. An encoding (like UTF-8 or UTF-16) is the concrete rule for turning those numbers into bytes. One character set can have several encodings, which is why both pieces of information matter.

Legacy encodings and mojibake

Before Unicode, many regional encodings existed - Latin-1, Windows-1252, Shift-JIS - each covering a limited set of characters. Reading a file written in one of these as if it were another produces "mojibake," the scrambled accented characters you sometimes see in output.

Encoding in CI

CI runs across machines that may have different default locales, so a job that works locally can fail in CI purely because the runner assumes a different encoding when reading a source file. Pinning the encoding in the environment removes this whole class of nondeterminism.

Pinning encoding in a job

# Make a Python build encoding-deterministic in CI
env:
  PYTHONUTF8: "1"
steps:
  - run: python generate_docs.py

Why it matters for reproducibility

A reproducible build should produce identical output everywhere. Encoding is a hidden input: the same source can yield different bytes if read under different encodings. Standardizing on UTF-8 everywhere - editors, files, and CI - eliminates this source of drift.

Key takeaways

A character encoding maps characters to bytes and back; reader and writer must agree on it.
A character set is the inventory of characters; an encoding is how those become bytes.
CI encoding bugs come from locale differences between machines - pin UTF-8 for deterministic builds.