What Is UTF-8? Unicode Text Encoding Explained

UTF-8 is the dominant text encoding that represents every Unicode character using one to four bytes, staying compatible with plain ASCII.

UTF-8 is how nearly all modern text is stored and transmitted. It can represent every character in every language plus emoji, while encoding plain English the same way ASCII does. Getting encoding right matters in CI because a file read with the wrong encoding produces garbled characters or outright crashes a build step.

What UTF-8 is

UTF-8 is a variable-length encoding of Unicode. Common ASCII characters use a single byte, while other characters use two, three, or four bytes. This means any valid ASCII file is already valid UTF-8, which made adoption painless and is a big reason UTF-8 won.

Unicode versus encoding

Unicode is the catalog that assigns every character a number (a code point). An encoding like UTF-8 is the scheme for turning those numbers into actual bytes. The same character can be encoded different ways; UTF-8 is just the most widely used scheme for doing it.

Why UTF-8 became the default

UTF-8 is backward-compatible with ASCII, compact for Western text, and capable of representing everything. The web standardized on it, and most languages and tools now default to it. When two systems disagree on encoding, the bug usually disappears once both use UTF-8.

Encoding issues in CI

Build failures from encoding usually trace to a locale mismatch: a runner with a non-UTF-8 locale mis-reads a source file with accented characters or emoji, producing mojibake or a decode error. Forcing a UTF-8 locale in the job environment makes builds deterministic across machines.

Setting a UTF-8 locale in CI

# Force a UTF-8 locale in a CI job
env:
  LANG: C.UTF-8
  LC_ALL: C.UTF-8
steps:
  - run: python build.py

Latchkey note

Latchkey runners ship with a UTF-8 locale by default, matching GitHub-hosted runners, so source files with non-ASCII characters build the same way without extra locale wrangling.

Key takeaways

UTF-8 encodes every Unicode character in one to four bytes and is fully ASCII-compatible.
Unicode assigns characters numbers; UTF-8 is the scheme that turns those numbers into bytes.
Most CI encoding bugs are locale mismatches - forcing a UTF-8 locale makes builds deterministic.