What Is a Byte Order Mark (BOM)? Explained

A byte order mark (BOM) is a special sequence at the very start of a text file that signals its encoding and byte order.

The byte order mark is a tiny, invisible prefix some editors add to the front of a text file. It was meant to help readers detect the encoding, but in practice a stray BOM is a frequent cause of baffling CI failures - a script that "looks fine" but errors on its very first line. Knowing it exists saves hours of debugging.

What a BOM is

A byte order mark is a specific sequence of bytes at the start of a file that indicates its Unicode encoding and, for multi-byte encodings, the byte order. It is metadata, not content - it is invisible when the text is displayed, but the bytes are really there.

BOM in UTF-16 versus UTF-8

For UTF-16 the BOM is genuinely useful: it tells the reader whether the bytes are big-endian or little-endian. For UTF-8 there is no byte-order ambiguity, so a UTF-8 BOM serves no real purpose and is best omitted - yet some Windows editors add one anyway.

Why a UTF-8 BOM causes bugs

Many tools do not expect a BOM and treat those leading bytes as part of the content. A shell script with a BOM fails because the interpreter chokes on garbage before the shebang; a JSON or YAML parser may reject the file; a CSV header gains invisible characters. The error messages rarely mention the BOM.

BOMs in CI

A script that runs locally but fails on the runner with a cryptic "command not found" or syntax error on line 1 is often a BOM problem. A lint or pre-commit rule that strips BOMs, or a check in CI that rejects them, prevents the file from ever reaching a runner.

# Fail CI if any tracked file has a UTF-8 BOM
steps:
  - run: |
      if grep -rlP "^\xEF\xBB\xBF" --include="*.sh" .; then
        echo "BOM found in shell script"; exit 1
      fi

Practical advice

Configure your editor to save files as UTF-8 without a BOM, especially for shell scripts, JSON, YAML, and CSV. When a first-line error makes no sense, checking for a hidden BOM should be near the top of your list.

Key takeaways

• A BOM is invisible leading bytes that signal a files encoding and, for UTF-16, its byte order.
• A UTF-8 BOM is pointless and frequently breaks shell scripts, JSON, YAML, and CSV parsing.
• In CI, a baffling first-line error is often a stray BOM - save files as UTF-8 without a BOM.

Related guides