iconv: Convert Text Encodings in CI
iconv -f <from> -t <to> reads text in one character encoding and writes it out in another.
When a build tool chokes on non-UTF-8 source, iconv re-encodes the file. The trap is deciding what to do with bytes that have no target representation.
What it does
iconv reads from stdin or a file, decodes it using the -f (from) encoding, and re-encodes it with -t (to), writing to stdout. Common in CI to normalize legacy ISO-8859-1 (latin1) or WINDOWS-1252 files to UTF-8 before a linter or compiler reads them.
Common usage
iconv -f latin1 -t utf-8 input.txt -o output.txt
# transliterate characters that have no exact target
iconv -f utf-8 -t ascii//TRANSLIT notes.txt
# drop untranslatable bytes instead of erroring
iconv -f utf-8 -t ascii//IGNORE notes.txt
iconv --list # show every supported encoding nameOptions
| Flag | What it does |
|---|---|
| -f <enc> | Encoding of the input (from) |
| -t <enc> | Encoding of the output (to) |
| -o <file> | Write to a file instead of stdout |
| //TRANSLIT | Suffix on -t: approximate untranslatable chars |
| //IGNORE | Suffix on -t: silently drop untranslatable bytes |
| -c | Omit invalid characters from the output |
| -l, --list | List all known encoding names |
In CI
To normalize a checked-out file in place, write to a temp path then move it back: iconv -f latin1 -t utf-8 f.txt -o f.tmp && mv f.tmp f.txt. Piping iconv ... > f.txt truncates the file before iconv reads it and loses the content.
Common errors in CI
"iconv: illegal input sequence at position N" means a byte is not valid in the -f encoding (often you guessed latin1 but it was UTF-8, or vice versa). Add //IGNORE to -t or -c to skip bad bytes, or fix the -f guess. "iconv: conversion from X unsupported" means the encoding name is wrong; check iconv --list (names are case-insensitive but exact, e.g. UTF-8, ISO-8859-1).