Files and data
What is Character encoding?
Also called UTF-8 or text encoding.
Character encoding is the rule a text file uses to turn letters and symbols into bytes. UTF-8 is the most common encoding today.
How it works
Computers store text as numbers. An encoding is the table that says which number means which character. ASCII covered unaccented English letters; later encodings such as Windows-1252 and ISO-8859-1 added accented letters for Western languages; UTF-8 covers every script in Unicode.
A text file doesn’t always say which encoding it uses. The program opening it has to guess, and when it guesses wrong, accented letters, currency symbols and quotation marks turn into sequences such as é, ’ or a black diamond with a question mark. This is called mojibake.
Encoding vs file format
A CSV, JSON or plain-text file is still the same format whatever its encoding. The encoding only decides how the characters inside it are stored as bytes. That is why a CSV can open perfectly in one program and show garbled names in another: nothing is wrong with the table, only with how its text was read.
A practical example
The name José saved as UTF-8 and opened as Windows-1252 appears as José. The bytes are fine; the program simply read them with the wrong rule.
What to watch out for
Re-saving garbled text in a new encoding usually locks the damage in. Go back to the original file and open it with the correct encoding instead of fixing the visible characters by hand.
Tools you can use
- Clean CSV · Blank rows, duplicates and stray spaces out
- CSV to Excel · Keeps leading zeros and long IDs intact
- JSON Formatter · Validate, format and see exactly what is wrong
Go a little deeper
- Why does my CSV show strange characters?
- CSV vs Excel: what is the difference?
- How to fix invalid JSON
- How to remove duplicate rows from a CSV
Common questions
Why does my CSV show strange characters in Excel?
The file is probably UTF-8 without a byte order mark, and Excel guessed an older regional encoding. Importing the file and choosing UTF-8, or saving it as UTF-8 with a BOM, fixes it.
What is a BOM?
A byte order mark is three invisible bytes at the start of a UTF-8 file. It isn’t required, but some programs, including Excel, use it to recognize the file as UTF-8.
Which encoding should I use?
UTF-8. It can represent every language and emoji, it is the default on the web and in most modern software, and plain English text in UTF-8 is identical to ASCII.