What is a byte order mark?

A byte order mark is a short sequence of bytes at the very start of a text file whose only job is to describe how the rest of the file is encoded. It is not part of the text. You cannot see it in an editor, it is not counted in the character count, and it will not appear if you search for it.

It is usually called a BOM, and it is the reason for two apparently unrelated complaints: accented letters surviving a trip into Excel, and a shell script failing on its first line with an error that makes no sense.

The problem it was invented to solve

Text is stored as numbers, and numbers larger than 255 do not fit in one byte. UTF-16 stores each character as at least two bytes, which raises a question with no natural answer: which byte comes first?

Processors disagree. Some are big-endian and write the most significant byte first; others are little-endian and write it last. The character A, code point 65, is stored as 00 41 on one machine and 41 00 on the other. Hand a file written by one to the other, and every character comes out as a different, usually nonsensical, character.

The Unicode standard solved this with a small trick. Reserve one code point, U+FEFF, as a marker and put it at the start of the file. The code point that would result from reading those two bytes backwards, U+FFFE, is permanently defined as not a character. So a program reads the first two bytes:

• FE FF — U+FEFF, read correctly. The file is big-endian.
• FF FE — U+FFFE, which cannot exist. The bytes must be reversed. The file is little-endian.

One self-describing signal, no configuration, no guessing. That is a genuinely elegant piece of design, and it is where the name comes from: it marks the byte order.

Then UTF-8 arrived and the name stopped making sense

UTF-8 has no byte order to mark. It is defined as a sequence of individual bytes, in a fixed order, and the question of which end comes first never arises. A BOM in a UTF-8 file is answering a question nobody asked.

It got used anyway, because it turned out to be useful for something else: as a signature. A file starting with the UTF-8 encoding of U+FEFF is almost certainly UTF-8, because that byte sequence is vanishingly unlikely to occur by accident at the start of a file in any other encoding. Microsoft adopted it, and once Windows tools wrote it, everyone else had to be able to read it.

So the UTF-8 BOM is a byte order mark that marks no byte order. It says one thing: this file is UTF-8.

What the bytes actually are

Encoding	Bytes at the start of the file
UTF-8	EF BB BF
UTF-16 big-endian	FE FF
UTF-16 little-endian	FF FE
UTF-32 little-endian	FF FE 00 00

EF BB BF is the one you will meet, and it is the one worth memorizing. When someone says “the file has a BOM” without qualification, they mean those three bytes.

Why it solves problems

Without a BOM, a program opening a text file has to guess the encoding, and there is no reliable way to guess. The bytes are the same either way; only the interpretation differs.

Excel’s guess, on Windows, is the legacy code page rather than UTF-8. So a UTF-8 CSV file containing å — stored as the two bytes C3 A5 — gets read as Windows-1252, where those two bytes mean Ã and ¥. The name Håkan arrives as HÃ¥kan. Nothing is corrupted; the file is being read with the wrong assumption.

Three bytes at the front remove the guess. Excel sees EF BB BF, switches to UTF-8, and the name arrives intact. This is the entire difference between the two CSV options in Excel’s Save dialog, which is covered in CSV UTF-8 vs CSV. You can check whether a file has one, along with which delimiter it actually uses, by opening it in a CSV viewer that reports what it finds instead of assuming.

Why it causes problems

The BOM was designed for Windows and for Unicode-aware editors. Everything else on a computer predates it, and a great deal of software treats the first byte of a file as significant.

Shell scripts stop working. A script starts with #!/bin/sh on line one. With a BOM, line one starts with three invisible bytes, the kernel does not find the interpreter marker where it expects it, and you get an error naming a file that plainly exists.

PHP sends output too early. Any bytes outside the PHP tags are output. A BOM before the opening tag is three bytes of output, sent before any header can be set. The result is the classic “headers already sent” warning, pointing at line 1 of a file that looks empty there.

JSON parsers reject the file. The JSON specification defines a document as starting with a value. A BOM is not a value. Strict parsers fail on character one, and the error message is usually something unhelpful about an unexpected token.

The first column name is wrong. In a CSV whose header row is id,name,email, a program that reads the BOM as text sees the first field as the three BOM bytes followed by id, which is not equal to id. Every lookup by column name fails on the first column only, and the two strings look identical in every error message, which is a genuinely confusing way for an import to break.

Files compare as different when they are not. A BOM added by one editor and not another makes the first line of a diff show as changed with no visible difference. Whole afternoons have gone into that one.

Seeing it, including the  version

The BOM is invisible when read correctly. It becomes visible in exactly the situation it was meant to prevent: when the file is read with the wrong encoding.

Read EF BB BF as Windows-1252 and you get three real characters — ï, », ¿. So a page or a file starting with  has a UTF-8 BOM being read as a legacy encoding. It is the same failure mode as Ã¥ for å, applied to the marker rather than to the text, and it usually means a web page is not declaring its encoding or a template file was saved with a BOM by an editor that assumed one was wanted.

To check deliberately:

hexdump -C file.csv | head -1
file file.csv

hexdump shows the raw bytes; look for ef bb bf at offset zero. The file command reports “UTF-8 Unicode (with BOM) text” when one is present. In VS Code the status bar shows either “UTF-8” or “UTF-8 with BOM”, and clicking it lets you save the file the other way.

When you want one, and when you do not

The rule is short.

Add a BOM to CSV and plain text files that will be opened by Excel on Windows and contain any character outside plain English. This is the case the BOM genuinely solves, and there is no better option available inside Excel.

Leave it off everything else: source code, JSON, YAML, HTML, shell scripts, config files, anything served over HTTP, anything read by a Unix tool. The web has declared encoding in headers and meta tags for decades, and the Unicode standard itself does not recommend a BOM in UTF-8. Every problem in the previous section comes from a BOM in a file that should not have had one.

To remove one from a file you already have:

sed -i '1s/^\xEF\xBB\xBF//' file.txt

Or open it in a decent editor and save it as UTF-8 without BOM. If none of the encoding vocabulary here is familiar, what Unicode is covers code points and encodings from the start, and UTF-8 vs ASCII explains why UTF-8 ended up as the encoding everything defaults to.

More on this

• UTF-8 vs ASCIIASCII had 128 characters and no room for the rest of the world. UTF-8 won its replacement not by being cleverest, but by making every existing ASCII file valid.