What is Unicode?

Unicode is a catalog. It assigns a unique number — a code point — to every character in every writing system humans use, plus a great many symbols that are not writing at all. The letter A is code point 65. The letter å is 229. The waving hand emoji is 128,075.

That is all Unicode does. It says which number means which character. It says nothing about how those numbers are stored in a file, and that second question is where nearly every text bug lives.

Some Unicode characters resemble styled letters while remaining different code points. A fancy text generator makes that distinction inspectable: it maps supported characters locally, preserves the rest, and shows the exact output code points without claiming to create a font file.

Unicode is not UTF-8

This distinction is the one worth internalizing.

Unicode is the mapping from characters to numbers. UTF-8 is one way of turning those numbers into bytes. UTF-16 is another. They describe the same characters and produce completely different files.

UTF-8 is variable-width: plain English letters take one byte each, most European accented letters take two, most Asian characters take three, and emoji take four. That design is why UTF-8 won — an English text file in UTF-8 is byte-identical to the same file in ASCII, so thirty years of existing files kept working.

It is also why å becomes Ã¥ when a file is read with the wrong assumption. Those are its two bytes, interpreted one at a time.

Keeping the two layers apart explains a whole family of bugs that look unrelated. Base64, for instance, is a third layer again: it turns bytes into printable characters so they survive an email or a URL, and it never learns what those bytes meant. Decode a base64 string and you are back to bytes, still needing to be told which encoding wrote them — which is why a decoder that shows the encoding it used can hand back Västerås where one that always assumes UTF-8 hands back nonsense with a confident face.

Code points are not characters, either

Here the catalog gets subtle in ways that break code.

A family emoji looks like one character. It is four people plus three invisible joining characters — seven code points that a font renders as a single picture. A flag is two letters from an alphabet that exists only to make flags. An é might be one code point, or it might be e followed by a combining accent, and both look identical on screen while comparing as different strings.

What a reader perceives as one character is called a grapheme cluster. Counting those correctly is why a properly built character counter reports a family emoji as one character while a naive one reports five — and why removing emoji has to match whole sequences rather than individual code points. When the goal is to find and copy one of those canonical sequences instead, emoji search keeps the complete Unicode form together rather than copying whichever individual code point happens to look familiar.

The invisible characters

Unicode includes characters that render as nothing at all: zero-width spaces, non-breaking spaces, soft hyphens, byte order marks, and directional marks. Every one of them has a legitimate use, and every one of them causes confusion when it appears where it was not expected.

A zero-width space pasted into a CSV file breaks the column silently. A non-breaking space in code looks exactly like a normal space and fails to compile. These are not exotic edge cases — they arrive routinely from word processors and web pages.

Homoglyphs

Different code points can look identical. Latin а and Cyrillic а are different characters that most fonts render the same way. This is the mechanism behind homoglyph phishing: a domain name that reads as apple.com but is not.

It is also a routine nuisance in ordinary data, where a name copied from one system will not match the same name in another because one of them contains a Cyrillic letter nobody typed deliberately.