Don't Panic — Text Encoding and Unicode

Character encoding is the agreement that turns text into bytes and back again. Computers are devoted to bytes, which are excellent at being bytes and notably poor at explaining which alphabet they were meant to become. Before Unicode, different systems gave the same byte patterns different jobs. The result was mojibake: a document that had taken a brief, unwanted vacation through the wrong decoder.

Unicode supplies the shared map. It gives each character, an abstract unit of written language, a number called a code point. The code point is not the picture on screen; that picture is a glyph chosen by a font. Nor is it necessarily a byte. An encoding form such as UTF-8 turns the code point into bytes, and UTF-8 is now the usual choice because ASCII remains itself and other characters fit into one to four bytes.

Keep the layers separate: character, code point, code units, bytes. This prevents the most common category error in text work, namely declaring that the fifth byte is the fifth character. It may be sitting inside a UTF-8 sequence, looking innocent in hexadecimal and becoming much less innocent when cut in two. Decode before you index text, and cut byte streams only at character boundaries.

The second surprise is that one visible symbol can be several code points. An é can arrive as one precomposed code point or as an e with a combining acute mark. Both can look the same while their bytes disagree. Normalization chooses a standard representation for a use case; NFC handles canonical equivalents, while NFKC also folds compatibility forms and therefore loses distinctions. A byte comparison, a text equality comparison, and locale-aware sorting are three different questions wearing similar hats.

There are more edges because text is sociable. UTF-16 uses surrogate pairs for supplementary characters. A grapheme cluster is the user-perceived editing unit, not always one code point. A malformed byte sequence needs strict handling. Look-alike characters can spoof an identifier. None of this makes Unicode unmanageable; it means text needs a declared boundary and an explicit policy instead of a hopeful shrug.

Read the intro when the layer model or UTF forms need a full explanation. Use the slides for the relationships at a glance, then keep the cheatsheet nearby for byte patterns, normalization, and diagnosis. The practice reference and exercise turn the theory into byte output and a strict-decoding check. Field Notes covers the operational traps that appear after the strings leave the whiteboard and enter real systems.

Don't Panic — Text Encoding and Unicode

Relevant careers