Text Encoding and Unicode

Text looks natural on a screen, but a computer stores only bytes. A character encoding is the mapping that turns characters into bytes and bytes back into characters. Without an agreed encoding, the same bytes render as different text on different systems.

The problem is older than the web. In the 1980s, a byte held 256 patterns and each vendor or language assigned those patterns to a different set of characters. A document written on one machine became mojibake on another. Mixing languages in one document was hard because each encoding covered one script family. Software shipped with a patchwork of code pages and guessed which one applied.

Unicode solved that. It is a single coded character set that assigns a unique number, called a code point, to every character it covers. The repertoire covers modern and historical scripts, mathematical symbols, punctuation, and emoji. Unicode 17.0 assigns 159,801 characters across 172 scripts. Text encoded as Unicode can travel between systems without losing characters, and a string has one meaning everywhere.

Unicode separates three ideas that older encodings collapsed: the character repertoire, the code points that number it, and the encoding forms that serialize code points to code units and bytes. Keeping those layers distinct is the key to working with text correctly.

Characters, code points, and bytes

Three terms anchor everything else.

A character is the smallest unit of written language with semantic value. A letter, a digit, a punctuation mark, and a symbol are characters. Unicode draws a strict line between a character and the glyph that renders it. The letter A is one character; many typefaces draw it with different shapes. Those shapes are glyphs, and Unicode does not encode them.

A code point is the number Unicode assigns to a character. Code points range from U+0000 to U+10FFFF, a space of 1,114,112 values. The notation U+0041 means the code point at hexadecimal 41, which is the character LATIN CAPITAL LETTER A. Code points group into 17 planes of 65,536 values each. Plane 0 is the Basic Multilingual Plane and holds most modern scripts. The other planes hold supplementary characters such as historic scripts, musical symbols, and supplementary CJK ideographs.

A code unit is the minimal piece an encoding form uses. UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units, and UTF-32 uses 32-bit code units. The encoding form maps each code point to a sequence of code units; the encoding scheme then serializes those units as bytes.

character  ->  code point  ->  code units  ->  bytes
   A            U+0041          41              41
   ₣            U+20A3          E2 82 A3        E2 82 A3
   ❤            U+2764          E2 9D A4        E2 9D A4
   𠮷            U+20BB7         D842 DFB7      F0 A0 AE B7