Binary Data and Encoding

Computers store and move bits. A bit has one of two values: zero or one. Eight bits form a byte, which can represent 256 distinct bit patterns. Those patterns have no built-in meaning.

Consider the byte 01000001. You can interpret it as the unsigned integer 65. In UTF-8, it encodes the letter A. In an image, the same pattern might be one color channel. The bits do not tell you which interpretation is correct. A data format does.

This is the central idea of binary data: representation and meaning are separate. To read bytes correctly, you need metadata or a shared agreement about type, width, byte order, and encoding.

Build values from bits

Binary is a positional numeral system with base two. Each position has a power-of-two weight.

bits:     1  0  1  1  0  1  1  0
weights:128 64 32 16  8  4  2  1
value:  128 + 32 + 16 + 4 + 2 = 182

An unsigned byte covers values from 0 through 255. More bytes provide more patterns. A 16-bit field has 65,536 patterns, while a 32-bit field has more than four billion.

Hexadecimal is a compact way to write bits. Each hexadecimal digit represents four bits, so two digits represent one byte. The byte above is B6 in hexadecimal. Hex dumps, protocol documents, file signatures, memory addresses, and debugging tools use hexadecimal because byte boundaries stay visible.

Interpretation needs a contract

A sequence such as 00 10 is not self-describing. As an unsigned 16-bit integer, it can mean 16 or 4096 depending on byte order. As two independent bytes, it means the pair 0 and 16. As text, it contains a null byte followed by a control character.

A useful binary contract specifies:

type — integer, floating-point value, text, image sample, instruction, or another defined value;
width — the number of bits or bytes assigned to the value;
signedness — whether an integer representation includes negative values;
byte order — the significance order of bytes in a multi-byte value;
encoding — the mapping between abstract values and bit or byte sequences;
framing — where a value starts and ends;
validation rules — which patterns are valid and how errors are handled.

Without the contract, parsing becomes guesswork. File formats and network protocols exist partly to make this contract explicit.

Integers and byte order

An unsigned integer uses every bit pattern for a nonnegative value. A signed integer assigns some patterns to negative values. Two's complement is a common signed representation. You still need the width: the byte FF is 255 when unsigned and minus one when interpreted as an 8-bit two's-complement integer.

Multi-byte values also need a byte order. Big-endian places the most significant byte first. Little-endian places the least significant byte first.

value: 0x1234
big-endian bytes:    12 34
little-endian bytes: 34 12

Byte order changes the storage sequence, not the mathematical value. A format chooses an order, and a parser follows that choice. Many Internet protocols use big-endian network byte order. Other formats deliberately use little-endian order.

Floating-point values are approximations

Floating-point formats represent a wide range of magnitudes with limited precision. IEEE 754 specifies binary and decimal floating-point formats, operations, conversions, exceptions, infinities, and values that are not numbers.

A binary floating-point format cannot represent every decimal fraction exactly. For example, a value written as decimal 0.1 is normally stored as a nearby binary value. Arithmetic can expose the difference through rounding.

This is not corrupted data. It is a property of finite representation. Use floating point when range and efficient approximate arithmetic matter. Use a decimal or scaled-integer representation when exact decimal units matter, and document the scale.

Text has several layers

Text is not “just bytes.” Keep these layers separate:

character -> code point -> encoding form -> bytes

A character is an abstract text element. Unicode assigns a code point to an encoded character, written in a form such as U+0041. An encoding form maps code points to code units. An encoding scheme serializes those units as bytes.

UTF-8 uses one to four bytes for a Unicode scalar value. The ASCII range uses the same single-byte values in UTF-8, so plain ASCII text is also valid UTF-8. UTF-8 must still be decoded and validated. An arbitrary byte sequence is not automatically text, and a byte offset is not the same thing as a character position.

Visible text adds another complication. A user-perceived symbol can contain more than one code point. Unicode can also provide canonically equivalent sequences. Byte equality, code-point equality, and user-visible equality can therefore answer different questions.

Base encodings are not encryption

Some channels are designed for text rather than arbitrary bytes. A base encoding maps binary data to a restricted character alphabet so it can pass through such a channel.

Base16 writes four bits per character. Base64 processes 24 input bits as four groups of six bits and maps them to printable characters. Padding can complete the final output group. Base64 expands the data; it does not hide it. Anyone who knows the alphabet can decode it.

Encoding changes representation. Compression reduces redundancy. Encryption uses a key to protect confidentiality. Hashing produces a fixed-size digest used for purposes such as integrity checks. These operations solve different problems, even when their outputs all look like opaque text.

Parse defensively

Binary parsers operate at a trust boundary. A length field can exceed the remaining input. Integer arithmetic can overflow. A UTF-8 sequence can be malformed. A Base64 string can contain characters outside its alphabet. A decoder that accepts noncanonical forms can let two distinct inputs represent the same output.

Treat every field as a claim to verify:

Check that enough bytes remain before reading.
Validate lengths before allocation or copying.
Apply the format's exact width, signedness, and byte order.
Reject malformed or forbidden encodings.
Preserve raw bytes when you need forensic evidence or lossless forwarding.
Convert at system boundaries, then use typed values internally.

Where this skill matters

You use binary representation when you inspect a file header, implement a protocol, read a database page, diagnose corrupted text, exchange numeric data, analyze packets, or design an application programming interface.

The goal is not to memorize every format. Learn to ask the right questions: What do these bytes represent? Which specification defines the mapping? Where are the boundaries? Which byte order applies? What inputs are invalid? What information is lost during conversion?

A practical learning path

Convert small unsigned values among binary, decimal, and hexadecimal.
Split multi-byte integers and reconstruct them in both byte orders.
Compare signed and unsigned interpretations at a fixed width.
Encode sample Unicode code points as UTF-8 and validate their byte sequences.
Work through the RFC 4648 Base16 and Base64 test vectors.
Inspect a documented file or network header with a hex viewer.
Study one serialization format and implement strict bounds checking.
Explore floating-point rounding and choose representations from domain requirements.