Binary Data and Encoding
Binary data and encoding covers how computers represent, store, and transmit information below the text layer: number systems, byte ordering, character encodings, serialization formats, and the conversions needed when moving data between systems that make different assumptions about representation.
itComputer fundamentals | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Binary Data and Encoding: Don't Panic
Binary data is what remains when computers stop pretending that a value arrives with a label attached. A byte is a pattern of eight bits. It can be the number sixty-five, the letter A in UTF-8, a color component, or one small piece of an instruction. The byte has not become confused. It has merely declined to explain itself.
The missing explanation is the format. It supplies the type, width, signedness, byte order, encoding, framing, and validation rules. This is why a value like 00 10 can become sixteen, four thousand ninety-six, or two separate bytes depending on the agreement in force. The reliable move is wonderfully unglamorous: find the specification before assigning meaning.
Hexadecimal is the pocket notebook for this work. One hex digit represents four bits, and two represent one byte, so byte boundaries remain visible while you inspect a file or packet. It does not decode anything by itself. It does make it harder for a byte to disguise itself as a friendly-looking decimal number.
Numbers have their own bureaucratic requirements. An integer needs a width and signedness. A multi-byte value needs byte order. Floating point adds the rather important detail that many decimal fractions land near, rather than on, a binary value. None of this is corruption. It is the cost of fitting a wide world of values into finite bits.
Text supplies the most convincing impersonation. A Unicode code point is a numeric position for a character, while UTF-8 maps Unicode scalar values into one to four bytes. A displayed symbol can involve several code points, so byte offsets, code-point counts, and visible character counts are different measurements. The computer is not being difficult. It is being literal.
Then there is Base64, which turns arbitrary bytes into a restricted text alphabet for a transport channel. It is encoding, not encryption. Decoding it produces bytes, not necessarily text, truth, or a small brass plaque saying what happens next.
Start with the Cheatsheet when a value looks wrong: it has the interpretation checklist and fast diagnosis table. The Slides connect the choices into one map. The practice session turns those rules into a repeatable inspection loop, and the exercise makes a small framed record fail in a useful way. The Reference tab is for the specifications, because bytes only become friendly after the contract is written down.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://csrc.nist.gov/glossary/term/byte
Supports
- A byte as a sequence of eight bits
- A byte as one of 256 integer values from zero through 255
- https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-2/
Supports
- Separation of characters, code points, encoding forms, and encoding schemes
- UTF-8, UTF-16, and UTF-32 encoding forms
- UTF-8 code units as bytes and lack of an endian-order issue within those units
- Byte serialization and byte-order requirements for wider code units
- User-perceived characters potentially containing multiple code points
- https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/
Supports
- Formal definitions of code point, code unit, Unicode scalar value, and encoding scheme
- Well-formed and ill-formed code-unit sequences
- Canonical equivalence and normalization behavior
- Quiz answer about distinct byte sequences and equivalent text
- https://www.rfc-editor.org/info/rfc3629/
Supports
- UTF-8 as a one-to-four-octet representation of Unicode scalar values
- Preservation of ASCII values as single octets
- Valid UTF-8 byte-sequence syntax
- Security consequences of accepting invalid sequences
- Quiz answers about UTF-8 structure and validation
- https://www.rfc-editor.org/info/rfc4648/
Supports
- Base16, Base32, standard Base64, and URL-safe Base64 alphabets
- Four, five, and six input bits represented per output symbol
- Base64 grouping of three input octets into four output characters
- Padding, canonical encoding, non-alphabet handling, and covert-channel concerns
- Published Base16, Base32, and Base64 test vectors
- Quiz answers about hexadecimal, Base64 purpose, and strict decoding
- https://www.rfc-editor.org/rfc/rfc3550.html#section-4
Supports
- Network byte order as most significant byte first
- Big-endian interpretation of multi-byte integer fields
- A primary protocol example of explicit byte-order and alignment rules
- Quiz answer defining big-endian order
- https://www.rfc-editor.org/rfc/rfc2910.html
Supports
- Signed integer fields encoded with two's-complement binary
- Explicit integer widths of one, two, or four octets
- Big-endian order for those multi-octet integers
- Quiz answer about width and signedness
- https://standards.ieee.org/ieee/754/6210/
Supports
- Binary and decimal floating-point interchange and arithmetic formats
- Arithmetic operations, conversions, exceptions, infinities, and nonnumber values
- Finite floating-point formats and representability constraints
- Quiz answers about rounding and selecting an exact scaled representation
- https://www.rfc-editor.org/rfc/rfc20
Supports
- 1968 ASCII standardization as a common coded character set
- https://www.unicode.org/history/
Supports
- Unicode project origins in 1987 and incorporation in 1991
- https://www.rfc-editor.org/rfc/rfc3629
Supports
- UTF-8 origin in 1992 and later standardization
- https://www.rfc-editor.org/rfc/rfc2045
Supports
- 1996 MIME transfer encoding milestone
- https://www.rfc-editor.org/rfc/rfc4648
Supports
- 2006 standardization of Base-N encodings
- https://protobuf.dev/overview/
Supports
- Schema-directed binary serialization and the associated schema contract
- https://avro.apache.org/docs/
Supports
- Schema availability for binary files and RPC
- https://github.com/msgpack/msgpack/blob/master/spec.md
Supports
- MessagePack type system and byte-array formats
- https://flatbuffers.dev/
Supports
- Schema-driven binary buffers designed for direct access
- https://capnproto.org/
Supports
- Cap'n Proto structural validation and binary message representation
