File Formats and Serialization

A file format is a set of rules for representing information as bytes. Serialization is the act of converting an in-memory value into such a representation. Deserialization reverses that conversion. The representation may live in a file, cross a network, enter a message queue, or remain in a database column.

These terms overlap, but they are not identical. A format describes the representation. A serializer and deserializer implement the conversion. A protocol adds rules about exchanging representations. A schema describes the expected structure and types. An application still supplies the meaning: a field named temperature needs an agreed unit even when its syntax and type are valid.

The representation pipeline

Structured data passes through several layers:

application values
      ↓ map to a data model
logical records, arrays, maps, scalars
      ↓ serialize with format rules
bytes or text
      ↓ store or transmit with metadata
file, message, database value, or HTTP representation
      ↓ identify, parse, and validate
reconstructed values
      ↓ apply application meaning
usable information

Every boundary can lose information. A language may distinguish integer sizes that the format does not. A map may allow key types that JSON objects cannot express. A timestamp may become a string without its time zone rule. A decimal may become a binary floating-point number. Round-trip testing checks whether encoding and decoding preserve the distinctions the application needs.

File Formats and Serialization

File Formats and Serialization

The representation pipeline

Syntax, structure, and meaning

Relevant careers