Data Serialization

Expected level: Working

Serialization turns data held in memory (objects, lists, maps) into a sequence of bytes or text that can be saved or sent. Deserialization reads it back into memory.

Common formats

Format Kind Typical use
JSON Text Web APIs, configuration, data exchange
XML Text Older enterprise APIs, document formats
YAML Text Human-edited configuration
CSV Text Tables, spreadsheet import and export
Protocol Buffers, Avro, MessagePack Binary Compact and fast service-to-service messages

Text formats are easy to read and debug. Binary formats are smaller and faster but need tooling and, usually, a schema. See JSON Fundamentals for more on the most common one.

Round trip

# Python
import json
text = json.dumps({"id": 7, "name": "Ada", "tags": ["admin", "dev"]})   # object -> text
data = json.loads(text)                                                 # text -> object
// JavaScript
const text = JSON.stringify({ id: 7, name: "Ada", tags: ["admin", "dev"] });
const data = JSON.parse(text);

What does not survive

Formats support fewer types than programming languages do. Expect to handle:

  • Dates and times: JSON has no date type, so send an ISO 8601 string in UTC and parse it explicitly. See Dates, Times, and Time Zones.
  • Large and exact numbers: parsers may turn numbers into floating point, losing precision. Send ids and money as strings or integers of the smallest unit. See Number Representation.
  • Missing, null and empty mean different things. Agree on which you use.
  • Custom objects, functions and cyclic references need converting to plain data first.
  • Use UTF-8 for text unless told otherwise.

Handling data safely

  • Treat incoming data as untrusted. Validate its shape and values after parsing (Input Validation).
  • Never deserialize untrusted data with formats that can create arbitrary objects, such as Python’s pickle or native object serialization in several languages. This can lead to code execution.
  • Agree on a contract. Document field names and types, ideally with a schema (for example JSON Schema or OpenAPI).
  • Plan for change. Adding optional fields is usually safe; renaming or removing one breaks existing readers. Version your format when you must break it.
  • Expect unknown fields. Many JSON libraries ignore extra fields by default, while a strict mode can reject them. Readers should tolerate fields they do not know, so a newer writer does not break an older reader.
  • Keep secrets out of serialized logs and files.

Try it

Write a small program that builds a structure containing a string, a number, a list and a date. Serialize it to JSON, save it to a file, read it back and compare the result with the original. Which parts came back as a different type?

Common mistakes

  • Sending dates as local-time strings instead of ISO 8601 in UTC.
  • Sending large ids or money as floating-point numbers.
  • Deserializing untrusted data with formats that can create arbitrary objects, such as pickle.
  • Renaming or removing a field without versioning.

Check yourself

  • Why is deserializing untrusted pickle data dangerous?
  • How should a date and a large id be sent in JSON?
  • Why should readers tolerate fields they do not know?

Related topics

Terms

Further reading