Course outline

2.9

Serialization: JSON, XML, Protobuf, Avro and Thrift

Text vs binary formats, their size and speed, and which one to pick for APIs, events and storage.

45 min Reading + animations

Tip: tap or hover any dotted-underlined term to see what it means.

By the end of this lesson you will be able to

  • Explain what serialization (marshalling) and deserialization are, and why every network call, cache entry, queue message and file needs them.
  • Place any format on two axes, text vs binary and schema-less vs schema-based, and predict its size, speed and evolution behaviour from that.
  • Use JSON safely: its type gaps (no 64-bit integers, no binary, no dates, no comments), the 2⁵³ problem with large IDs, base64 cost, and JSON Schema.
  • Read XML’s strengths (namespaces, XSD, mixed content) and its main security trap, XXE.
  • Decode Protocol Buffers by hand: tags, field numbers, wire types, varints and zigzag, unknown fields, presence in proto3, and code generation.
  • Explain how Avro works without tags (writer vs reader schema, resolution by name, schema registry) and how Thrift adds protocols and transports.
  • Know when MessagePack, CBOR, FlatBuffers or Cap’n Proto fit, and quote honest size and speed numbers.
  • Recognise unsafe deserialization (pickle, Java serialization, XXE) and choose a format for a public API, internal RPC, events and storage.

#Where this lesson fits

Every lesson so far has sent data somewhere. Lesson 1.1 showed a request with a JSON body and a Content-Type header; lesson 1.5 placed data formats in the presentation layer, next to character encoding (UTF-8) and compression, and listed JSON and XML among them. Lesson 1.3 showed a JWT: JSON, base64url-encoded. And lesson 2.3 covered how HTTP compresses a body with gzip, Brotli or zstd.

This lesson opens up the body itself: how an object in one program’s memory becomes bytes that a completely different program can turn back into an object. The formats taught here are JSON, XML, Protocol Buffers (Protobuf), Apache Avro and Apache Thrift, plus a few others you will meet in production.

Two neighbours are deliberately kept short here:

  • Changing a format safely (adding, removing and renaming fields; backward, forward and full compatibility) is lesson 2.10, schema evolution. This lesson shows only the mechanisms that make it possible.
  • API design (REST vs gRPC vs GraphQL) is Chapter 26, messaging and the schema registry in depth are Chapters 30–32, and columnar formats (Parquet, ORC) are Chapter 42.

#What serialization is and why it exists

A running program keeps data as objects in memory: structs and classes with fields, pointers to strings stored elsewhere on the heap, padding chosen by the compiler, integers in the CPU’s byte order. None of that can leave the process. Another program cannot read your memory, a different language would not understand the layout, and a pointer means nothing on another machine.

  • Serialization (also called marshalling or encoding) turns an in-memory object or data structure into a flat sequence of bytes in an agreed format, so it can be saved to a file or sent over the network to another machine.
  • Deserialization (unmarshalling, decoding, parsing) does the reverse: it reads those bytes and builds a new object with the same values.

You serialize far more often than you notice:

WhereExample
Network callsAn HTTP API body, a gRPC message, a WebSocket frame
StorageA database row or document, a log line, a backup, a file in a data lake
CachesA user profile stored in Redis or Memcached
Queues and streamsA Kafka message, an SQS job, an event on a bus
Between processesA worker pool on one machine, a browser and its web worker

Serialization round trip: object → bytes → object

Only bytes can cross a network or sit in a file. Serialization turns an in-memory object into bytes; deserialization builds a new object from them.

Scenario

The order service tells the email service about a new user, as JSON over HTTP.

Order servicewritten in Go
Wire or disk
Email servicewritten in Java
User{id: 101, name: "Rahim", …} in memory
Step 1 of 6
1

In memory the object is a struct with pointers to strings elsewhere in the heap, padding between fields and a layout chosen by Go. No other process can read that memory, and a Java program would not understand the layout even if it could.

Every format has to answer the same five questions, and the answers explain almost everything about its size, speed and safety:

  1. Which types exist? Integers of which sizes, floats, strings, bytes, booleans, null, dates, maps, lists.
  2. How is a field identified? By name (JSON, XML, MessagePack), by number (Protobuf, Thrift) or by position in a schema (Avro).
  3. Where does one value end? By delimiters that must be escaped (quotes, commas, closing tags) or by a length prefix (binary formats).
  4. Where is the schema? Inside every message (self-describing), in a separate file compiled into code, or in a registry.
  5. Text or binary? Can a human read the bytes, or only a program?

#Two axes: text vs binary, schema-less vs schema-based

The notes split formats into two families:

  1. Human-readable text formats: JSON, XML, YAML. Easy to read and debug, but larger and slower to parse.
  2. Binary formats: Protocol Buffers, Apache Avro, Apache Thrift. Not readable by people, but compact, memory-efficient and fast to process.

A second axis matters just as much: does the data carry its own field names, or does the reader need a schema?

Schema-less (self-describing)Schema-based
TextJSON, XML, YAMLJSON or XML validated by JSON Schema or XSD (the schema is optional, the data still carries names)
BinaryMessagePack, CBOR, BSON (field names inside, just more compact)Protobuf, Thrift (field numbers on the wire), Avro (nothing on the wire but values)

Self-describing data can be read by anyone at any time, which is why public APIs love it, but it repeats every field name in every message. Schema-based binary data sends only numbers or nothing at all, so it is small, but a reader without the schema sees noise.

Here is the notes’ user record encoded six ways. Watch the green value bytes stay the same while the amber and blue structure bytes shrink:

One user record, six encodings

The notes’ user {id: 101, name: "Rahim", email: "rahim@example.com", isActive: true}. The values are the same 24–29 bytes everywhere; what differs is how much structure travels with them.

Scenario

Minified JSON, the way an API sends it. Pretty-printed with two-space indent it is 86 bytes.

{"id":

brace + key "id" · 6

101

101 as 3 digits · 3

Size so far: 9 bytes (values 3 · structure 6)

field names / keys (text)the data itselfframing / headers
Step 1 of 4
1

JSON is self-describing: the key "id" travels inside the message, so any reader can understand it without a schema. The price is paid on every message.

#JSON

JSON (JavaScript Object Notation) is the most popular data-exchange format of the web. It is the de facto standard for browsers and REST APIs, and almost every language has a built-in parser. The notes’ example:

{
  "id": 101,
  "name": "Rahim",
  "email": "rahim@example.com",
  "isActive": true
}

It is a text, key-value format and schema-less (self-describing): the field names travel with the data, so no separate schema is needed to read it.

Advantages (from the notes): very simple, best for people to read, debugging is easy (you can read a response with curl), built-in support nearly everywhere, the default for browsers and REST.

Disadvantages (from the notes): verbosity, because keys are sent again in every message (69 bytes for 29 bytes of data in our example); parsing overhead, because turning text into objects costs CPU; and no schema enforcement, because nothing stops a sender from putting a string where a number was expected (JSON Schema can add checks, below).

#What JSON can and cannot say

JSON has exactly six kinds of value: string, number, boolean (true/false), null, object, array. Everything else is a convention between sender and receiver. The current standard is RFC 8259 (2017), which also requires UTF-8 for JSON exchanged between systems.

Missing in JSONWhat people do insteadThe catch
64-bit integersSend them as stringsEvery client must know which strings are numbers
Binary database64 inside a string33% bigger, extra CPU, whole value in memory
Dates and timesISO 8601 strings such as "2026-11-02T10:15:00Z"Time zones and precision are up to you
CommentsNone in data; JSON variants with comments exist for config filesStandard parsers reject them
Trailing commasNoneA common hand-editing error
NaN, Infinitynull or a stringSome encoders emit invalid JSON
Unique keysThe standard only says keys should be uniqueParsers disagree on duplicates (first wins vs last wins), which attackers exploit when two services parse the same body differently

#The 2⁵³ problem: large IDs in JavaScript

The JSON grammar allows numbers with any number of digits, but JavaScript parses every number into an IEEE 754 double. A double has a 53-bit mantissa, so integers are exact only up to 2⁵³ − 1 = 9,007,199,254,740,991 (Number.MAX_SAFE_INTEGER). Database BIGINT keys, Snowflake-style IDs (timestamp + machine + counter) and many payment or order IDs go past that.

JSON.parse('{"id":1234567890123456789}').id returns 1234567890123456800. No error, no warning; just a different number, which can point at another record.

A 64-bit ID through JSON.parse in a browser

JavaScript numbers are IEEE 754 doubles: integers are exact only up to 2⁵³ − 1 = 9,007,199,254,740,991. Database and Snowflake-style IDs often go higher.

Scenario

BrowserJavaScript
Order APIJava, long ids
GET /orders/latest
Step 1 of 3
1

The page asks for the latest order.

The fixes, best first:

  1. Send IDs as strings ("id": "1234567890123456789"). IDs are labels, not quantities. Twitter’s API added id_str next to id for exactly this reason, and Protobuf’s official JSON mapping writes every int64 as a string.
  2. If clients must calculate with big integers, parse them into BigInt with a parser that supports it.
  3. Decide it in the API contract on day one: changing a number to a string later is a breaking change (Chapter 26).

#Binary data and base64

JSON strings are text, so files, images, hashes and encrypted blobs must be base64-encoded: every 3 bytes become 4 characters, a 33% size increase plus CPU on both sides (the JWT in lesson 1.3 is the same idea with the URL-safe alphabet). For anything bigger than a few kilobytes, send the bytes as bytes (a raw or multipart body, or a pre-signed upload URL) and put only an ID in the JSON. Binary formats have a real bytes type with just a length prefix.

#JSON Schema

JSON Schema describes what a valid document looks like: types, required fields, ranges, patterns, and whether extra keys are allowed. The current version is 2020-12, and OpenAPI 3.1 uses it for request and response bodies.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id", "name", "email"],
  "properties": {
    "id": { "type": "string", "pattern": "^[0-9]{1,19}$" },
    "name": { "type": "string", "maxLength": 100 },
    "email": { "type": "string", "format": "email" },
    "isActive": { "type": "boolean", "default": true }
  },
  "additionalProperties": false
}

Unlike Protobuf, the schema is not needed to read the data; it is a validation step you choose to run, usually at the API edge. That keeps JSON flexible and also means nothing is enforced unless someone wires the validator in.

#XML

XML (Extensible Markup Language) is an older and very rich markup language, widely used in enterprise systems. The notes’ example:

<User>
  <Id>101</Id>
  <Name>Rahim</Name>
  <Email>rahim@example.com</Email>
  <IsActive>true</IsActive>
</User>

It is a tag-based tree: like HTML, every element has an opening and a closing tag. It supports schemas: an XSD (XML Schema Definition) or the older DTD (Document Type Definition) can validate a document strictly, including types, order and cardinality.

Advantages (from the notes): excellent for complex trees and metadata; namespaces let documents from different systems be mixed without name clashes; attributes carry metadata next to the content.

<order xmlns="urn:example:shop" xmlns:pay="urn:example:payments">
  <user id="101" active="true">Rahim</user>
  <pay:method>card</pay:method>
</order>

Here id and active are attributes, and pay:method belongs to the payments namespace, so it cannot be confused with a method element from the shop.

Disadvantages (from the notes): extremely verbose, because every name appears twice (100 bytes for our user, about 1.45 times the JSON); heavy parsing, whether with a DOM parser (builds the whole tree in memory) or a streaming SAX parser (fires events as it reads, little memory but harder code); and in modern web APIs and microservices it has largely given way to JSON and Protobuf.

XML is far from dead, though. It is still the format of SOAP web services, SAML single sign-on, RSS and Atom feeds, SVG images, Office documents (a .docx is a zip of XML files), Android layouts, Maven’s pom.xml, and much of banking and e-invoicing (ISO 20022 payment messages).

#XXE: the parser that reads your files

XML has a feature JSON lacks: a document can declare entities, named pieces of text, and an external entity can point at a URL or a file. An old or misconfigured parser fetches it while parsing. That is the XXE (XML external entity) attack: an attacker’s document reads /etc/passwd, calls internal URLs (server-side request forgery) or, with nested entities (“billion laughs”), expands a few kilobytes into gigabytes of memory. The fix is one setting: disable DTDs and external entities in the parser. It is animated with the other deserialization attacks below.

#Protocol Buffers (Protobuf)

Protocol Buffers were created at Google (used internally since about 2001, open-sourced in 2008). Protobuf is language-neutral and platform-neutral, a very fast binary format, and the message format of gRPC. The notes’ schema:

// user.proto
syntax = "proto3";

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
  bool is_active = 4;
}

How it works, as the notes list it:

  1. Schema in a .proto file: the shape of the data is defined first.
  2. Field numbers (tags): every field gets a unique number (id = 1). When encoding, the name is never sent, only the number.
  3. Code generation: the compiler protoc generates classes for Python, Go, Java, C++, C#, Kotlin, Swift and more; your code uses those classes instead of building bytes by hand.
protoc --python_out=. --go_out=. user.proto

#On the wire: tags, wire types and varints

A Protobuf message is just a list of fields, each written as a key followed by a value. The key is one varint holding the field number and a 3-bit wire type:

key = (field_number << 3) | wire_type

The wire type tells a reader how to find the end of the value without knowing the schema:

Wire typeNameUsed forHow long is the value?
0VARINTint32, int64, uint32, uint64, sint32, sint64, bool, enumUntil a byte with top bit 0
1I64fixed64, sfixed64, doubleAlways 8 bytes
2LENstring, bytes, nested messages, packed repeated fieldsA varint length, then that many bytes
3, 4SGROUP, EGROUPGroups (deprecated, proto2 only)Until the matching end group
5I32fixed32, sfixed32, floatAlways 4 bytes

Our user, byte by byte (30 bytes in total):

08 65                                   field 1, varint: 101
12 05 52 61 68 69 6d                    field 2, LEN 5: "Rahim"
1a 11 72 61 68 69 6d 40 65 78 61 6d     field 3, LEN 17: "rahim@example.com"
      70 6c 65 2e 63 6f 6d
20 01                                   field 4, varint: 1 (true)

0x08 is field 1 with wire type 0 (1 × 8 + 0); 0x12 is field 2 with wire type 2 (2 × 8 + 2 = 18); 0x1a is field 3 (26); 0x20 is field 4 (32).

Varints store an integer 7 bits per byte, low group first; the top bit of each byte is a continuation bit (1 = another byte follows). So 0–127 take one byte, 128–16,383 two, and a full 64-bit value up to ten. The classic example is 300 = 1 0010 1100: low 7 bits 010 1100 with the continuation bit gives ac, the remaining 10 gives 02, so 300 is ac 02.

Negative numbers are the trap. An int32 of −1 is sign-extended to 64 bits and takes 10 bytes. The sint32/sint64 types first apply zigzag encoding, which maps 0, −1, 1, −2, 2 to 0, 1, 2, 3, 4, so −1 takes one byte. For values that are usually large and evenly spread (hashes, random IDs) fixed64/fixed32 are smaller than varints.

Varint encoding, step by step

Protobuf, Avro and Thrift compact all store integers as varints: 7 bits of the number per byte, and the top bit says whether another byte follows.

Scenario

The classic example: a field `int32 id = 1` holding 300.

1 0010 1100

300 in binary (9 bits)

the data itselfbinary tag: field number + type
Step 1 of 6
1

300 needs 9 bits. A fixed int32 would always spend 4 bytes; a varint spends only as many bytes as the number needs.

Two more space savers: field numbers 1–15 have a one-byte key (16–2047 take two), so give them to the most frequent fields; and in proto3 a field equal to its default (0, false, empty string) is not written at all. Repeated numbers are packed by default in proto3: one key, one length, then all the varints.

#Field numbers are the contract

Because only numbers travel, the .proto file’s field numbers are the real contract, and the names are just for code. Renaming email to email_address changes nothing on the wire; changing email = 3 to email = 5 breaks every reader. Field numbers go up to 536,870,911 (2²⁹ − 1), and 19000–19999 are reserved for the implementation.

When an old reader meets a field number it does not know, the wire type still tells it how long the value is, so it skips it and, since protobuf 3.5 (2017), keeps it in the message’s unknown fields. When the message is serialized again, those bytes go out unchanged. This is what lets a new producer talk to old consumers.

An old Protobuf reader meets a new field

Version 2 of the producer added `string referral_code = 5`. Version 1 readers have never heard of field 5.

Scenario

A v2 message (id, name, is_active, referral_code) read by v1 code.

08

field 1, varint · 1

65

101 · 1

12

field 2, length-delimited · 1

05

length 5 · 1

526168696d

Rahim · 5

20

field 4, varint · 1

01

true · 1

Reader has built: User{id: 101, name: "Rahim", is_active: true}

Size so far: 11 bytes (values 7 · structure 4)

binary tag: field number + typethe data itselflength prefixskipped or kept unparsed
Step 1 of 3
1

Fields 1, 2 and 4 are in the v1 schema, so they decode normally.

What you may and may not change, and the reserved keyword in detail, are in lesson 2.10.

#Defaults, presence and optional

proto3 does not send default values, so by default a reader cannot tell “the field was 0” from “the field was not set”. That is fine for a counter and wrong for, say, a discount that may legitimately be 0. The fixes:

  • optional int32 discount = 6; (back in proto3 since protobuf 3.15, 2021) gives the field explicit presence: generated code gets has_discount(), and an explicit 0 is sent.
  • Wrapper types such as google.protobuf.Int32Value did the same job before that.
  • proto3 has no required. proto2 had it, and it caused outages: a required field can never be removed without breaking someone. Validate required business fields in code or with a validation library instead.
  • Newer Protobuf releases replace syntax = "proto2" / "proto3" with editions (edition = "2023";), which turn these behaviours into per-file or per-field features. The wire format does not change.

#gRPC, JSON mapping and tooling

Protobuf is the default message format of gRPC, which adds services (rpc GetUser(GetUserRequest) returns (User);), HTTP/2 transport, streaming and generated clients; gRPC itself is taught in Chapter 26. Protobuf also has an official JSON mapping, so the same message can be served as JSON to browsers: field names become lowerCamelCase (is_active → isActive), int64 becomes a string, and bytes become base64.

Advantages (from the notes, corrected): very small, because tags replace names (30 vs 69 bytes here); fast, because there is no text to scan (often several times faster than general-purpose JSON libraries, though how much depends heavily on language, library and data, see below); typed: generated code will not let you put a string into an int32 field.

Disadvantages (from the notes): binary, so you cannot read it in a text editor (protoc --decode_raw helps, see Try it yourself); and an extra code generation step in every build.

#Apache Avro

Apache Avro is a binary serialization framework from the Hadoop project, now heavily used with Apache Kafka for big-data pipelines. Its schema is written in JSON. The notes’ schema:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": "string"},
    {"name": "is_active", "type": "boolean"}
  ]
}

The key difference from Protobuf: Avro sends no field names, no tags and no types in the data, only the raw values one after another, in schema order (the notes’ wording “like Protobuf” is misleading here: Protobuf does send a tag per field). Ints and longs are zigzag varints, strings are a length plus UTF-8 bytes, a boolean is one byte. Our user is 27 bytes: ca 01 (101), 0a + “Rahim”, 22 + the email, 01.

The consequence: to read Avro you must have the exact schema it was written with (the writer’s schema). Without it the bytes are meaningless; nothing in them even marks where one field ends.

#Writer’s schema and reader’s schema

Avro’s answer to change is schema resolution. The reader decodes with the writer’s schema, then maps the result onto its own reader’s schema:

  • Fields are matched by name (or by an alias, which is how you rename).
  • A field the writer has and the reader does not is ignored.
  • A field the reader has and the writer does not gets the default from the reader’s schema; without a default, resolution fails.
  • Some types may be promoted: int → long → float → double, string ↔ bytes.

Optional fields are written as a union with null: {"name": "referral_code", "type": ["null", "string"], "default": null}.

#Where the writer’s schema lives

  • Avro object container files (Hadoop, data lakes): the full schema is stored once in the file header, followed by blocks of records separated by sync markers. Blocks can be compressed and the file can be split between workers, which is why Avro became the row format of the Hadoop world.
  • Kafka with a schema registry: sending the schema with every message would cost more than the data, so each message starts with 5 bytes: a magic byte 0 and a 4-byte schema ID. The consumer fetches the schema for that ID from the registry once and caches it. The registry also checks compatibility when a new schema version is registered and refuses breaking ones.
  • Avro RPC exchanges schemas in a handshake (rarely used today).

Avro in Kafka: writer schema, reader schema and the registry

Avro bytes carry no field names or tags. The reader decodes with the writer’s schema, then resolves it against its own schema field by field, by name.

Scenario

v2 added `referral_code` (type ["null","string"], default null). The consumer still runs v1.

Producerwriter schema v2
Schema registry
Kafka topicusers
Consumerreader schema v1
register schema v2
Step 1 of 6
1

On first use the producer’s serializer registers its schema under the subject users-value.

Avro does not require code generation: it can be used with generic records (a map-like object built from the schema at run time), which suits pipelines that handle many different schemas, such as Kafka Connect, Spark and data lakes. Generated classes (“specific records”) are available too. (The notes call this “dynamic typing”; the data is still strictly typed by the schema.)

Advantages (from the notes, corrected): very compact, because the data holds only values; and with a schema registry, Kafka messages carry a 5-byte ID instead of the schema. The notes call Avro “currently the smallest binary format”: it is often the smallest for dense records like ours (27 bytes vs Protobuf’s 30), but not always. Avro zigzags every int (101 costs 2 bytes, not 1), and every optional field costs at least a 1-byte union index even when it is null, while Protobuf writes nothing for an unset field. For sparse records with many empty optional fields, Protobuf is often smaller.

Disadvantage (from the notes): the reader must know the writer’s schema (or an equivalent one), otherwise the data is meaningless. That makes the registry (or the file header) a critical dependency.

#Apache Thrift

Apache Thrift was created at Facebook (open-sourced in 2007) and given to the Apache Software Foundation. It is a cross-language RPC and serialization framework. Like Protobuf, it uses an IDL (interface definition language) file with numbered fields. The notes’ example, plus the service it would usually come with:

// user.thrift
struct User {
  1: i32 id,
  2: string name,
  3: string email,
  4: bool isActive
}

service UserService {
  User getUser(1: i32 id)
}

It is more than a format. The thrift compiler generates the data classes and complete RPC clients and servers, built from swappable layers:

LayerChoicesWhat it decides
ProtocolBinary, compact, JSONHow values become bytes (our user: 48 bytes binary, 31 compact)
TransportBuffered, framed, memory, HTTP, fileHow bytes are framed and moved (the notes’ “Buffered, Framed, Memory”)
ServerSimple, threaded, thread pool, non-blockingHow requests are handled (a non-blocking server needs the framed transport)

Thrift: your struct inside a protocol inside a transport

Thrift is a full RPC stack. The protocol decides how values become bytes; the transport decides how bytes are framed and moved. Both are chosen in code and are not negotiated.

Scenario

Binary protocol over a framed transport, the common server setup.

Struct User1: i32 id, 2: string name, 3: string email, 4: bool isActive
Step 1 of 5
1

The generated client turns getUser(42) into a call. The data types come from the .thrift IDL file, compiled for each language with the thrift compiler.

Advantages (from the notes): one IDL generates both serialization and a complete client and server; several transport mechanisms are supported.

Disadvantages (from the notes, updated): a smaller community than Protobuf and harder documentation. Today Thrift is mostly found in systems built in the 2010s (Facebook’s own fork, fbthrift; Twitter’s Finagle stack; older Cassandra and HBase interfaces; Hadoop-era tools). New projects almost always choose Protobuf and gRPC.

#Other formats you will meet

  • MessagePack: “binary JSON”. Same maps and lists, no schema, but type bytes and lengths instead of quotes and punctuation: 50 bytes for our user. Easy to swap in where JSON is used (Redis Lua scripts, Fluentd logs, some real-time apps).
  • CBOR (Concise Binary Object Representation): an IETF standard (RFC 8949) with a similar idea to MessagePack, designed for small devices; used in IoT protocols and in passkeys, whose authenticator data is CBOR.
  • BSON: MongoDB’s storage and wire format. Adds types JSON lacks (dates, 64-bit integers, binary, object IDs) and stores lengths so a document can be traversed quickly; usually not smaller than JSON.
  • Zero-copy formats, FlatBuffers and Cap’n Proto: the bytes are laid out so they can be read in place, without a parsing step that builds objects. Accessing one field of a large message costs almost nothing, which suits games, mobile apps loading big read-only data, and machine-learning model files (FlatBuffers comes from Google and is used for on-device machine-learning models; Cap’n Proto, by Protobuf’s former lead, is used inside Cloudflare Workers). The cost: bigger messages (alignment, offsets) and less convenient APIs.
  • Columnar formats, Parquet and ORC: store each column together instead of each row, for analytics that read a few columns of billions of rows. They are a storage design topic: Chapter 42.
  • YAML: the notes’ third text format. Great for hand-written configuration (Kubernetes, CI pipelines), poor on the wire: slow to parse, indentation-sensitive, and full of surprises (in older YAML 1.1 parsers the country code NO becomes false).
  • Language-native formats: Java serialization, Python’s pickle, PHP’s serialize, .NET’s BinaryFormatter (now removed from .NET). Convenient inside one program, dangerous across any trust boundary (next section), and unreadable from other languages.

#Size and speed: honest numbers

Sizes for the notes’ user record, all computed byte by byte:

FormatBytesNotes
XML100138 with the <?xml …?> declaration; more with indentation
JSON6986 pretty-printed
MessagePack50Keys still included
Thrift binary protocol48Fixed-width ints and lengths
Thrift compact protocol31Varints, field-id deltas
Protobuf30Tags + varints
Avro2732 with the Kafka registry prefix

And for 1,000 similar users (synthetic test data):

RawgzipBrotli
JSON79.9 KB9.4 KB5.0 KB
Protobuf39.9 KB9.5 KBn/a

What to say about speed in an interview, honestly:

  • It depends far more on the language and library than on the format. With mainstream libraries in Java, Go or C++, Protobuf decoding is commonly 2–10 times faster than JSON (the notes’ “5–10×” is one point in that range, not a law).
  • Fast JSON parsers such as simdjson parse at gigabytes per second, faster than many Protobuf decoders.
  • In the browser, JSON.parse is native C++ and very fast; a JavaScript Protobuf library is often no faster, and the win is the smaller payload.
  • Zero-copy formats skip parsing entirely, so “decode” time is near zero until you touch the fields.
  • Measure with your own data: string-heavy records gain little from binary (strings cost the same everywhere); number-heavy and many-small-message workloads gain a lot.

#Security: never deserialize untrusted data into arbitrary types

Data formats such as JSON and Protobuf can only produce values: numbers, strings, lists, and the one type your code asked for. Some formats can instead describe any object of any class, and rebuilding that object runs code. Fed by an attacker, they run the attacker’s code.

Unsafe deserialization: when the data chooses the code

Some formats can describe any object of any class, and rebuilding it runs code. Fed attacker bytes, they run attacker code.

Scenario

An import endpoint accepts a pickled object, because it was quick to write.

Attacker
API server
Server OS
POST /import (crafted pickle)
Step 1 of 3
1

A pickle is not just data: it is a small program of instructions that rebuild an object, and one instruction can say “call this function with these arguments”.

DangerExampleDefence
Arbitrary object deserializationPython pickle.loads, Java ObjectInputStream.readObject (gadget chains through libraries on the classpath), .NET BinaryFormatter, PHP unserialize, unsafe YAML loadersNever on untrusted input. Use JSON or Protobuf into a fixed type; if you must, allow-list classes (Java’s deserialization filters) and use yaml.safe_load
XML external entitiesXXE reading files, internal requests, billion-laughs memory blow-upDisable DTDs and external entities in the parser
Polymorphic JSONA JSON library configured to create the class named in a "@type" fieldTurn off default typing; allow-list subtypes
Resource exhaustion1 GB bodies, JSON nested 100,000 levels deep (stack overflow), huge strings or arraysBody size limits, depth limits, length limits in the schema
Parser differentialsDuplicate keys or odd numbers read differently by a gateway and a serviceOne parser and one canonical form at the boundary; reject duplicates

Insecure deserialization is a recurring entry in the OWASP Top 10 (it sits under software and data integrity failures), and it is taught again with the other web attacks in Chapter 34.

#Choosing a format

The notes’ comparison table, with the corrections from this lesson:

JSONXMLProtobufAvroThrift
TypeTextTextBinaryBinaryBinary (or JSON protocol)
Schema needed to read?No (JSON Schema optional)No (XSD optional)Yes (or only numbers and raw values)Yes, the exact writer schemaYes
Field identity on the wireNamesNames (twice)NumbersNone (position)Numbers
Size (our user)Medium, 69 BVery large, 100 BSmall, 30 BSmallest here, 27 BSmall, 31 B (compact)
Parse speedMedium (fast with good libraries)SlowFastFastFast
Schema evolutionManual, by conventionManual, XSD versionsEasy: field numbers, unknown fields keptExcellent: reader/writer resolution, registry checksEasy: field ids
Typical usePublic web APIs, REST, browsers, configLegacy and enterprise integrations, SOAP, SAML, documents, configMicroservices, gRPC, mobileKafka, Hadoop, data pipelinesCross-language RPC in older stacks

A practical rule of thumb:

BoundaryGood defaultWhy
Public API, partners, browsersJSON + OpenAPI / JSON Schema, with HTTP compressionEvery client can read it; debuggable; compression removes most of the size gap
Internal service-to-service RPCProtobuf over gRPCSmall, fast, typed, generated clients; the .proto is the contract
Events and streams (Kafka)Avro or Protobuf with a schema registry (JSON Schema is supported too)Many readers over a long time; compatibility enforced before data is written
Analytics storageParquet (or ORC); Avro for row-oriented landing filesColumnar scans and compression (Chapter 42)
CachesWhatever the service already uses; Protobuf or MessagePack to save memoryCache memory is expensive; keep a version in the key or value
Config filesYAML, JSON or TOMLWritten by people, read rarely
Games, real-time binary over WebSocketProtobuf, MessagePack or FlatBuffersMany small messages per second
Constrained devicesCBOR or ProtobufTiny code and messages

Which format where: one shop, four formats

Real systems mix formats. Each boundary picks the one that fits who reads it, how often, and for how long.

Scenario

Browser
API gateway
Order service
Payment service
Mobile app
Kafka
Data lake
JSON

Boundary

Format

JSON (+ gzip / br)

Readers

any browser, partner, curl

Contract

OpenAPI + JSON Schema

Step 1 of 2
1

Outside clients you do not control get JSON: every language and tool reads it, you can debug it by eye, and HTTP compression shrinks it well. Size matters less than reach here.

#Interview angles

  • “Why gRPC/Protobuf inside and JSON outside?” Inside, you control both ends and pay for every byte and CPU cycle at high call rates; outside, reach and debuggability matter more, and compression narrows the size gap.
  • “Your JavaScript client shows wrong order IDs.” 64-bit IDs above 2⁵³ lose precision in JSON.parse: send them as strings.
  • “How do you add a field without breaking consumers?” Protobuf: new field number, old readers skip and preserve it. Avro: add it with a default; the registry checks compatibility. Details in lesson 2.10.
  • “Why can’t a consumer read Avro without the registry?” The data has no tags; only the writer’s schema gives the bytes meaning.
  • “Is it safe to store pickled objects in Redis?” Only if nobody untrusted can ever write to that Redis. Prefer a data format.

#Try it yourself

# 1. The 2^53 problem (Node.js)
node -e 'console.log(JSON.parse("{\"id\":1234567890123456789}").id)'
#   1234567890123456800
node -e 'console.log(Number.MAX_SAFE_INTEGER)'
#   9007199254740991

# 2. JSON size, and what gzip does to one small message
printf '%s' '{"id":101,"name":"Rahim","email":"rahim@example.com","isActive":true}' | wc -c          # 69
printf '%s' '{"id":101,"name":"Rahim","email":"rahim@example.com","isActive":true}' | gzip -c | wc -c   # about 89

# 3. Encode the user with protoc and look at the 30 bytes (save the notes' user.proto first)
printf 'id: 101 name: "Rahim" email: "rahim@example.com" is_active: true' \
  | protoc --encode=User user.proto | xxd
#   00000000: 0865 1205 5261 6869 6d1a 1172 6168 696d  .e..Rahim..rahim
#   00000010: 4065 7861 6d70 6c65 2e63 6f6d 2001       @example.com .

# 4. Decode without the schema: you get numbers, not names
printf 'id: 101 name: "Rahim" email: "rahim@example.com" is_active: true' \
  | protoc --encode=User user.proto | protoc --decode_raw
#   1: 101
#   2: "Rahim"
#   3: "rahim@example.com"
#   4: 1

# 5. 300 as a varint
printf 'id: 300' | protoc --encode=User user.proto | xxd     # 08ac 02

# 6. A pickle is a program, not data (safe: this only disassembles it)
python3 -c "import pickle, pickletools; pickletools.dis(pickle.dumps({'id': 101}))"

To inspect Kafka messages encoded with Avro and a registry, Confluent’s kafka-avro-console-consumer (or any registry-aware tool) decodes them; a plain consumer shows the 5-byte prefix followed by unreadable bytes.

#Summary

Key takeaways

  • Serialization (marshalling) turns in-memory objects into bytes for the network, disk, caches and queues; deserialization builds new objects from them. Both sides agree only on the format, never on language or memory layout.
  • Formats differ in text vs binary and schema-less vs schema-based. Field identity is a name (JSON, XML, MessagePack), a number (Protobuf, Thrift) or a position (Avro).
  • JSON: universal and readable, but no 64-bit integers (exact only to 2⁵³ − 1: send big IDs as strings), no binary (base64 +33%), no dates, no comments. JSON Schema validates; it is not needed to read.
  • XML: verbose (names twice) and heavy to parse, but rich (namespaces, attributes, XSD); disable external entities to stop XXE.
  • Protobuf: key = (field number << 3) | wire type; varints (300 = ac 02), zigzag for sint, defaults not sent, optional for presence. Field numbers are the contract; old readers skip and keep unknown fields. Never reuse a number.
  • Avro: no tags at all, values in schema order; read with the writer’s schema, resolve to the reader’s schema by name, defaults fill gaps; schema in the file header or a registry (5-byte prefix in Kafka).
  • Thrift: IDL + full RPC stack with pluggable protocols (binary, compact, JSON) and transports (buffered, framed…) that both ends must configure identically.
  • Honest numbers: our user is 100 / 69 / 50 / 31 / 30 / 27 bytes (XML, JSON, MessagePack, Thrift compact, Protobuf, Avro); compression closes most of the gap for big payloads; speed depends on the library, so measure.
  • Never deserialize untrusted data into arbitrary types (pickle, Java serialization, BinaryFormatter, unsafe YAML): decode into fixed types and validate.
  • Choose by boundary: JSON for public APIs, Protobuf/gRPC inside, Avro or Protobuf + registry for events, columnar files for analytics.