2.9
Serialization: JSON, XML, Protobuf, Avro and Thrift
Text vs binary formats, their size and speed, and which one to pick for APIs, events and storage.
45 min Reading + animations
Tip: tap or hover any dotted-underlined term to see what it means.
By the end of this lesson you will be able to
- Explain what serialization (marshalling) and deserialization are, and why every network call, cache entry, queue message and file needs them.
- Place any format on two axes, text vs binary and schema-less vs schema-based, and predict its size, speed and evolution behaviour from that.
- Use JSON safely: its type gaps (no 64-bit integers, no binary, no dates, no comments), the 2⁵³ problem with large IDs, base64 cost, and JSON Schema.
- Read XML’s strengths (namespaces, XSD, mixed content) and its main security trap, XXE.
- Decode Protocol Buffers by hand: tags, field numbers, wire types, varints and zigzag, unknown fields, presence in proto3, and code generation.
- Explain how Avro works without tags (writer vs reader schema, resolution by name, schema registry) and how Thrift adds protocols and transports.
- Know when MessagePack, CBOR, FlatBuffers or Cap’n Proto fit, and quote honest size and speed numbers.
- Recognise unsafe deserialization (pickle, Java serialization, XXE) and choose a format for a public API, internal RPC, events and storage.
#Where this lesson fits
Every lesson so far has sent data somewhere. Lesson 1.1 showed a request with a JSON body and a Content-Type header; lesson 1.5 placed data formats in the presentation layer, next to character encoding (UTF-8) and compression, and listed JSON and XML among them. Lesson 1.3 showed a JWT: JSON, base64url-encoded. And lesson 2.3 covered how HTTP compresses a body with gzip, Brotli or zstd.
This lesson opens up the body itself: how an object in one program’s memory becomes bytes that a completely different program can turn back into an object. The formats taught here are JSON, XML, Protocol Buffers (Protobuf), Apache Avro and Apache Thrift, plus a few others you will meet in production.
Two neighbours are deliberately kept short here:
- Changing a format safely (adding, removing and renaming fields; backward, forward and full compatibility) is lesson 2.10, schema evolution. This lesson shows only the mechanisms that make it possible.
- API design (REST vs gRPC vs GraphQL) is Chapter 26, messaging and the schema registry in depth are Chapters 30–32, and columnar formats (Parquet, ORC) are Chapter 42.
#What serialization is and why it exists
A running program keeps data as objects in memory: structs and classes with fields, pointers to strings stored elsewhere on the heap, padding chosen by the compiler, integers in the CPU’s byte order. None of that can leave the process. Another program cannot read your memory, a different language would not understand the layout, and a pointer means nothing on another machine.
- Serialization (also called marshalling or encoding) turns an in-memory object or data structure into a flat sequence of bytes in an agreed format, so it can be saved to a file or sent over the network to another machine.
- Deserialization (unmarshalling, decoding, parsing) does the reverse: it reads those bytes and builds a new object with the same values.
You serialize far more often than you notice:
| Where | Example |
|---|---|
| Network calls | An HTTP API body, a gRPC message, a WebSocket frame |
| Storage | A database row or document, a log line, a backup, a file in a data lake |
| Caches | A user profile stored in Redis or Memcached |
| Queues and streams | A Kafka message, an SQS job, an event on a bus |
| Between processes | A worker pool on one machine, a browser and its web worker |
Serialization round trip: object → bytes → object
Only bytes can cross a network or sit in a file. Serialization turns an in-memory object into bytes; deserialization builds a new object from them.
Scenario
The order service tells the email service about a new user, as JSON over HTTP.
In memory the object is a struct with pointers to strings elsewhere in the heap, padding between fields and a layout chosen by Go. No other process can read that memory, and a Java program would not understand the layout even if it could.
Every format has to answer the same five questions, and the answers explain almost everything about its size, speed and safety:
- Which types exist? Integers of which sizes, floats, strings, bytes, booleans, null, dates, maps, lists.
- How is a field identified? By name (JSON, XML, MessagePack), by number (Protobuf, Thrift) or by position in a schema (Avro).
- Where does one value end? By delimiters that must be escaped (quotes, commas, closing tags) or by a length prefix (binary formats).
- Where is the schema? Inside every message (self-describing), in a separate file compiled into code, or in a registry.
- Text or binary? Can a human read the bytes, or only a program?
#Two axes: text vs binary, schema-less vs schema-based
The notes split formats into two families:
- Human-readable text formats: JSON, XML, YAML. Easy to read and debug, but larger and slower to parse.
- Binary formats: Protocol Buffers, Apache Avro, Apache Thrift. Not readable by people, but compact, memory-efficient and fast to process.
A second axis matters just as much: does the data carry its own field names, or does the reader need a schema?
| Schema-less (self-describing) | Schema-based | |
|---|---|---|
| Text | JSON, XML, YAML | JSON or XML validated by JSON Schema or XSD (the schema is optional, the data still carries names) |
| Binary | MessagePack, CBOR, BSON (field names inside, just more compact) | Protobuf, Thrift (field numbers on the wire), Avro (nothing on the wire but values) |
Self-describing data can be read by anyone at any time, which is why public APIs love it, but it repeats every field name in every message. Schema-based binary data sends only numbers or nothing at all, so it is small, but a reader without the schema sees noise.
Here is the notes’ user record encoded six ways. Watch the green value bytes stay the same while the amber and blue structure bytes shrink:
One user record, six encodings
The notes’ user {id: 101, name: "Rahim", email: "rahim@example.com", isActive: true}. The values are the same 24–29 bytes everywhere; what differs is how much structure travels with them.
Scenario
Minified JSON, the way an API sends it. Pretty-printed with two-space indent it is 86 bytes.
{"id":
brace + key "id" · 6
101
101 as 3 digits · 3
Size so far: 9 bytes (values 3 · structure 6)
JSON is self-describing: the key "id" travels inside the message, so any reader can understand it without a schema. The price is paid on every message.
#JSON
JSON (JavaScript Object Notation) is the most popular data-exchange format of the web. It is the de facto standard for browsers and REST APIs, and almost every language has a built-in parser. The notes’ example:
{
"id": 101,
"name": "Rahim",
"email": "rahim@example.com",
"isActive": true
}
It is a text, key-value format and schema-less (self-describing): the field names travel with the data, so no separate schema is needed to read it.
Advantages (from the notes): very simple, best for people to read, debugging is easy (you can read a response with curl), built-in support nearly everywhere, the default for browsers and REST.
Disadvantages (from the notes): verbosity, because keys are sent again in every message (69 bytes for 29 bytes of data in our example); parsing overhead, because turning text into objects costs CPU; and no schema enforcement, because nothing stops a sender from putting a string where a number was expected (JSON Schema can add checks, below).
#What JSON can and cannot say
JSON has exactly six kinds of value: string, number, boolean (true/false), null, object, array. Everything else is a convention between sender and receiver. The current standard is RFC 8259 (2017), which also requires UTF-8 for JSON exchanged between systems.
| Missing in JSON | What people do instead | The catch |
|---|---|---|
| 64-bit integers | Send them as strings | Every client must know which strings are numbers |
| Binary data | base64 inside a string | 33% bigger, extra CPU, whole value in memory |
| Dates and times | ISO 8601 strings such as "2026-11-02T10:15:00Z" | Time zones and precision are up to you |
| Comments | None in data; JSON variants with comments exist for config files | Standard parsers reject them |
| Trailing commas | None | A common hand-editing error |
NaN, Infinity | null or a string | Some encoders emit invalid JSON |
| Unique keys | The standard only says keys should be unique | Parsers disagree on duplicates (first wins vs last wins), which attackers exploit when two services parse the same body differently |
#The 2⁵³ problem: large IDs in JavaScript
The JSON grammar allows numbers with any number of digits, but JavaScript parses every number into an IEEE 754 double. A double has a 53-bit mantissa, so integers are exact only up to 2⁵³ − 1 = 9,007,199,254,740,991 (Number.MAX_SAFE_INTEGER). Database BIGINT keys, Snowflake-style IDs (timestamp + machine + counter) and many payment or order IDs go past that.
JSON.parse('{"id":1234567890123456789}').id returns 1234567890123456800. No error, no warning; just a different number, which can point at another record.
A 64-bit ID through JSON.parse in a browser
JavaScript numbers are IEEE 754 doubles: integers are exact only up to 2⁵³ − 1 = 9,007,199,254,740,991. Database and Snowflake-style IDs often go higher.
Scenario
The page asks for the latest order.
The fixes, best first:
- Send IDs as strings (
"id": "1234567890123456789"). IDs are labels, not quantities. Twitter’s API addedid_strnext toidfor exactly this reason, and Protobuf’s official JSON mapping writes everyint64as a string. - If clients must calculate with big integers, parse them into BigInt with a parser that supports it.
- Decide it in the API contract on day one: changing a number to a string later is a breaking change (Chapter 26).
#Binary data and base64
JSON strings are text, so files, images, hashes and encrypted blobs must be base64-encoded: every 3 bytes become 4 characters, a 33% size increase plus CPU on both sides (the JWT in lesson 1.3 is the same idea with the URL-safe alphabet). For anything bigger than a few kilobytes, send the bytes as bytes (a raw or multipart body, or a pre-signed upload URL) and put only an ID in the JSON. Binary formats have a real bytes type with just a length prefix.
#JSON Schema
JSON Schema describes what a valid document looks like: types, required fields, ranges, patterns, and whether extra keys are allowed. The current version is 2020-12, and OpenAPI 3.1 uses it for request and response bodies.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["id", "name", "email"],
"properties": {
"id": { "type": "string", "pattern": "^[0-9]{1,19}$" },
"name": { "type": "string", "maxLength": 100 },
"email": { "type": "string", "format": "email" },
"isActive": { "type": "boolean", "default": true }
},
"additionalProperties": false
}
Unlike Protobuf, the schema is not needed to read the data; it is a validation step you choose to run, usually at the API edge. That keeps JSON flexible and also means nothing is enforced unless someone wires the validator in.
#XML
XML (Extensible Markup Language) is an older and very rich markup language, widely used in enterprise systems. The notes’ example:
<User>
<Id>101</Id>
<Name>Rahim</Name>
<Email>rahim@example.com</Email>
<IsActive>true</IsActive>
</User>
It is a tag-based tree: like HTML, every element has an opening and a closing tag. It supports schemas: an XSD (XML Schema Definition) or the older DTD (Document Type Definition) can validate a document strictly, including types, order and cardinality.
Advantages (from the notes): excellent for complex trees and metadata; namespaces let documents from different systems be mixed without name clashes; attributes carry metadata next to the content.
<order xmlns="urn:example:shop" xmlns:pay="urn:example:payments">
<user id="101" active="true">Rahim</user>
<pay:method>card</pay:method>
</order>
Here id and active are attributes, and pay:method belongs to the payments namespace, so it cannot be confused with a method element from the shop.
Disadvantages (from the notes): extremely verbose, because every name appears twice (100 bytes for our user, about 1.45 times the JSON); heavy parsing, whether with a DOM parser (builds the whole tree in memory) or a streaming SAX parser (fires events as it reads, little memory but harder code); and in modern web APIs and microservices it has largely given way to JSON and Protobuf.
XML is far from dead, though. It is still the format of SOAP web services, SAML single sign-on, RSS and Atom feeds, SVG images, Office documents (a .docx is a zip of XML files), Android layouts, Maven’s pom.xml, and much of banking and e-invoicing (ISO 20022 payment messages).
#XXE: the parser that reads your files
XML has a feature JSON lacks: a document can declare entities, named pieces of text, and an external entity can point at a URL or a file. An old or misconfigured parser fetches it while parsing. That is the XXE (XML external entity) attack: an attacker’s document reads /etc/passwd, calls internal URLs (server-side request forgery) or, with nested entities (“billion laughs”), expands a few kilobytes into gigabytes of memory. The fix is one setting: disable DTDs and external entities in the parser. It is animated with the other deserialization attacks below.
#Protocol Buffers (Protobuf)
Protocol Buffers were created at Google (used internally since about 2001, open-sourced in 2008). Protobuf is language-neutral and platform-neutral, a very fast binary format, and the message format of gRPC. The notes’ schema:
// user.proto
syntax = "proto3";
message User {
int32 id = 1;
string name = 2;
string email = 3;
bool is_active = 4;
}
How it works, as the notes list it:
- Schema in a
.protofile: the shape of the data is defined first. - Field numbers (tags): every field gets a unique number (
id = 1). When encoding, the name is never sent, only the number. - Code generation: the compiler
protocgenerates classes for Python, Go, Java, C++, C#, Kotlin, Swift and more; your code uses those classes instead of building bytes by hand.
protoc --python_out=. --go_out=. user.proto
#On the wire: tags, wire types and varints
A Protobuf message is just a list of fields, each written as a key followed by a value. The key is one varint holding the field number and a 3-bit wire type:
key = (field_number << 3) | wire_type
The wire type tells a reader how to find the end of the value without knowing the schema:
| Wire type | Name | Used for | How long is the value? |
|---|---|---|---|
| 0 | VARINT | int32, int64, uint32, uint64, sint32, sint64, bool, enum | Until a byte with top bit 0 |
| 1 | I64 | fixed64, sfixed64, double | Always 8 bytes |
| 2 | LEN | string, bytes, nested messages, packed repeated fields | A varint length, then that many bytes |
| 3, 4 | SGROUP, EGROUP | Groups (deprecated, proto2 only) | Until the matching end group |
| 5 | I32 | fixed32, sfixed32, float | Always 4 bytes |
Our user, byte by byte (30 bytes in total):
08 65 field 1, varint: 101
12 05 52 61 68 69 6d field 2, LEN 5: "Rahim"
1a 11 72 61 68 69 6d 40 65 78 61 6d field 3, LEN 17: "rahim@example.com"
70 6c 65 2e 63 6f 6d
20 01 field 4, varint: 1 (true)
0x08 is field 1 with wire type 0 (1 × 8 + 0); 0x12 is field 2 with wire type 2 (2 × 8 + 2 = 18); 0x1a is field 3 (26); 0x20 is field 4 (32).
Varints store an integer 7 bits per byte, low group first; the top bit of each byte is a continuation bit (1 = another byte follows). So 0–127 take one byte, 128–16,383 two, and a full 64-bit value up to ten. The classic example is 300 = 1 0010 1100: low 7 bits 010 1100 with the continuation bit gives ac, the remaining 10 gives 02, so 300 is ac 02.
Negative numbers are the trap. An int32 of −1 is sign-extended to 64 bits and takes 10 bytes. The sint32/sint64 types first apply zigzag encoding, which maps 0, −1, 1, −2, 2 to 0, 1, 2, 3, 4, so −1 takes one byte. For values that are usually large and evenly spread (hashes, random IDs) fixed64/fixed32 are smaller than varints.
Varint encoding, step by step
Protobuf, Avro and Thrift compact all store integers as varints: 7 bits of the number per byte, and the top bit says whether another byte follows.
Scenario
The classic example: a field `int32 id = 1` holding 300.
1 0010 1100
300 in binary (9 bits)
300 needs 9 bits. A fixed int32 would always spend 4 bytes; a varint spends only as many bytes as the number needs.
Two more space savers: field numbers 1–15 have a one-byte key (16–2047 take two), so give them to the most frequent fields; and in proto3 a field equal to its default (0, false, empty string) is not written at all. Repeated numbers are packed by default in proto3: one key, one length, then all the varints.
#Field numbers are the contract
Because only numbers travel, the .proto file’s field numbers are the real contract, and the names are just for code. Renaming email to email_address changes nothing on the wire; changing email = 3 to email = 5 breaks every reader. Field numbers go up to 536,870,911 (2²⁹ − 1), and 19000–19999 are reserved for the implementation.
When an old reader meets a field number it does not know, the wire type still tells it how long the value is, so it skips it and, since protobuf 3.5 (2017), keeps it in the message’s unknown fields. When the message is serialized again, those bytes go out unchanged. This is what lets a new producer talk to old consumers.
An old Protobuf reader meets a new field
Version 2 of the producer added `string referral_code = 5`. Version 1 readers have never heard of field 5.
Scenario
A v2 message (id, name, is_active, referral_code) read by v1 code.
08
field 1, varint · 1
65
101 · 1
12
field 2, length-delimited · 1
05
length 5 · 1
526168696d
Rahim · 5
20
field 4, varint · 1
01
true · 1
Reader has built: User{id: 101, name: "Rahim", is_active: true}
Size so far: 11 bytes (values 7 · structure 4)
Fields 1, 2 and 4 are in the v1 schema, so they decode normally.
What you may and may not change, and the reserved keyword in detail, are in lesson 2.10.
#Defaults, presence and optional
proto3 does not send default values, so by default a reader cannot tell “the field was 0” from “the field was not set”. That is fine for a counter and wrong for, say, a discount that may legitimately be 0. The fixes:
optional int32 discount = 6;(back in proto3 since protobuf 3.15, 2021) gives the field explicit presence: generated code getshas_discount(), and an explicit 0 is sent.- Wrapper types such as
google.protobuf.Int32Valuedid the same job before that. - proto3 has no
required. proto2 had it, and it caused outages: a required field can never be removed without breaking someone. Validate required business fields in code or with a validation library instead. - Newer Protobuf releases replace
syntax = "proto2"/"proto3"with editions (edition = "2023";), which turn these behaviours into per-file or per-field features. The wire format does not change.
#gRPC, JSON mapping and tooling
Protobuf is the default message format of gRPC, which adds services (rpc GetUser(GetUserRequest) returns (User);), HTTP/2 transport, streaming and generated clients; gRPC itself is taught in Chapter 26. Protobuf also has an official JSON mapping, so the same message can be served as JSON to browsers: field names become lowerCamelCase (is_active → isActive), int64 becomes a string, and bytes become base64.
Advantages (from the notes, corrected): very small, because tags replace names (30 vs 69 bytes here); fast, because there is no text to scan (often several times faster than general-purpose JSON libraries, though how much depends heavily on language, library and data, see below); typed: generated code will not let you put a string into an int32 field.
Disadvantages (from the notes): binary, so you cannot read it in a text editor (protoc --decode_raw helps, see Try it yourself); and an extra code generation step in every build.
#Apache Avro
Apache Avro is a binary serialization framework from the Hadoop project, now heavily used with Apache Kafka for big-data pipelines. Its schema is written in JSON. The notes’ schema:
{
"type": "record",
"name": "User",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": "string"},
{"name": "is_active", "type": "boolean"}
]
}
The key difference from Protobuf: Avro sends no field names, no tags and no types in the data, only the raw values one after another, in schema order (the notes’ wording “like Protobuf” is misleading here: Protobuf does send a tag per field). Ints and longs are zigzag varints, strings are a length plus UTF-8 bytes, a boolean is one byte. Our user is 27 bytes: ca 01 (101), 0a + “Rahim”, 22 + the email, 01.
The consequence: to read Avro you must have the exact schema it was written with (the writer’s schema). Without it the bytes are meaningless; nothing in them even marks where one field ends.
#Writer’s schema and reader’s schema
Avro’s answer to change is schema resolution. The reader decodes with the writer’s schema, then maps the result onto its own reader’s schema:
- Fields are matched by name (or by an alias, which is how you rename).
- A field the writer has and the reader does not is ignored.
- A field the reader has and the writer does not gets the default from the reader’s schema; without a default, resolution fails.
- Some types may be promoted: int → long → float → double, string ↔ bytes.
Optional fields are written as a union with null: {"name": "referral_code", "type": ["null", "string"], "default": null}.
#Where the writer’s schema lives
- Avro object container files (Hadoop, data lakes): the full schema is stored once in the file header, followed by blocks of records separated by sync markers. Blocks can be compressed and the file can be split between workers, which is why Avro became the row format of the Hadoop world.
- Kafka with a schema registry: sending the schema with every message would cost more than the data, so each message starts with 5 bytes: a magic byte
0and a 4-byte schema ID. The consumer fetches the schema for that ID from the registry once and caches it. The registry also checks compatibility when a new schema version is registered and refuses breaking ones. - Avro RPC exchanges schemas in a handshake (rarely used today).
Avro in Kafka: writer schema, reader schema and the registry
Avro bytes carry no field names or tags. The reader decodes with the writer’s schema, then resolves it against its own schema field by field, by name.
Scenario
v2 added `referral_code` (type ["null","string"], default null). The consumer still runs v1.
On first use the producer’s serializer registers its schema under the subject users-value.
Avro does not require code generation: it can be used with generic records (a map-like object built from the schema at run time), which suits pipelines that handle many different schemas, such as Kafka Connect, Spark and data lakes. Generated classes (“specific records”) are available too. (The notes call this “dynamic typing”; the data is still strictly typed by the schema.)
Advantages (from the notes, corrected): very compact, because the data holds only values; and with a schema registry, Kafka messages carry a 5-byte ID instead of the schema. The notes call Avro “currently the smallest binary format”: it is often the smallest for dense records like ours (27 bytes vs Protobuf’s 30), but not always. Avro zigzags every int (101 costs 2 bytes, not 1), and every optional field costs at least a 1-byte union index even when it is null, while Protobuf writes nothing for an unset field. For sparse records with many empty optional fields, Protobuf is often smaller.
Disadvantage (from the notes): the reader must know the writer’s schema (or an equivalent one), otherwise the data is meaningless. That makes the registry (or the file header) a critical dependency.
#Apache Thrift
Apache Thrift was created at Facebook (open-sourced in 2007) and given to the Apache Software Foundation. It is a cross-language RPC and serialization framework. Like Protobuf, it uses an IDL (interface definition language) file with numbered fields. The notes’ example, plus the service it would usually come with:
// user.thrift
struct User {
1: i32 id,
2: string name,
3: string email,
4: bool isActive
}
service UserService {
User getUser(1: i32 id)
}
It is more than a format. The thrift compiler generates the data classes and complete RPC clients and servers, built from swappable layers:
| Layer | Choices | What it decides |
|---|---|---|
| Protocol | Binary, compact, JSON | How values become bytes (our user: 48 bytes binary, 31 compact) |
| Transport | Buffered, framed, memory, HTTP, file | How bytes are framed and moved (the notes’ “Buffered, Framed, Memory”) |
| Server | Simple, threaded, thread pool, non-blocking | How requests are handled (a non-blocking server needs the framed transport) |
Thrift: your struct inside a protocol inside a transport
Thrift is a full RPC stack. The protocol decides how values become bytes; the transport decides how bytes are framed and moved. Both are chosen in code and are not negotiated.
Scenario
Binary protocol over a framed transport, the common server setup.
The generated client turns getUser(42) into a call. The data types come from the .thrift IDL file, compiled for each language with the thrift compiler.
Advantages (from the notes): one IDL generates both serialization and a complete client and server; several transport mechanisms are supported.
Disadvantages (from the notes, updated): a smaller community than Protobuf and harder documentation. Today Thrift is mostly found in systems built in the 2010s (Facebook’s own fork, fbthrift; Twitter’s Finagle stack; older Cassandra and HBase interfaces; Hadoop-era tools). New projects almost always choose Protobuf and gRPC.
#Other formats you will meet
- MessagePack: “binary JSON”. Same maps and lists, no schema, but type bytes and lengths instead of quotes and punctuation: 50 bytes for our user. Easy to swap in where JSON is used (Redis Lua scripts, Fluentd logs, some real-time apps).
- CBOR (Concise Binary Object Representation): an IETF standard (RFC 8949) with a similar idea to MessagePack, designed for small devices; used in IoT protocols and in passkeys, whose authenticator data is CBOR.
- BSON: MongoDB’s storage and wire format. Adds types JSON lacks (dates, 64-bit integers, binary, object IDs) and stores lengths so a document can be traversed quickly; usually not smaller than JSON.
- Zero-copy formats, FlatBuffers and Cap’n Proto: the bytes are laid out so they can be read in place, without a parsing step that builds objects. Accessing one field of a large message costs almost nothing, which suits games, mobile apps loading big read-only data, and machine-learning model files (FlatBuffers comes from Google and is used for on-device machine-learning models; Cap’n Proto, by Protobuf’s former lead, is used inside Cloudflare Workers). The cost: bigger messages (alignment, offsets) and less convenient APIs.
- Columnar formats, Parquet and ORC: store each column together instead of each row, for analytics that read a few columns of billions of rows. They are a storage design topic: Chapter 42.
- YAML: the notes’ third text format. Great for hand-written configuration (Kubernetes, CI pipelines), poor on the wire: slow to parse, indentation-sensitive, and full of surprises (in older YAML 1.1 parsers the country code
NObecomesfalse). - Language-native formats: Java serialization, Python’s pickle, PHP’s
serialize, .NET’sBinaryFormatter(now removed from .NET). Convenient inside one program, dangerous across any trust boundary (next section), and unreadable from other languages.
#Size and speed: honest numbers
Sizes for the notes’ user record, all computed byte by byte:
| Format | Bytes | Notes |
|---|---|---|
| XML | 100 | 138 with the <?xml …?> declaration; more with indentation |
| JSON | 69 | 86 pretty-printed |
| MessagePack | 50 | Keys still included |
| Thrift binary protocol | 48 | Fixed-width ints and lengths |
| Thrift compact protocol | 31 | Varints, field-id deltas |
| Protobuf | 30 | Tags + varints |
| Avro | 27 | 32 with the Kafka registry prefix |
And for 1,000 similar users (synthetic test data):
| Raw | gzip | Brotli | |
|---|---|---|---|
| JSON | 79.9 KB | 9.4 KB | 5.0 KB |
| Protobuf | 39.9 KB | 9.5 KB | n/a |
What to say about speed in an interview, honestly:
- It depends far more on the language and library than on the format. With mainstream libraries in Java, Go or C++, Protobuf decoding is commonly 2–10 times faster than JSON (the notes’ “5–10×” is one point in that range, not a law).
- Fast JSON parsers such as simdjson parse at gigabytes per second, faster than many Protobuf decoders.
- In the browser,
JSON.parseis native C++ and very fast; a JavaScript Protobuf library is often no faster, and the win is the smaller payload. - Zero-copy formats skip parsing entirely, so “decode” time is near zero until you touch the fields.
- Measure with your own data: string-heavy records gain little from binary (strings cost the same everywhere); number-heavy and many-small-message workloads gain a lot.
#Security: never deserialize untrusted data into arbitrary types
Data formats such as JSON and Protobuf can only produce values: numbers, strings, lists, and the one type your code asked for. Some formats can instead describe any object of any class, and rebuilding that object runs code. Fed by an attacker, they run the attacker’s code.
Unsafe deserialization: when the data chooses the code
Some formats can describe any object of any class, and rebuilding it runs code. Fed attacker bytes, they run attacker code.
Scenario
An import endpoint accepts a pickled object, because it was quick to write.
A pickle is not just data: it is a small program of instructions that rebuild an object, and one instruction can say “call this function with these arguments”.
| Danger | Example | Defence |
|---|---|---|
| Arbitrary object deserialization | Python pickle.loads, Java ObjectInputStream.readObject (gadget chains through libraries on the classpath), .NET BinaryFormatter, PHP unserialize, unsafe YAML loaders | Never on untrusted input. Use JSON or Protobuf into a fixed type; if you must, allow-list classes (Java’s deserialization filters) and use yaml.safe_load |
| XML external entities | XXE reading files, internal requests, billion-laughs memory blow-up | Disable DTDs and external entities in the parser |
| Polymorphic JSON | A JSON library configured to create the class named in a "@type" field | Turn off default typing; allow-list subtypes |
| Resource exhaustion | 1 GB bodies, JSON nested 100,000 levels deep (stack overflow), huge strings or arrays | Body size limits, depth limits, length limits in the schema |
| Parser differentials | Duplicate keys or odd numbers read differently by a gateway and a service | One parser and one canonical form at the boundary; reject duplicates |
Insecure deserialization is a recurring entry in the OWASP Top 10 (it sits under software and data integrity failures), and it is taught again with the other web attacks in Chapter 34.
#Choosing a format
The notes’ comparison table, with the corrections from this lesson:
| JSON | XML | Protobuf | Avro | Thrift | |
|---|---|---|---|---|---|
| Type | Text | Text | Binary | Binary | Binary (or JSON protocol) |
| Schema needed to read? | No (JSON Schema optional) | No (XSD optional) | Yes (or only numbers and raw values) | Yes, the exact writer schema | Yes |
| Field identity on the wire | Names | Names (twice) | Numbers | None (position) | Numbers |
| Size (our user) | Medium, 69 B | Very large, 100 B | Small, 30 B | Smallest here, 27 B | Small, 31 B (compact) |
| Parse speed | Medium (fast with good libraries) | Slow | Fast | Fast | Fast |
| Schema evolution | Manual, by convention | Manual, XSD versions | Easy: field numbers, unknown fields kept | Excellent: reader/writer resolution, registry checks | Easy: field ids |
| Typical use | Public web APIs, REST, browsers, config | Legacy and enterprise integrations, SOAP, SAML, documents, config | Microservices, gRPC, mobile | Kafka, Hadoop, data pipelines | Cross-language RPC in older stacks |
A practical rule of thumb:
| Boundary | Good default | Why |
|---|---|---|
| Public API, partners, browsers | JSON + OpenAPI / JSON Schema, with HTTP compression | Every client can read it; debuggable; compression removes most of the size gap |
| Internal service-to-service RPC | Protobuf over gRPC | Small, fast, typed, generated clients; the .proto is the contract |
| Events and streams (Kafka) | Avro or Protobuf with a schema registry (JSON Schema is supported too) | Many readers over a long time; compatibility enforced before data is written |
| Analytics storage | Parquet (or ORC); Avro for row-oriented landing files | Columnar scans and compression (Chapter 42) |
| Caches | Whatever the service already uses; Protobuf or MessagePack to save memory | Cache memory is expensive; keep a version in the key or value |
| Config files | YAML, JSON or TOML | Written by people, read rarely |
| Games, real-time binary over WebSocket | Protobuf, MessagePack or FlatBuffers | Many small messages per second |
| Constrained devices | CBOR or Protobuf | Tiny code and messages |
Which format where: one shop, four formats
Real systems mix formats. Each boundary picks the one that fits who reads it, how often, and for how long.
Scenario
Boundary
Format
JSON (+ gzip / br)
Readers
any browser, partner, curl
Contract
OpenAPI + JSON Schema
Outside clients you do not control get JSON: every language and tool reads it, you can debug it by eye, and HTTP compression shrinks it well. Size matters less than reach here.
#Interview angles
- “Why gRPC/Protobuf inside and JSON outside?” Inside, you control both ends and pay for every byte and CPU cycle at high call rates; outside, reach and debuggability matter more, and compression narrows the size gap.
- “Your JavaScript client shows wrong order IDs.” 64-bit IDs above 2⁵³ lose precision in
JSON.parse: send them as strings. - “How do you add a field without breaking consumers?” Protobuf: new field number, old readers skip and preserve it. Avro: add it with a default; the registry checks compatibility. Details in lesson 2.10.
- “Why can’t a consumer read Avro without the registry?” The data has no tags; only the writer’s schema gives the bytes meaning.
- “Is it safe to store pickled objects in Redis?” Only if nobody untrusted can ever write to that Redis. Prefer a data format.
#Try it yourself
# 1. The 2^53 problem (Node.js)
node -e 'console.log(JSON.parse("{\"id\":1234567890123456789}").id)'
# 1234567890123456800
node -e 'console.log(Number.MAX_SAFE_INTEGER)'
# 9007199254740991
# 2. JSON size, and what gzip does to one small message
printf '%s' '{"id":101,"name":"Rahim","email":"rahim@example.com","isActive":true}' | wc -c # 69
printf '%s' '{"id":101,"name":"Rahim","email":"rahim@example.com","isActive":true}' | gzip -c | wc -c # about 89
# 3. Encode the user with protoc and look at the 30 bytes (save the notes' user.proto first)
printf 'id: 101 name: "Rahim" email: "rahim@example.com" is_active: true' \
| protoc --encode=User user.proto | xxd
# 00000000: 0865 1205 5261 6869 6d1a 1172 6168 696d .e..Rahim..rahim
# 00000010: 4065 7861 6d70 6c65 2e63 6f6d 2001 @example.com .
# 4. Decode without the schema: you get numbers, not names
printf 'id: 101 name: "Rahim" email: "rahim@example.com" is_active: true' \
| protoc --encode=User user.proto | protoc --decode_raw
# 1: 101
# 2: "Rahim"
# 3: "rahim@example.com"
# 4: 1
# 5. 300 as a varint
printf 'id: 300' | protoc --encode=User user.proto | xxd # 08ac 02
# 6. A pickle is a program, not data (safe: this only disassembles it)
python3 -c "import pickle, pickletools; pickletools.dis(pickle.dumps({'id': 101}))"
To inspect Kafka messages encoded with Avro and a registry, Confluent’s kafka-avro-console-consumer (or any registry-aware tool) decodes them; a plain consumer shows the 5-byte prefix followed by unreadable bytes.
#Summary
Key takeaways
- Serialization (marshalling) turns in-memory objects into bytes for the network, disk, caches and queues; deserialization builds new objects from them. Both sides agree only on the format, never on language or memory layout.
- Formats differ in text vs binary and schema-less vs schema-based. Field identity is a name (JSON, XML, MessagePack), a number (Protobuf, Thrift) or a position (Avro).
- JSON: universal and readable, but no 64-bit integers (exact only to 2⁵³ − 1: send big IDs as strings), no binary (base64 +33%), no dates, no comments. JSON Schema validates; it is not needed to read.
- XML: verbose (names twice) and heavy to parse, but rich (namespaces, attributes, XSD); disable external entities to stop XXE.
- Protobuf: key =
(field number << 3) | wire type; varints (300 =ac 02), zigzag forsint, defaults not sent,optionalfor presence. Field numbers are the contract; old readers skip and keep unknown fields. Never reuse a number. - Avro: no tags at all, values in schema order; read with the writer’s schema, resolve to the reader’s schema by name, defaults fill gaps; schema in the file header or a registry (5-byte prefix in Kafka).
- Thrift: IDL + full RPC stack with pluggable protocols (binary, compact, JSON) and transports (buffered, framed…) that both ends must configure identically.
- Honest numbers: our user is 100 / 69 / 50 / 31 / 30 / 27 bytes (XML, JSON, MessagePack, Thrift compact, Protobuf, Avro); compression closes most of the gap for big payloads; speed depends on the library, so measure.
- Never deserialize untrusted data into arbitrary types (pickle, Java serialization,
BinaryFormatter, unsafe YAML): decode into fixed types and validate. - Choose by boundary: JSON for public APIs, Protobuf/gRPC inside, Avro or Protobuf + registry for events, columnar files for analytics.
Answer on your own first, then read the explanation, even when you got it right.
Self-checkQuestion
What does deserialization (unmarshalling) do?
Self-checkQuestion
A team sends a C struct straight from memory over a socket to a Java service. The bytes arrive intact, but the id field reads as 1694498816 instead of 101. What is the most likely cause?
Self-checkQuestion
Which two statements about text and binary formats are correct?
Choose 2
Self-checkQuestion
A browser app receives {orderId: 9007199254740993} from a Java API and then requests that order. The API answers 404. Why?
Self-checkQuestion
What is the best long-term fix for 64-bit IDs in a public JSON API used by browsers?
Self-checkQuestion
A mobile app uploads a 6 MB photo inside a JSON body as base64. About how big is the request body?
Self-checkQuestion
Which statement about JSON Schema is correct?
Self-checkQuestion
A gateway keeps the first value of a duplicated JSON key and the service behind it keeps the last. What can an attacker do?
Self-checkQuestion
Which feature does XML have that JSON lacks?
Self-checkQuestion
A partner integration posts XML to an old parser. An attacker’s document defines an entity pointing at file:///etc/passwd and the response contains the file. What is the attack and the fix?
Self-checkQuestion
In Protobuf, what is the key byte for field number 3 with wire type 2 (length-delimited)?
Self-checkQuestion
How is the number 300 encoded as a Protobuf varint?
Self-checkQuestion
A temperature field is declared int32 and often holds small negative values. Messages are bigger than expected. What change helps most?
Self-checkQuestion
Which two choices make Protobuf messages smaller?
Choose 2
Self-checkQuestion
A v1 Protobuf reader receives a message containing field number 7, which its schema does not define, with wire type 2. What happens?
Self-checkQuestion
A v1 service in the middle of a pipeline reads a Protobuf message, edits one field and writes it on. Downstream v2 services suddenly lose a newly added field, but only when the middle service runs protobuf 3.2. Why?
Self-checkQuestion
A team deleted int32 age = 5 and months later added int32 zip_code = 5. Some services still run the old schema. What happens?
Self-checkQuestion
In proto3 without the optional keyword, a reader sees discount = 0. What can it conclude?
Self-checkQuestion
Which statement about required fields is correct?
Self-checkQuestion
Why can a Kafka consumer not decode an Avro message without the writer’s schema, while it could skip unknown Protobuf fields?
Self-checkQuestion
An Avro consumer uses reader schema v1. Messages arrive written with v2, which added referral_code with default null. What does schema resolution do?
Self-checkQuestion
Which two things are in the 5-byte prefix of a Kafka message produced with Confluent’s Avro serializer?
Choose 2
Self-checkQuestion
A developer adds a country field of type string with no default to an Avro schema and deploys the producer. The registry uses backward compatibility. What happens?
Self-checkQuestion
The schema registry is down. Which consumers are affected?
Self-checkQuestion
For which record is Protobuf likely to be smaller than Avro?
Self-checkQuestion
A Thrift client uses the buffered transport and the server uses the framed transport. Calls fail with frame size (2147549185) larger than max length. Why?
Self-checkQuestion
Which statement about Thrift is correct?
Self-checkQuestion
A game sends 60 small state updates per second per player over WebSockets and the payload must stay readable as maps without a schema build step. Which format fits best?
Self-checkQuestion
What do zero-copy formats such as FlatBuffers and Cap’n Proto offer?
Self-checkQuestion
A web API serves 2 MB JSON responses with gzip enabled. A colleague proposes switching to Protobuf to save bandwidth. What is the most accurate expectation?
Self-checkQuestion
Which two services are at risk of remote code execution through deserialization?
Choose 2
Self-checkQuestion
Which is not a sensible defence for a public endpoint that accepts JSON?
Self-checkQuestion
An e-commerce company designs its data formats. Which combination follows this lesson’s guidance?