Schema Evolution and Compatibility

By the end of this lesson you will be able to

• Explain why changing a message or record format is hard in a distributed system: rolling deploys, independent teams, rollbacks and data that lives for years.
• Define backward, forward and full compatibility (and their transitive variants) precisely, from the point of view of the writer schema and the reader schema.
• Decide who must deploy first for each compatibility mode, and recover when the order was wrong.
• Apply the evolution rules of Protobuf (field numbers, reserved, type changes, enums), Avro (defaults, aliases, unions, promotions), JSON (tolerant readers, JSON Schema) and Thrift.
• Rename a field, change a type or remove a field without an outage, using expand–migrate–contract.
• Describe how a schema registry checks compatibility and what the 5-byte prefix on every record means, and how event-sourced systems version events with upcasters.
• Catch breaking changes in CI with compatibility checks, breaking-change linters and consumer-driven contract tests.

#Where this lesson fits

Lesson 2.9 taught how data is turned into bytes: JSON and XML with names in every record, Protobuf and Thrift with field numbers, Avro with values only and a writer schema needed to read them. This lesson is about time: what happens when that format changes while old bytes and old programs still exist.

The format between two programs is a contract, the same way the request and response format is the contract between a client and a server (lesson 1.1). Changing one side of a contract is easy. Changing it while the other side keeps running, without stopping anything, is the whole subject here. Lesson 1.2 showed the rolling deploy (what is a server) and lesson 1.13 the load balancers behind it (web and app servers): those deploys are exactly when old and new formats meet.

Schema evolution is the ability to keep reading and writing data while its schema changes, without stopping the system or rewriting everything that already exists. A schema is the blueprint of the data: which fields exist, their types, which are optional and what their defaults are.

#Why schemas change, and why it is hard

Systems change every week. A table that has four fields today gets a fifth tomorrow, or loses one. An event gains a currency, an API response gets a new nested object, an ID becomes a string. On one machine you would change the code and the data together. In a distributed system you cannot, for four reasons:

And one more that teams forget: rollbacks. If v2 wrote data and you roll back to v1, old code must read new data.

Three lessons from the animation carry through the whole topic:

1. Additive changes are the safe kind: a new optional field with a sensible default.
2. A rename is a remove plus an add in any format where names are the contract (JSON, Avro). Done in one step, it breaks somebody.
3. Readers that write data back must keep fields they do not understand, or a rollback, or a v1 instance in a mixed fleet, quietly deletes data.

#Backward, forward and full compatibility

Every compatibility question has two schemas in it: the writer schema that encoded the data and the reader schema the code uses to decode it. The names describe the relationship from the point of view of the new schema:

 Backward compatibility : new code (new schema) can read OLD data
 Forward compatibility  : old code (old schema) can read NEW data
 Full compatibility     : both backward and forward

• Backward compatible change: a reader with the new schema can read data written with the old one. Typical way: add a new field as optional or with a default, so old records that lack it still decode.
• Forward compatible change: a reader with the old schema can read data written with the new one. This works because old readers ignore fields they do not know. Removing a field is the dangerous move here: it breaks old readers that still need it, unless that field had a default in their schema.
• Full compatible change: both. In practice, only adding or removing optional fields that have defaults.
• Breaking change: neither, for example changing a field’s type from a number to a string.

A schema registry turns these into modes that it enforces on every new version:

What each mode allows depends on the format (an Avro field without a default behaves differently from a Protobuf field), but for Avro the rule of thumb is: BACKWARD allows deleting fields and adding fields with defaults; FORWARD allows adding fields and deleting fields that had defaults; FULL allows only adding or deleting fields with defaults.

#Transitive: compatible with every version, not just the last

The plain modes compare a new schema only with the latest one. That protects consumers that read near the head of a topic. It does not protect anyone who reads old data: a new consumer that replays a topic with infinite retention, a compacted topic that keeps the newest record per key forever, an event store, or Avro files in a data lake. Compatibility is not transitive: v3 can be compatible with v2, and v2 with v1, while v3 is not compatible with v1.

Use the _TRANSITIVE modes for any data that is kept and re-read. Use the plain modes only where data is consumed quickly and never replayed.

#Who deploys first

The compatibility mode is a promise about one direction. The deploy order must follow it, because the registry checks the schema, not your deploy order:

Two consequences engineers learn the hard way:

• Rollbacks need forward compatibility. Suppose a producer change is only backward compatible, the producer ships, writes data, and then has to be rolled back. The old producer code is now a reader of new data, which BACKWARD never promised. If you want a free rollback, the change must be compatible in both directions (FULL), or you ship it in two releases: first teach every reader the new shape, then start writing it.
• Rolling back the producer does not remove what it wrote. Bad records stay in the topic. The way out is usually forward: ship a reader that can handle them, or skip them deliberately (Chapter 30 covers dead-letter queues).

#The rules, format by format

The rules differ because the formats identify fields differently (lesson 2.9): Protobuf and Thrift by number, Avro by name with the writer schema at hand, JSON by name with no schema at all.

#Protobuf: never touch a field number

On the wire every Protobuf field is (field number, wire type, value); names never travel. Each side compiles its own copy of the .proto, so they agree only on numbers. The rules follow from that:

1. Never change a field’s number, and never reuse a number. Data written with the old meaning will be read with the new one. This does not corrupt the whole message, but the field it hits is silently misread or lost, which is often worse because nothing fails.
2. Adding a field is safe in both directions. Old readers skip the unknown field (the wire type tells them how many bytes to skip) and, since Protobuf 3.5, proto3 keeps unknown fields and writes them back out when the message is re-serialized. New readers see the default (zero, empty string, false) when old data lacks the field.
3. Deleting a field is safe if you reserve it. Remove the field and reserve both its number and its name so nobody can bring them back with a new meaning:

message OrderPlaced {
  reserved 3;                 // was coupon_code, removed 2026-03
  reserved "coupon_code";
  string order_id = 1;
  int64  amount_paisa = 2;    // money as integer minor units, never double
  string currency = 4;        // added in v2; empty means BDT
  optional string gift_note = 5;  // optional: “not sent” differs from “empty”
}

4. Renaming is safe on the binary wire (names are not sent) but breaks anything that uses names: the JSON mapping of Protobuf, the text format, field masks, and every line of code that refers to the old name. Treat a rename as an API change for any JSON clients.
5. Changing a type is mostly unsafe. A few groups share a wire encoding and can be swapped, with care: int32, uint32, int64, uint64 and bool (values may be truncated, as in a C cast); sint32 and sint64; string and bytes (if the bytes are valid UTF-8); fixed32 and sfixed32; fixed64 and sfixed64. Anything else, such as int64 to string, gives a wire-type mismatch: most libraries then treat the value as an unknown field, so it silently reads as empty.
6. Enums are numbers. In proto3 enums are open: an old reader that receives a new value (say 4) keeps the number even though it has no name for it. The wire survives, but code with a switch and no default branch does not. Always handle unknown values, and make value 0 an ..._UNSPECIFIED placeholder so a missing value is never mistaken for a real one.
7. required is gone. proto2’s required turned every later removal into a breaking change, so proto3 dropped it (Protobuf Editions keep it only as a legacy feature for old files). Validate presence in code instead. Use optional in proto3 (supported since 3.15) when you must tell “not set” from the default.
8. Be careful with oneof and with changing a single field to repeated. Moving existing fields into or out of a oneof, or changing cardinality, can lose data in some languages; prefer adding a new field.

#Avro: defaults and names

Avro bytes contain no names and no numbers, just values in the order of the writer schema. To read, the reader needs the exact writer schema (from the file header of an Avro container file, or from a schema registry), then resolves it against its own reader schema, field by field, by name:

• A field in the writer but not in the reader: skipped.
• A field in the reader but not in the writer: the reader’s default is used; with no default, reading fails.
• A field in both: the types must match or be a legal promotion: int → long, float, double; long → float, double; float → double; string ↔ bytes.

So the Avro rules are:

1. Every added field needs a default (for backward compatibility: new readers on old data).
2. Only remove fields that had a default (for forward compatibility: old readers fill in the default when the field disappears).
3. Rename with aliases in the reader schema. An alias makes a rename backward compatible only: old readers still look for the old name. Upgrade every reader before any writer.
4. Optional means a union with null, and the default must match the first type in the union, so put "null" first: {"name": "note", "type": ["null", "string"], "default": null}.
5. Enums: adding a symbol breaks old readers unless their enum declares a default symbol (supported since Avro 1.9).

The notes’ example schema, with one correction:

{
  "type": "record",
  "name": "OrderCreated",
  "fields": [
    { "name": "orderId",  "type": "string" },
    { "name": "amount",   "type": { "type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2 } },
    { "name": "currency", "type": "string", "default": "BDT" }
  ]
}

The default on currency is what lets a new reader read old events that have no currency. The notes typed amount as a double: never store money in binary floating point (0.1 + 0.2 is not 0.3). Use a decimal logical type or an integer count of the smallest unit (paisa, cents).

#JSON: tolerant readers and weak schemas

JSON has no schema on the wire, so the contract is the field names and the types of their values, and compatibility depends entirely on how readers are written. The rule is the tolerant reader: read only the fields you need, ignore everything else, and treat a missing optional field as its default.

Common JSON traps: a rename (amount → total) looks like a removal to every reader; a number that becomes a string ("amount": "FIFTEEN HUNDRED" in the notes) breaks typed readers; null, a missing key and an empty string mean different things to different languages; and 64-bit IDs lose precision in JavaScript numbers (lesson 2.9), which is why many APIs send IDs as strings from day one.

JSON Schema describes and validates JSON, but it was built for validation, not evolution, so registry compatibility checks for it are weaker and easy to misread. The key switch is additionalProperties:

• Closed ("additionalProperties": false): any new field fails validation in old readers, so every addition is forward incompatible.
• Open (the default): additions pass, but a rename also “passes”, because the old name simply becomes an allowed extra property. The checker cannot see that a required meaning disappeared.

Many teams keep JSON for public APIs (readability, universal support) and use Avro or Protobuf with a registry for internal events, where the stronger rules pay off.

#Thrift

Thrift identifies fields by numeric IDs, like Protobuf, so the same rules apply: never change or reuse an ID, add new fields as optional with new IDs, and leave a comment listing retired IDs so nobody reuses them. Old readers skip unknown fields (and usually do not keep them). Avoid required: the Thrift community’s rule is “required is forever”, because removing it later breaks every old reader.

#Change-by-change cheat sheet

#A schema registry, at overview level

When dozens of services share events, someone has to hold the schemas and enforce the rules. A schema registry is a central service that stores every version of every schema, assigns each one an ID, and checks each new version against the subject’s compatibility mode. It matters most in the Kafka ecosystem. Here is the shape of it; Chapter 32 covers the registry in full.

The notes’ motivating example: the Order service sends {"orderId": "ORD-101", "amount": 1500} and the Inventory service consumes it. Adding "currency": "BDT" may or may not hurt Inventory, depending on how it reads; changing to {"orderId": 101, "amount": "FIFTEEN HUNDRED"} changes both types and will break old consumers. A registry turns “may or may not” into a check that runs before the change ships.

How it works (Confluent Schema Registry, the most common one):

1. Register in CI. The pipeline submits the new schema for a subject (by default named after the topic, such as orders-value). The registry checks it against the subject’s mode (BACKWARD by default) and either assigns an ID or refuses it with 409 Conflict.
2. Produce. The producer’s serializer looks up the schema’s ID once and caches it. Each record is written as magic byte 0 + the 4-byte schema ID + the encoded body (Protobuf records also carry message indexes after the ID). Five bytes instead of the whole schema in every message.
3. Consume. The consumer’s deserializer reads the ID, fetches that schema once (GET /schemas/ids/{id}) and caches it, then decodes with the writer schema and resolves into its own reader schema.

Data never flows through the registry. It is a lookup service on the side, used when a client meets a schema it has not cached. The notes’ diagram draws the registry inline between producer and topic; that is the corrected picture.

What the registry gives you: a single contract between producers and consumers, fewer type mismatches, a version history, safe deploys and breaking changes caught early. What it costs: one more tier-0 service (new pods cannot start while it is down), IDs that only mean something inside one registry (copying topics between clusters means copying schemas too), and discipline: turn off auto-registration in production (auto.register.schemas=false) so only the reviewed pipeline can add schemas.

Other registries follow the same idea with their own details: Apicurio Registry (compatible with Confluent’s API), AWS Glue Schema Registry (modes such as BACKWARD_ALL, and its own header containing a schema version UUID instead of the 5-byte prefix), and the Buf Schema Registry for Protobuf modules.

#Event versioning and upcasting

Event sourcing stores every change as an immutable event and rebuilds state by replaying them (Chapter 32). That makes evolution stricter than anywhere else: the events are the source of truth, they are never edited, and a replay reads every version ever written, years later. The usual tools:

• Compatible evolution first. Add fields with defaults; never change the meaning of a field. Most changes should need nothing more.
• Version in the event: an explicit schemaVersion (or OrderPlaced.v2 as the type) so the reader knows what it holds.
• Upcasting: when the shape must change, write a small pure function that turns a v1 event into a v2 event while reading. Handlers see only the latest version; the stored event is untouched. Upcasters chain (v1 → v2 → v3).
• Copy and transform: rarely, write a new stream with every event converted, and switch over. Expensive, but it lets you drop old upcasters.
• Test the replay: keep a sample of every historical event version and replay it in CI. A missing upcaster only shows up on the day you rebuild a projection or recover from a disaster.

#API versioning: the same rules at the edge

A REST or gRPC API is a schema whose readers are other people’s code, so the same logic applies (Chapter 26 covers API design and versioning in depth):

• Additive changes need no new version: new optional request fields, new response fields, new endpoints. This only works if clients are tolerant readers, which is worth saying in your API documentation.
• Breaking changes get a new version: a path (/v2/orders), a header or a media type. Some providers version by date and translate old requests into the new shape on the server, which is upcasting for an API.
• Retire old versions loudly: announce a date, send the Deprecation header (RFC 9745) and the Sunset header (RFC 8594), measure who still calls the old version, and only then remove it.
• Mobile clients are the long tail: an app version installed today may call your API for years, so public response formats deserve FULL-style care.

#Database schemas: the same idea, expand and contract

A database table is a schema too, and a rolling deploy means old and new application code query it at the same time. The safe pattern is expand → migrate → contract, which you already saw in the rename scenario:

1. Expand: add the new column (nullable or with a default). Old code ignores it.
2. Migrate: deploy code that writes both columns, backfill old rows in batches, then switch reads to the new column.
3. Contract: once no running code uses the old column, drop it.

Each step is backward and forward compatible with the code running next to it. Chapter 46 covers zero-downtime migrations in detail.

#Catching breaking changes before production

Rules nobody checks are suggestions. Put these in CI:

• Schema compatibility checks: ask the registry before merging (POST /compatibility/subjects/{subject}/versions/latest), or use the build plugins that do the same.
• Breaking-change linters: buf breaking compares .proto files against the main branch and fails on a reused number, a removed field without reserved, a type change and more (rule sets for wire-only, wire plus JSON, or full source compatibility).
• Golden samples: keep encoded examples of every old version and assert the new code still reads them.
• Consumer-driven contract tests (Pact is the best-known tool): each consumer publishes what it actually uses, the provider’s build verifies its real messages against every consumer’s contract, and a can-i-deploy check blocks a release that would break a consumer running in production. A schema check knows the types; a contract test knows what each consumer depends on.

#Schema evolution in system design

Choosing a mode:

Failure signatures to recognise:

Interview angles: “How do you rename a field in an event used by twelve teams?” (expand–migrate–contract, never in one step). “What is the difference between backward and forward compatibility, and which deploys first?” “Why does Protobuf have reserved?” “How do you evolve events in an event-sourced system?” (compatible changes, versions, upcasters). “Why is a schema ID in every Kafka record, and what if the registry is down?” Good answers always name who reads, who writes, and in which order they deploy.

#Try it yourself

Check a Protobuf change against the main branch with Buf (it fails on a reused number or a removed field that was not reserved):

buf breaking --against '.git#branch=main'
# output similar to:
# order.proto:12:3:Field "3" on message "OrderPlaced" changed name from "coupon_code" to "shipping_note".

Ask a schema registry whether a schema would be accepted, without registering it (Confluent API, local test registry):

curl -s -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"OrderCreated\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"string\"}]}"}' \
  'http://localhost:8081/compatibility/subjects/orders-value/versions/latest?verbose=true'
# {"is_compatible":false,"messages":["... the reason, naming the field amount and both types ..."]}

curl -s http://localhost:8081/config/orders-value                     # the subject's mode
curl -s -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"compatibility": "FULL_TRANSITIVE"}' http://localhost:8081/config/orders-value

Watch Avro schema resolution in Python with fastavro: write with v1, read with v2.

import io
from fastavro import parse_schema, schemaless_writer, schemaless_reader

v1 = parse_schema({"type": "record", "name": "Order", "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "amount", "type": "long"}]})
v2 = parse_schema({"type": "record", "name": "Order", "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "amount", "type": "long"},
    {"name": "currency", "type": "string", "default": "BDT"}]})

buf = io.BytesIO()
schemaless_writer(buf, v1, {"orderId": "o-101", "amount": 150000})
buf.seek(0)
print(schemaless_reader(buf, v1, v2))
# {'orderId': 'o-101', 'amount': 150000, 'currency': 'BDT'}   default filled in

Remove "default": "BDT" from v2 and run it again: the read fails, because the new reader has nothing to put in currency. That is the backward check a registry would have run for you.

#Summary

Key takeaways

• Schema evolution = changing a data format while old and new code and old and new data coexist: rolling deploys, independent teams, rollbacks, stored data and old clients make that the normal case, not the exception.
• Backward: new reader reads old data (deploy consumers first). Forward: old reader reads new data (deploy producers first). Full: both (any order). Transitive: against every earlier version, needed for anything replayed or stored.
• Additive changes with defaults are the safe default. A rename is a remove plus an add; do it with expand → migrate → contract. Type changes are breaks; add a new field instead.
• Protobuf: never change or reuse a field number, reserved deleted numbers and names, adding fields is safe, unknown fields are kept, few type changes are safe, handle unknown enum values, no required.
• Avro: fields resolved by name; added fields need defaults, removed fields should have had them; renames via aliases (backward only); optional = union with null first; only listed type promotions.
• JSON: be a tolerant reader; watch strict deserializers and closed JSON Schemas; never store money as a float.
• A schema registry checks every new version in CI and puts magic byte + 4-byte schema ID in each record; data never passes through it. Event stores add versions and upcasters; APIs add versions and deprecation headers; databases use expand and contract; contract tests catch what types alone miss.

Self-check: schema evolution and compatibility32 questions