Schema Registry compatibility modes as a design decision

What it is

A Schema Registry is a service that stores versioned schemas and assigns each one an ID. Producers register a schema, get an ID, and prefix each message with it; consumers read the ID, fetch the schema, and deserialise. The message on the wire carries five bytes of framing (a magic byte plus a four-byte schema ID) instead of the schema itself, which is the mechanical benefit: with Avro in particular, the schema is larger than most messages, so shipping it per message would be absurd.

The interesting part is not storage, it is enforcement. When a producer tries to register a new version of a schema, the registry checks it against previous versions under a configured compatibility mode and rejects it if it would break somebody. That check is a deploy-time gate on a data contract, and choosing the mode is choosing who is allowed to deploy first.

That last sentence is the whole page. The compatibility modes look like a serialisation detail and they are actually an organisational policy about upgrade ordering, encoded in a config value that is usually set once by whoever created the cluster and never revisited.

The confusion to clear up: compatibility is not validation. The registry does not check that your data matches the schema at runtime (the serialiser does that locally). It checks that a schema version relates correctly to other schema versions. A registry with the strictest mode set will happily accept garbage data that conforms to a schema; what it prevents is a schema change that would make old consumers fail on new data or new consumers fail on old data.

The problem it solves

Kafka topics are long-lived and multi-consumer. A topic written by one team is read by six others, on their own deploy schedules, with a retention window that means messages written under last month's schema are still being read today. Three failures follow:

The breaking deploy. A producer team adds a required field. Every consumer using the old schema now fails to deserialise, in production, immediately, and the producer team finds out through someone else's pager. Without a registry there is no mechanism that could have caught this; the wire format is just bytes.

The replay failure. A consumer team updates its schema, tests against live traffic, and ships. Two weeks later they need to reprocess from the start of retention and every message older than the change fails, because the change was fine going forward and not backward.

Schema archaeology. Without a registry, the authoritative definition of what is on a topic is "whatever the producer's code does," and answering "what fields does this topic have" means reading someone else's repository. The registry makes the contract a queryable artifact, which matters more than it sounds: it is the difference between a topic being a product and a topic being a side effect.

Mechanics

The modes

Compatibility is checked between a candidate new schema and one or more existing versions. Read the names as answering "can X read data written with Y."

The upgrade-order column is the part that matters and the part that is least obvious, so here is the derivation for BACKWARD:

BACKWARD means a consumer using schema v2 can read data written with v1. It says nothing about a v1 consumer reading v2 data. So if a producer deploys v2 first, the consumers still on v1 will encounter v2 data with no guarantee at all. The consumers must go first. Once every consumer is on v2, the producer can switch, and any v1 data still in the retention window remains readable.

And for FORWARD, it inverts: a v1 consumer can read v2 data, so the producer can deploy first and consumers catch up on their own schedule.

That is why BACKWARD is the sensible default for most event topics: consumers usually outnumber producers, and a policy that lets consumers upgrade independently, ahead of the producer, without coordination, is the one that scales across teams. It also matches the reality that you replay old data more often than you replay new data with old code.

Why "optional" means "has a default"

In Avro, a field is safely removable or addable only if it has a default value, because that is what lets a reader fill in the gap.

{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "orderId",  "type": "string"},
    {"name": "amount",   "type": "double"},
    {"name": "currency", "type": "string", "default": "USD"}
  ]
}

If a v2 reader encounters v1 data with no currency field, it uses "USD". That is the entire mechanism of backward compatibility, and it means a field without a default can never be added under BACKWARD, which is the rejection developers hit first and find surprising.

For a genuinely nullable field the idiom is a union with null first:

{"name": "couponCode", "type": ["null", "string"], "default": null}

The order matters: Avro requires the default to match the first branch of the union, so ["string", "null"] with "default": null is invalid.

Protobuf handles this differently and more forgivingly: every field is optional in proto3 with an implicit zero default, and unknown fields are preserved on round-trip. The consequence is that most protobuf changes are automatically both backward and forward compatible, and the dangerous operations are narrower: reusing a field number, changing a field's type, or changing a field's name in a way that matters to JSON mapping. reserved exists precisely to stop the first one:

message Order {
  reserved 4, 7 to 9;              // numbers of deleted fields: never reuse
  reserved "legacyDiscountCode";   // and the names

  string order_id = 1;
  double amount   = 2;
  string currency = 3;
}

Subject naming: the decision that shapes everything

A subject is the scope within which versions are compared. The strategy decides what the registry considers "the same evolving thing."

The default forces one record type per topic, which is usually right and is occasionally the wrong constraint. If you need several event types on one topic to preserve ordering between them (an OrderCreated, OrderShipped and OrderCancelled stream where relative order matters), TopicRecordNameStrategy is the correct choice, and reaching for it deliberately is a better answer than the common workaround of an Avro union of all event types in one schema, which makes every event type's evolution entangled with every other's.

Where the check happens

Properties props = new Properties();
props.put("value.serializer", KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://schema-registry:8081");

// Fail at startup if the schema is not already registered, rather than
// auto-registering from whatever the producer happens to have compiled in.
props.put("auto.register.schemas", false);
props.put("use.latest.version", true);

auto.register.schemas=false in production is the single most valuable setting here. The default (true) means any producer can register a new schema version simply by starting up, which turns the compatibility gate into something enforced at runtime by whichever service deployed first. With it false, schema registration becomes a deliberate step in CI, reviewable like any other contract change, and a producer with an unregistered schema fails fast at startup rather than at the first message.

The Gradle and Maven Avro plugins support a testSchemaCompatibility goal that checks a candidate schema against the registry, which is where the check belongs: in the pull request, not in the deploy.

A worked example: the required field that took down six consumers

A payments platform. Topic payment-events, one producer team, six consumer teams. Compatibility mode BACKWARD (the default), auto.register.schemas left at true.

The producer team added a fraud score:

{"name": "fraudScore", "type": "double"}

No default, so under BACKWARD this should have been rejected: a v2 reader cannot read v1 data because there is no value to supply for fraudScore. The registry did reject it in their staging environment.

What happened instead. The developer, blocked, changed the topic-level compatibility to NONE to unblock a demo, intending to revert:

curl -X PUT http://schema-registry:8081/config/payment-events-value \
  -H "Content-Type: application/json" -d '{"compatibility": "NONE"}'

The revert did not happen. Two weeks later a different change, the removal of a deprecated merchantCategory field, registered cleanly under NONE and deployed to production.

09:14  producer deploys v3 (merchantCategory removed)
09:14  4 of 6 consumers begin throwing:
       org.apache.avro.AvroTypeException: Found Order, expecting Order,
       missing required field merchantCategory
09:16  consumer groups start failing their poll loops, lag climbing
09:21  first page (from a downstream staleness alert, not from the consumers)
09:40  producer rolled back; consumers recovered from the retained messages

Two of the six consumers were unaffected, because they had merchantCategory with a default in their reader schema. The other four had generated their reader classes from the producer's schema directly, with no default, which is the common pattern and is exactly what makes the registry's guarantee necessary.

The three fixes, in the order they were worth:

1. compatibility moved to FULL_TRANSITIVE and locked at the registry level with topic-level overrides requiring an approved config change. The reasoning for transitive rather than plain: they replay from the start of a 30-day retention window during reconciliation, so compatibility against the immediately previous version is not enough. Plain FULL would let v1 and v3 be mutually incompatible as long as each was compatible with its neighbour, and a replay reads all of them.

2. auto.register.schemas=false everywhere, with registration moved into CI. A schema change became a pull request with a compatibility check as a build step, which is where a data contract change belongs.

3. A deprecation protocol, because the actual goal (removing a field) is legitimate and FULL_TRANSITIVE never permits removing a required field. The protocol is the two-phase expand-and-contract pattern:

Phase 1 (release N):    add a default to merchantCategory. Compatible.
                        Announce deprecation; consumers stop reading it.
Phase 2 (release N+k):  verify no consumer reads it (registry does not know this,
                        so: audit consumer schemas via the registry API, and
                        confirm with a log-based check).
Phase 3 (release N+k+1): remove the field. Now compatible, because it has a default
                        and no reader depends on it.

That is the same expand-and-contract shape as an online schema change in a relational database, and pointing out the parallel is worth doing because teams that have internalised it for databases often have not connected it to event schemas.

The measurable outcome over the following six months: 31 schema changes across the topic, zero consumer-breaking incidents, and 4 schema changes rejected in CI that would previously have reached production. The 4 rejections are the number that justifies the policy.

Production evidence

Confluent Schema Registry is the reference implementation, and its own storage is a compacted Kafka topic _schemas with a single partition (see log compaction), rebuilt into an in-memory index at startup. The single partition is deliberate: schema registration must be totally ordered.

Apicurio Registry (Red Hat) is the main open alternative, supporting Avro, Protobuf, JSON Schema, OpenAPI and AsyncAPI in one registry, with the same compatibility model. Its existence matters practically because Confluent's registry is under the Confluent Community License, which is not OSI-approved, and that has driven a number of teams to Apicurio.

AWS Glue Schema Registry integrates with MSK and Kinesis and implements the same compatibility modes, which is a useful signal that the model has been settled on across vendors rather than being one company's design.

Protobuf's own compatibility rules are documented by Google as part of the language, and buf breaking has become the standard CI enforcement tool for them, independent of any Kafka registry. Teams using protobuf on Kafka often run buf for the schema check and use the registry purely for ID resolution, which is a reasonable split.

LinkedIn's original Avro-on-Kafka work predates Confluent's registry and their writing on it is where the "consumers upgrade first" convention comes from: with hundreds of consumer applications and a small number of producing systems, any policy requiring producers to coordinate with consumers before deploying was unworkable.

The debate

Which mode should be the default? Confluent ships BACKWARD. My position is that FULL_TRANSITIVE should be the default for any topic with more than one consuming team, and I would accept the friction. The argument: BACKWARD alone permits adding a field that old consumers cannot handle in a forward direction, which is fine in theory (they ignore unknown fields in Avro) and breaks in practice when a consumer's deserialiser is strict or when a downstream system copies the record. Transitive matters whenever you replay beyond one version, which any topic with a 7-day-plus retention will eventually do.

The counter-argument, which is real: FULL_TRANSITIVE forbids ever removing a required field or renaming anything, so schemas accumulate deprecated fields permanently. Teams end up with an event with 60 fields, 20 of which nothing reads. That is a genuine cost, and the honest answer is that the deprecation protocol above is the only way out and it takes two releases and an audit. Choosing FULL_TRANSITIVE means committing to that discipline.

Avro or Protobuf? Both work with the registry. Protobuf's compatibility model is more forgiving (unknown field preservation, implicit defaults), its generated code is more pleasant in most languages, and buf gives you excellent CI tooling. Avro's advantages are a much more compact binary encoding when the schema is known (no field tags on the wire at all), native support for schema resolution between distinct reader and writer schemas, and deep integration with the Hadoop and Spark ecosystem. For a new system I would choose Protobuf unless the data lands in a data lake read by Spark, where Avro's ecosystem integration is still meaningfully better. JSON Schema is the third option and I would avoid it for high-volume topics: the encoding is large and the compatibility semantics are the least well-specified of the three.

Should the registry be in the request path? Producers and consumers cache schema IDs aggressively, so a registry outage does not immediately break traffic, but a new schema ID appearing during an outage will fail a consumer that has not cached it. Treat the registry as a tier-1 dependency: replicate it, monitor it, and ensure consumers fail loudly rather than dropping messages when a lookup fails. The failure mode to avoid is a consumer configured to skip records it cannot deserialise, which turns a registry outage into silent data loss.

Is NONE ever defensible? For a topic with exactly one producer and one consumer, both in the same deployable unit, shipped together, yes, and the registry is then only doing ID resolution. The moment a second consumer appears, that assumption is dead and usually nobody notices. My rule: NONE requires a named owner and a comment explaining why, and topic-level overrides should be a reviewed config change rather than a curl command anyone can run, which is the specific failure in the worked example.

Follow-up Q&A

"You need to add a required field. What do you actually do?"

You cannot, under any mode stricter than NONE, and that is correct rather than inconvenient. The path is expand-and-contract: add the field with a default, so it is compatible; deploy producers so real values start flowing; wait out the retention window so every message in the topic has a real value; and if you need the field to be truly required, enforce that in application-level validation rather than in the schema. The schema's job is to make old and new data mutually readable, and a required field is by definition incompatible with data written before it existed. If the field is genuinely mandatory from a business standpoint, that is a new event type, not a new version of the old one.

"Which compatibility mode, and who deploys first?"

BACKWARD: consumers first, because a new consumer can read old data but an old consumer has no guarantee about new data. FORWARD: producers first, because old consumers can read new data. FULL: either order, which is why it costs the most in what it forbids. I would state the mode and the deploy order together in the topic's documentation, because the mode is meaningless to a team that has not connected it to their release process.

"Why transitive?"

Non-transitive checks only against the immediately previous version, so v1 and v3 can be mutually incompatible while v1-to-v2 and v2-to-v3 each pass. That is fine if you only ever process the newest data and never replay, and it breaks the first time you reprocess from the start of retention or bootstrap a new consumer from offset 0. Since bootstrapping a new consumer from the beginning is a completely normal operation, non-transitive modes are a trap on any topic with meaningful retention.

"What does auto.register.schemas=false buy you?"

It moves schema registration from "a side effect of a producer starting up" to "a deliberate step in CI." With the default true, the first producer to deploy registers whatever it was compiled against, so your data contract is decided by deploy ordering. With it false, a producer whose schema is not registered fails at startup, and registration happens as a reviewed step with the compatibility check running in the pull request. It converts a runtime surprise into a build failure.

"How does this interact with Kafka Streams and ksqlDB?"

Both are consumers and producers simultaneously, so both ends of the compatibility question apply to the same application, and internal changelog and repartition topics get their own subjects. A Streams topology whose value type changes requires resetting the application or handling both versions, because the changelog topic holds state written under the old schema and it is compacted, so old records persist indefinitely rather than ageing out. That combination, a compacted topic plus a schema change, is the sharpest edge in this area: with retention you can wait out old data, and with compaction you cannot.

"A consumer is failing to deserialise. Walk me through it."

Read the five-byte header: magic byte then a four-byte schema ID. Fetch that ID from the registry and compare it to what the consumer expects. Three usual causes. The schema ID is not in the registry the consumer is pointed at, which happens with separate registries per environment and a message copied between them. The consumer's reader schema lacks a default for a field the writer omitted. Or the message was not written by a registry-aware serialiser at all, so there is no magic byte and the first byte is data, which produces a confusing error about an unknown schema ID in the millions. That third one is common when a test harness or an old service publishes raw JSON to a topic everyone assumes is Avro.

Common misconceptions

"Backward compatible means old consumers can read new data." It means the opposite: a new consumer can read old data. The naming trips up nearly everyone, and the reliable way to keep it straight is to read the mode as describing the new schema's ability to reach backward in time.

"The registry validates my data." It validates schema versions against each other. Data validation is done locally by the serialiser against the schema you gave it. A registry with FULL_TRANSITIVE will not stop you publishing a negative price.

"Adding a field is always safe." Under BACKWARD it is safe only with a default. Under FORWARD adding a field is safe and removing one is not. Under FULL both operations require the field to be optional. There is no universally safe change other than adding an optional field with a default.

"Compatibility mode is a serialisation setting." It is a deploy-ordering policy. BACKWARD says consumers deploy first; FORWARD says producers do. Choosing it without connecting it to release process is how teams end up with a mode that forbids the changes they actually need to make.

"We use protobuf so compatibility is automatic." Protobuf is more forgiving, not free. Reusing a field number after deleting a field silently reinterprets old bytes as the new field, which is a data corruption bug rather than a deserialisation error. reserved and a CI check with buf breaking are still required.

Interview delivery note

Say this verbatim: "Compatibility mode is not a serialisation setting, it is a policy about who is allowed to deploy first. BACKWARD means consumers upgrade before producers; FORWARD means the reverse. I default to FULL_TRANSITIVE on any topic with more than one consuming team, and I accept that it means we can never remove a required field without a two-release deprecation." That reframing is the whole insight, and the second sentence proves you know what you are paying for it.

The senior-versus-staff separator is transitive versus non-transitive. A senior engineer explains backward and forward correctly. A staff engineer points out that non-transitive only checks the immediately previous version, so bootstrapping a new consumer from offset 0 can hit a version pair that was never checked against each other, and that since bootstrapping from zero is a normal operation, non-transitive is a trap on any topic with real retention.

The second signal is auto.register.schemas=false. Naming it unprompted says you have seen a data contract decided by deploy order, which is the failure this whole mechanism exists to prevent and which the default setting quietly permits.

Further reading

• Confluent documentation, "Schema Evolution and Compatibility," for the full mode matrix and the allowed-changes table per format.
• Apache Avro specification, "Schema Resolution," for the precise rules on defaults, unions and reader/writer schema matching.
• Protocol Buffers documentation, "Updating a Message Type," and the buf breaking rules, for protobuf's compatibility model and CI enforcement.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 4, for schema evolution as a general problem across Avro, Protobuf and Thrift.