Protobuf wire format and compatibility rules

What it is

Protocol Buffers encode a message as a sequence of key-value pairs, where the key is a varint packing the field number and a 3-bit wire type, and the value's encoding is determined by that wire type.

key = (field_number << 3) | wire_type

Three consequences follow from that one line, and they explain every compatibility rule in the specification:

1. Field names are not on the wire. Only numbers. Renaming a field is free at the binary level (and not free for JSON mapping or generated code).
2. A parser that meets an unknown field number can skip it, because the wire type tells it how many bytes to skip. That is why adding fields is safe.
3. A parser cannot tell whether a field was absent or set to its default, in proto3 without optional. The encoder simply omits default values.

Six wire types exist; three matter in practice:

(Types 3 and 4 were start-group and end-group, deprecated since proto2.)

What this is confused with: protobuf is not self-describing. Given bytes and no .proto, you can recover the structure (field numbers, wire types, nesting) but not the meaning: you cannot tell an int32 from an enum, a string from an embedded message, or which field is user_id. Avro carries a schema reference; protobuf carries nothing. That is why a Schema Registry (see Schema Registry compatibility) or a checked-in .proto is not optional infrastructure.

The problem it solves

Size and speed against JSON. A message with three small integers and a short string:

{"userId": 1042, "score": 87, "active": true, "name": "alice"}

That is 60 bytes of JSON, of which 38 are field names and punctuation. The protobuf encoding of the same data is 15 bytes, because the field names are replaced by single-byte keys and the integers are binary rather than decimal text. Parsing is correspondingly cheaper: no string scanning, no number parsing, no allocation for keys.

Schema evolution without coordinated deploys. This is the larger benefit and the reason protobuf is used where JSON would be fast enough. Because unknown fields are skippable and defaults are implicit, a service compiled against version 3 of a message can read data written by version 7 and vice versa, provided you follow the rules below. That property is what lets a hundred services deploy independently.

Generated code as the contract. The .proto file is the interface definition, and protoc produces types in every language from it. The compiler enforces that you cannot send a field the schema does not have, which is a class of bug JSON APIs handle with runtime validation and hope.

Mechanics

Encoding, byte by byte

message User {
  int32  id     = 1;
  string name   = 2;
  bool   active = 3;
}

Encoding {id: 150, name: "ab", active: true}:

Field 1 (id = 150):
  key   = (1 << 3) | 0  = 0x08          field 1, VARINT
  value = 150           = 0x96 0x01     varint: 150 = 0b10010110
                                        low 7 bits first, continuation bit set
  bytes: 08 96 01

Field 2 (name = "ab"):
  key    = (2 << 3) | 2 = 0x12          field 2, LEN
  length = 2            = 0x02
  data   = "ab"         = 0x61 0x62
  bytes: 12 02 61 62

Field 3 (active = true):
  key   = (3 << 3) | 0  = 0x18          field 3, VARINT
  value = true          = 0x01
  bytes: 18 01

Total: 08 96 01 12 02 61 62 18 01     = 9 bytes

Varints are little-endian base-128. Each byte carries 7 bits of payload and one continuation bit. 150 becomes 0x96 0x01: 0x96 is 1_0010110 (continuation set, payload 0010110 = 22) and 0x01 is 0_0000001 (payload 1), giving 1 << 7 | 22 = 150.

The consequence that matters for schema design: small numbers cost one byte, large ones cost up to ten. A field number under 16 encodes its key in one byte; 16 to 2047 takes two. That is why the specification says to reserve field numbers 1 to 15 for the fields that appear in every message.

The negative-number trap

int32  a = 1;   // -1 encodes as TEN bytes
sint32 b = 2;   // -1 encodes as ONE byte

A negative int32 is sign-extended to 64 bits before varint encoding, so -1 becomes 0xFFFFFFFFFFFFFFFF, which is ten varint bytes. sint32 uses zigzag encoding, which maps signed to unsigned so small magnitudes stay small regardless of sign:

zigzag(n) = (n << 1) ^ (n >> 31)      # for 32-bit

  0 ->  0        -1 ->  1        1 ->  2
 -2 ->  3         2 ->  4       -3 ->  5

Use sint32/sint64 for any field that is commonly negative (deltas, temperatures, coordinates relative to an origin, balance changes). Use int32/int64 when values are almost always non-negative, because zigzag doubles the magnitude and therefore costs an extra byte at each 7-bit boundary.

Packed repeated fields

repeated int32 values = 4;    // proto3: PACKED by default

Packed encoding writes the key once, then a length, then the values back to back:

Unpacked (proto2 default):  20 01 20 02 20 03      6 bytes for [1,2,3]
Packed   (proto3 default):  22 03 01 02 03         5 bytes for [1,2,3]

For 1,000 small integers the saving is roughly 1,000 bytes, since packing eliminates one key byte per element. Packing applies only to repeated scalar numeric types; repeated strings and messages are always length-delimited individually because they already carry their own lengths.

Unknown field preservation

A parser encountering field number 7 with wire type 2, when its schema has no field 7, skips the bytes and, in most implementations, retains them. If that message is re-serialised, field 7 comes back out unchanged.

This matters enormously for proxies and pass-through services:

service A (v7, has field 7) ──▶ service B (v3, no field 7) ──▶ service C (v7)

B parses, modifies field 2, re-serialises, and field 7 survives. Without unknown field preservation, B would silently strip data every time it touched a message, and the loss would appear at C as a field that intermittently vanishes depending on routing.

proto3 dropped unknown field preservation in version 3.0 and restored it in 3.5, after the data-loss problem it caused in practice. That reversal is a useful piece of history: the argument for dropping it was that unknown fields complicate the model, and production experience said the model was worth the complication.

The compatibility rules, and why each one holds

Safe:

Unsafe, in order of how badly it fails:

Reusing a field number is the one that causes data corruption rather than an error, which is why reserved exists:

message User {
  reserved 4, 7 to 9;                    // numbers of deleted fields
  reserved "legacy_email", "old_tier";   // and their names, for JSON and codegen

  int32  id     = 1;
  string name   = 2;
  bool   active = 3;
}

protoc then rejects any attempt to reuse those numbers or names, at compile time. Reserving on deletion is not a best practice, it is the mechanism that prevents a future data-corruption bug, and it costs one line.

proto3's default-value problem, and optional

proto3 originally removed field presence: a scalar field set to its default (0, "", false) is not encoded at all, so the receiver cannot distinguish "not set" from "set to zero."

message UpdateUser {
  int32 id       = 1;
  int32 age      = 2;    // age=0 and "don't change age" are IDENTICAL on the wire
  bool  verified = 3;    // verified=false and "not specified" are IDENTICAL
}

For a partial-update message this is a genuine bug: you cannot express "set verified to false." The workarounds were wrapper types (google.protobuf.Int32Value, a message so its presence is observable) and oneof tricks.

proto3.15 restored optional, which reintroduces explicit presence:

message UpdateUser {
  int32          id       = 1;
  optional int32 age      = 2;   // has_age() now exists
  optional bool  verified = 3;   // has_verified() now exists
}

optional in proto3 is implemented as a synthetic single-field oneof, which is why it is wire-compatible with a plain field: the encoding is identical, only the generated API gains a has_ method. Use optional for any field where "absent" and "default" mean different things, which is every patch or partial-update message.

Enums have a mandatory zero

enum Status {
  STATUS_UNSPECIFIED = 0;      // REQUIRED to be first in proto3
  STATUS_ACTIVE      = 1;
  STATUS_SUSPENDED   = 2;
}

The zero value is the implicit default, so it must mean "unset" rather than a real state. Making STATUS_ACTIVE = 0 means every message that forgot to set status claims to be active.

Enums are open in proto3: a parser receiving value 7 for an enum it does not know keeps it as an unrecognised value rather than erroring, and re-serialises it unchanged. That is what allows adding enum values without breaking old readers, and it means your code must handle an unknown enum value at runtime. A switch over an enum without a default branch is a bug waiting for the next schema version.

A worked example: a field number reused, and 400,000 corrupted records

An analytics platform. An event message, evolved over four years, consumed by eleven services.

message Event {
  string event_id   = 1;
  int64  timestamp  = 2;
  string user_id    = 3;
  int32  session_ms = 4;      // added 2021, deprecated 2023, DELETED 2024
  string page       = 5;
}

In 2024 session_ms was removed, with no reserved. Six months later, a different team added a field and, seeing 4 unused in the current .proto, reused it:

message Event {
  string event_id  = 1;
  int64  timestamp = 2;
  string user_id   = 3;
  int32  retry_count = 4;     // <- REUSED. Same wire type (VARINT).
  string page      = 5;
}

Same wire type, so nothing failed. No parse error, no exception, no alert.

What broke. The platform replayed from a 90-day event archive for backfills. Events written before the 2024 deletion still carried field 4 as session_ms, values in the tens of thousands (milliseconds). The new consumer read those as retry_count.

Replay of archived events (pre-2024 data):
  session_ms values: 8,000 - 180,000
  read as retry_count: 8,000 - 180,000

Downstream alerting rule: retry_count > 5  ->  page the owning team
Result: every replayed historical event triggered an alert.

The immediate symptom was an alert storm. The damaging one was quieter: a reliability dashboard computing "mean retries per event" over a window that mixed archived and live data reported a mean of 12,400, and a quarterly reliability report had already gone out based on it.

records affected:              ~400,000 (archived events with field 4 set)
time to detect:                6 days (found by an engineer questioning the dashboard)
time to root cause:            2 days
detection mechanism:           a human thinking a number looked wrong

Two days to root cause is the part worth dwelling on. Nothing in the logs pointed at protobuf. The consumer parsed successfully, the values were valid int32s, and the .proto in the repository was self-consistent. The only way to find it was to diff the schema's history and notice field 4 had two lives.

The fixes:

message Event {
  reserved 4;                          // NEVER reusable now; protoc enforces it
  reserved "session_ms", "retry_count";

  string event_id    = 1;
  int64  timestamp   = 2;
  string user_id     = 3;
  string page        = 5;
  int32  retry_count = 6;              // a NEW number
}

plus buf breaking in CI against the previous version:

# buf.yaml
version: v2
breaking:
  use:
    - FILE            # the strictest rule set

# In CI, on every PR touching a .proto
buf breaking --against '.git#branch=main'

buf breaking catches field-number reuse, wire-type changes, and removed fields without reserved, at pull-request time. The tool existed and was not wired in, which is the ordinary shape of this kind of incident.

Measured afterwards, over the following year:

breaking changes caught in CI:     7
breaking changes reaching prod:    0
CI time added by buf breaking:     ~4 seconds

Seven catches in a year, at four seconds per build. The corrupted 400,000 records could not be repaired (the original session_ms values were recoverable, but every downstream aggregate computed from them during the six-day window had to be recomputed).

Production evidence

Google uses protobuf for essentially all internal RPC, and the style guide's rules (reserve deleted numbers, never reuse, keep 1 to 15 for hot fields, always define a zero enum meaning unspecified) are derived from operating it at a scale where every mistake happens eventually.

gRPC uses protobuf as its default payload encoding, which is why the two are usually discussed together, though protobuf is transport-independent and gRPC can carry other encodings.

Buf's buf breaking and buf lint have become the standard CI enforcement, and Buf's rule categories (FILE, PACKAGE, WIRE_JSON, WIRE) encode exactly the distinction this page draws: WIRE compatibility is about the binary encoding, while WIRE_JSON additionally protects the JSON mapping, which is where field names start to matter.

Confluent's Schema Registry supports protobuf alongside Avro and JSON Schema, with the same compatibility modes, and its protobuf checker enforces the reserved-number and wire-type rules on registration.

proto3 restoring optional in 3.15 and unknown-field preservation in 3.5 are both reversals of simplifying decisions made in the original proto3 design. Both were driven by production experience: "no field presence" broke partial updates, and "drop unknown fields" caused silent data loss in proxy services. Knowing that these were removed and restored is useful, because a lot of published advice predates the restoration.

The debate

Protobuf or Avro? Both work with a Schema Registry, both handle evolution. Avro's advantages: a genuinely more compact encoding when the schema is known (no field tags on the wire at all, since the reader has the writer's schema), native reader/writer schema resolution, and better integration with Spark and the Hadoop lineage. Protobuf's advantages: better generated code in most languages, a far more forgiving evolution model (unknown field preservation, open enums), and buf as excellent CI tooling.

My position: protobuf for service-to-service RPC, Avro when the data lands in a lake read by Spark. The deciding factor is usually the ecosystem the data flows into, not the encoding's properties.

Protobuf or JSON for a public API? JSON, almost always. Protobuf's benefits (size, parse speed, generated types) accrue to high-volume internal traffic, and its costs (binary payloads you cannot inspect with curl, a build step for consumers, tooling that third parties may not have) fall hardest on external developers. gRPC-Web and Connect narrow the gap, and I would still default to JSON for anything third parties consume, with protobuf internally.

Is optional in proto3 worth using everywhere? No. It costs a has_ check everywhere the field is read and adds nothing when zero is a perfectly good default. Use it where absent and default genuinely differ: patch messages, nullable database columns, "did the client specify this" flags. Using it uniformly is defensive noise; omitting it on a partial-update message is a real bug.

Should you use required? proto2 had it; proto3 removed it and the removal was correct. A required field can never be removed, because old readers reject messages without it, so it is permanent in a way no other schema element is. Google's own style guidance describes required as harmful for exactly this reason. Validate required-ness in application code, where you can change your mind.

Is field-number reuse really that dangerous? It is the one change that causes silent corruption rather than an error, and it only manifests when old data meets new code, which means archives, replays, backfills and long-retention topics. A system that never replays historical data may never notice; a system with a 90-day archive absolutely will. Reserve on every deletion, and enforce it in CI, because the failure surfaces months later with no signal pointing at the schema.

Follow-up Q&A

"Why can you rename a field but not reuse a number?"

Field names are not on the wire; the key encodes only the field number and wire type. So a rename changes generated code and JSON mapping and changes nothing binary. A number is the identity, so reusing one means old bytes are reinterpreted as the new field. If the wire types match, that parses cleanly and produces wrong values with no error, which is why reserved exists and why buf breaking treats reuse as a breaking change.

"How does a parser skip a field it does not know?"

The key's low 3 bits are the wire type, which tells it the length: VARINT means read until a byte without the continuation bit, I64 means 8 bytes, LEN means read a varint length then that many bytes, I32 means 4 bytes. So it can always find the end of an unknown field without knowing what it is, which is the property that makes adding fields safe. Most implementations also retain those bytes so re-serialisation preserves them.

"Why is sint32 different from int32?"

Zigzag encoding. A negative int32 is sign-extended to 64 bits before varint encoding, so -1 takes ten bytes. sint32 maps signed to unsigned by interleaving (0, -1, 1, -2, 2 -> 0, 1, 2, 3, 4), so small magnitudes stay small regardless of sign. Use sint where values are commonly negative, int where they are usually positive, because zigzag doubles the magnitude and costs an extra byte at each 7-bit boundary. And never switch between them on an existing field: both are VARINT so it parses, and every value is silently wrong.

"What is the proto3 default-value problem?"

A scalar set to its default is not encoded, so the receiver cannot distinguish "not set" from "set to zero." For a partial-update message that means you cannot express "set verified to false," because false is indistinguishable from absent. The historical workaround was wrapper message types; since proto3.15 the answer is optional, which is implemented as a synthetic one-field oneof and is wire-compatible with a plain field.

"How do you enforce compatibility?"

buf breaking in CI against the main branch, which catches number reuse, wire-type changes, and deletions without reserved. Plus reserved on every deletion so protoc itself rejects reuse. Plus a Schema Registry if the messages go through Kafka, so registration is gated too. The thing to avoid is relying on code review, because the dangerous change (reusing a number that is absent from the current file) looks completely innocent in a diff.

"Field numbers 1 to 15 versus 16 and above?"

Numbers 1 to 15 encode their key in one byte, 16 to 2047 in two. So the fields present in every message should take the low numbers, and the guidance is to leave some low numbers unassigned for future hot fields. On a message sent billions of times a day, one byte per field per message is real, and it is a decision you cannot revisit later because changing a field number is a breaking change.

Common misconceptions

"Protobuf is self-describing." It is not. You can recover structure from raw bytes (protoc --decode_raw will show you field numbers and wire types) and not meaning. You cannot tell an int32 from an enum, or a string from an embedded message. The schema must travel separately.

"Protobuf handles compatibility automatically." It gives you the mechanisms (skippable unknown fields, implicit defaults, open enums). Reusing a number, changing a wire type, or switching int32 to sint32 all break, and the last two break silently. The rules are simple and they are rules.

"Renaming a field is a breaking change." Not on the wire. It breaks generated code and the JSON mapping, so it is a source-level and JSON-level break, and the binary encoding does not care.

"proto3 has no field presence." True until 3.15, which restored optional. A great deal of published advice about wrapper types predates this and should be ignored for new code.

"An unknown enum value is an error." In proto3, enums are open: an unrecognised value is preserved rather than rejected, which is what allows adding enum values safely. Your code has to handle it, and a switch without a default branch will not.

Interview delivery note

Say this verbatim: "The key is the field number shifted left three bits with the wire type in the low three, which means names are not on the wire and a parser can skip unknown fields by wire type. That single fact explains why renaming is free, adding is safe, and reusing a number silently corrupts data rather than erroring." Deriving the rules from the encoding, rather than listing them, is the difference between having read the documentation and understanding the format.

The senior-versus-staff separator is naming reuse as the silent one. A senior engineer lists the compatibility rules correctly. A staff engineer distinguishes changes that fail loudly (wire-type mismatch) from changes that fail silently (number reuse, int32 to sint32), notes that silent failures only surface when old data meets new code (archives, replays, backfills), and therefore insists on reserved plus buf breaking in CI rather than code review, because the dangerous diff looks innocent.

The second signal is sint32 versus int32. It is a small thing that shows you have read the encoding rather than just the API, and the "negative one costs ten bytes" detail is memorable and checkable.

Further reading

• Protocol Buffers documentation, "Encoding," for the wire format including varints, zigzag and packed repeated fields.
• Protocol Buffers documentation, "Updating A Message Type," for the authoritative compatibility rules.
• Buf documentation on breaking-change rule categories (WIRE, WIRE_JSON, PACKAGE, FILE), for what CI enforcement can and cannot catch.
• The proto3 optional proposal and the 3.15 release notes, for field presence and why it was restored.