Schema evolution

A topic outlives the code that writes to it. Producers deploy, consumers deploy on their own schedule, and the records already in the topic keep the shape they had when they were written. Every field you add, rename or remove is a change that some consumer will meet in the wrong order, and JSON's flexibility is exactly what hides it until production. This lesson is why a schema registry exists, what the compatibility modes promise, how to evolve a schema without breaking anyone, and the consumer that broke on a renamed field.

#Why a registry

A record's bytes carry no description of their own layout; producer and consumer agree on it out of band. With plain JSON the agreement is a class in each codebase and the hope that they match. A schema registry (Confluent's, or Apicurio or Karapace as open alternatives) makes the agreement explicit: every schema version is registered under a subject (by default <topic>-value), gets an id, and the serializer writes that id into the first bytes of every record. The consumer reads the id, fetches the schema once, caches it, and decodes exactly the layout the producer used, whether that was version 3 or version 11.

Two things follow. Records are compact, because the schema is not repeated in every message, only its id. And the registry can refuse a schema that would break existing consumers, at registration time, before a single record is written in the new shape. That refusal is the whole point.

#Compatibility modes

The registry checks a new version against the previous one(s) under the subject's compatibility mode:

Which one you need is about deploy order. BACKWARD fits "upgrade consumers first, then producers": the new consumer must handle old records still in the topic. FORWARD fits "upgrade producers first". Most teams cannot promise an order across services, and FULL_TRANSITIVE is the mode that lets producers and consumers deploy in any order at any time; it constrains every change to "add or remove optional fields", which is the constraint you want anyway.

#Evolving safely

Under FULL, the rules are short:

• Add a field with a default. Old consumers ignore it; new consumers reading old records get the default.
• Remove a field that has a default. New consumers never see it; old consumers reading new records fill in the default.
• Never rename. A rename is a delete plus an add, and the two are not linked: an old consumer reading a new record finds its field missing and gets the default, which is a value that says nothing, silently. Add the new field, dual-write both for as long as any consumer reads the old one, then remove the old one in a later version.
• Never change a type. int to long is safe in Avro's promotion rules; string to int is not; long to int loses data. Treat a type change as a new field.
• Never make an optional field required, or add a required one. That is the change the registry rejects, and the reason it rejects it is that old records lack the field.

The consumer that broke on a renamed field is the incident every team has once. A producer team renamed customerId to customer_id "for consistency". Under JSON with no registry, the consumer's Jackson mapping found no customerId, set it to null, and the service wrote three hours of ledger entries with a null customer before an alert on a downstream report fired. Under Avro with FULL compatibility, the registration would have failed with Incompatible schema: field customerId missing default, and the change would have been redone as add-then-remove. The registry turns a production incident into a build failure.

#Avro, Protobuf and JSON Schema

All three work with the registry; they differ in what they optimise for.

• Avro is the Kafka native. Compact binary, the schema in JSON, defaults required for optional fields (which is what makes evolution checkable), generated Java classes from .avsc files or GenericRecord at runtime. Its weakness is that the reader needs the writer's exact schema, which is what the registry provides; without one, Avro is awkward.
• Protobuf identifies fields by number, never by name, so renaming is genuinely free and unknown fields are skipped by design. Generated classes across every language, and the same definitions serve gRPC. Its evolution rules are simpler: never reuse a field number. The natural choice when the same types cross service boundaries over both Kafka and RPC.
• JSON Schema keeps the payload human-readable and validates it; it is the least compact and its compatibility checking is the weakest of the three, because JSON Schema was not designed with evolution in mind. Right for topics that people read with kafkacat and for integrations with partners who will not run a binary format.

Pick one per organisation, not per topic. Mixed formats mean mixed tooling, and the registry's value comes from everyone using it.

#Spring Kafka configuration

spring.kafka:
  producer:
    key-serializer: org.apache.kafka.common.serialization.StringSerializer
    value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
  consumer:
    value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
  properties:
    schema.registry.url: https://registry.internal
    specific.avro.reader: true          # generated classes, not GenericRecord
    auto.register.schemas: false        # production: schemas are registered by CI, not by the first producer to start
    use.latest.version: false           # serialise with the schema the class was generated from

auto.register.schemas=false is the production setting. With it on, a producer with a locally edited schema registers it on first send, which means a developer's laptop can change the contract for the whole organisation. Schemas go through the registry from a build step, with the compatibility check as the gate, and the producer fails fast if its schema is not there.

Wrap the deserializer in ErrorHandlingDeserializer as in the Retries lesson: a record written with a schema the consumer cannot resolve is a poison pill, and the registry being unreachable at startup is an outage of every consumer at once, so the registry gets the same availability attention as the brokers.

#Under the hood: the wire format, the resolution rules, and the check itself

A registry-serialised record starts with a magic byte (0) and a four-byte big-endian schema id, followed by the payload. The serializer caches schema → id after the first registration or lookup, so a running producer makes no registry calls; the deserializer caches id → schema, fetches on a miss with GET /schemas/ids/{id}, and fails the record if the registry is unreachable and the id is not cached, which is why a fresh consumer pod during a registry outage is an incident while the running ones are fine. Subjects, ids and versions are stored in a compacted Kafka topic, _schemas, and the registry is a stateless service in front of it that elects a leader for writes; its availability is Kafka's, plus the service's own.

Avro decoding is schema resolution between two schemas: the writer's (from the id in the record) and the reader's (the class the consumer was compiled against). Fields are matched by name; a field in the writer's schema absent from the reader's is skipped; a field in the reader's absent from the writer's takes the reader's default, and if it has no default resolution fails with AvroTypeException: Found X, expecting Y, missing required field. Types may be promoted (int → long → float → double, string ↔ bytes), unions are matched by branch, and enums by symbol with a default symbol since Avro 1.9. That name-matching is why a rename is a delete plus an add, and why the default is not decoration but the thing that makes the old record readable.

The registry's compatibility check is exactly a dry run of that resolution. BACKWARD asks: can a reader on the new schema resolve every record written with the previous one? It walks the new schema's fields; any field not in the old one must have a default. FORWARD runs the same walk with the roles swapped. FULL runs both, TRANSITIVE runs against every registered version rather than only the latest. Protobuf's check is structural instead: field numbers must not be reused with a different type, required fields do not exist (proto3), so most changes pass and the check is mostly about removed-then-reused numbers. JSON Schema's check compares constraints (required, additionalProperties, types), and it is the weakest because JSON Schema can express things (oneOf, conditional schemas) whose compatibility is undecidable in general, so the registry conservatively accepts or rejects patterns rather than proving them.

#Walkthrough: the enum that broke the consumers that had not changed

A payments producer added a new payment method to an Avro enum and deployed.

{"type": "enum", "name": "PaymentMethod",
 "symbols": ["CARD", "UPI", "NETBANKING",
             "WALLET"]}

1. The subject was BACKWARD, the default. Adding an enum symbol passed the check: a new reader (with WALLET) can read every old record. The producer deployed and began writing WALLET payments.
2. Six consuming services had not redeployed; their readers had the old enum. Avro resolution of an enum symbol absent from the reader's schema fails unless the reader's enum declares a default symbol; none did. Every WALLET record threw AvroTypeException: No match for WALLET, inside poll(), in six services at once.
3. Five of them had ErrorHandlingDeserializer and dead-lettered every wallet payment for four hours. The sixth did not, and crash-looped on the first one until it was rolled back. The DLTs held 12,000 payments by the time anyone connected the alerts.
4. The change was forward-incompatible: old readers could not read new records. BACKWARD never tested that direction, because it assumes consumers upgrade before producers, and nobody had told the producer team that six consumers existed. Under FULL_TRANSITIVE the registration would have failed: the old schema's enum has no default, so a new symbol is not forward-compatible.
5. Fixes, in order: the subject's mode changed to FULL_TRANSITIVE; the enum gained "default": "UNKNOWN" in every consumer's schema (a reader-side change, deployable at any time); the producer's schema gained the same; and the platform team added a registry check in CI that lists a subject's consumers from the registry's GET /subjects/{s}/versions/{v}/referencedby and the group metadata, so a producer change shows who it reaches.

Compatibility modes encode a deploy order. When nobody controls the order, the mode must cover both directions, and the schema itself must give every reader a way to say "I do not know this value".

#Try it yourself

#Misconceptions

• "The schema travels with the record." Only a five-byte id does; the schema is fetched and cached, and a cold consumer during a registry outage cannot decode.
• "BACKWARD protects consumers." It protects upgraded consumers reading old data. Consumers still on the old schema are protected only by FORWARD or FULL.
• "Adding an enum value is always safe." It is forward-incompatible unless readers declare a default symbol; old consumers fail on the new value.
• "Protobuf makes evolution free." It makes renames free and unknown fields harmless; reusing a field number, or changing a number's type, still breaks readers silently.
• "auto.register.schemas=true is convenient and harmless." It lets any producer instance define the contract for everyone. Register from CI with the compatibility check as the gate.

#Going deeper

• Confluent Schema Registry documentation: "Wire Format", "Schema Evolution and Compatibility" (with the per-mode tables) and the REST API reference.
• Apache Avro specification, "Schema Resolution", the complete matching and promotion rules in two pages.
• Protocol Buffers language guide, "Updating A Message Type", the rules for field numbers.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 4, "Encoding and Evolution".
• Apicurio Registry and Karapace documentation, for the open implementations of the same API.