Message Schemas: Avro and Schema Registry

Your Task API is publishing events to Kafka. Today, your task.created event looks like this:

Next month, the product team wants to add task priority. You add the field:

But the notification service consuming these events wasn't updated. When it receives a message with the new priority field, it crashes. Or worse, it silently ignores the field and loses data. You've just experienced schema drift---the silent killer of event-driven systems.

In production, you'll have dozens of services reading and writing events. Without schema enforcement, any producer can add, remove, or rename fields at will. Consumers break in unpredictable ways. Debugging becomes a forensic investigation: "Which version of the event was this consumer built for?"

This chapter introduces Apache Avro for binary schema-based serialization and Schema Registry for centralized schema management. By the end, you'll design event schemas that evolve safely, enforce contracts between producers and consumers, and prevent the integration failures that plague untyped messaging systems.

Why Schemas Matter: The Contract Problem

In a typical Kafka deployment without schemas, producers and consumers communicate through implicit contracts:

This works until someone changes something:

The core problem: JSON doesn't enforce structure. Any producer can send anything, and you won't discover the mismatch until runtime---often in production.

What Schemas Provide

Apache Avro Fundamentals

Apache Avro is a data serialization system that provides:

• Schema-based serialization: Data is always encoded with a schema
• Binary encoding: Compact messages without field names in payload
• Schema evolution: Add/remove fields with compatibility rules
• Language-agnostic: Works with Python, Java, Go, and more

Avro Schema Syntax

An Avro schema is JSON that defines your data structure:

Key components:

• type: Always "record" for structured data
• name: The schema name (used in subject naming)
• namespace: Package-like qualifier for uniqueness
• fields: Array of field definitions with names and types

Avro Field Types

Optional Fields with Union Types

To make a field optional, use a union type with null:

The ["null", "int"] union means the field can be either null or an integer. The default: null makes it optional---old messages without priority will deserialize with priority = null.

Complex Schema Example

Here's a complete schema for Task API events:

Schema Registry: Centralized Schema Management

Confluent Schema Registry provides:

• Central schema storage: Single source of truth for all schemas
• Schema versioning: Track all versions of each schema
• Compatibility enforcement: Block incompatible schema changes
• Schema ID in messages: Messages include schema ID, not full schema

How Schema Registry Works

When a producer sends a message:

The message payload starts with 5 bytes of metadata:

Installing Dependencies

Add the required packages to your project:

Or with pip:

Deploying Schema Registry

Important: Strimzi doesn't include Schema Registry. You need to deploy it separately. We'll use Apicurio Registry, which is Confluent Schema Registry-compatible and works well on Kubernetes.

Deploy Apicurio Registry

Create schema-registry.yaml:

Apply the configuration:

Output:

Wait for the pod to be ready:

Connection Reference

For local development, we use the NodePort URLs. For code running inside Kubernetes, use the internal URLs.

Integrating Schema Registry with Python

Setting Up the Schema Registry Client

Output:

Producer with Avro Serialization

Output:

Consumer with Avro Deserialization

Output:

Schema Evolution: Changing Schemas Safely

The real power of Schema Registry is compatibility enforcement. You can evolve schemas over time without breaking consumers.

Compatibility Modes

Backward Compatibility: The Default

With BACKWARD compatibility, consumers using the new schema can read data written with the old schema.

Safe changes (backward compatible):

Unsafe changes (breaks compatibility):

Example: Adding a Field Safely

Original schema (v1):

New schema (v2) - Adding priority:

This is backward compatible because:

1. New consumers can read old messages (priority defaults to null)
2. Old consumers can read new messages (they ignore unknown fields)

Example: Breaking Compatibility

Original schema:

Incompatible change - Adding required field:

When you try to register this schema:

Output (error):

Schema Registry blocks the incompatible change, preventing production breakage.

Checking Compatibility Before Registration

Always check compatibility before deploying schema changes:

Output:

Designing Schemas Through Collaboration

You've learned the mechanics of Avro schemas. Now let's design a real schema for the Task API.

Your starting point:

"I need to design event schemas for my Task API. I want to publish task lifecycle events."

Identifying requirements:

Consider what information each event needs:

• Event metadata: event_id, event_type, occurred_at
• Correlation: correlation_id for tracing across services
• Entity data: task_id, title, owner_id
• Optional data: priority, due_date, tags

Initial design attempt:

Evaluating the design:

This schema has problems:

1. No event metadata (how do you trace events across services?)
2. Priority is required (old producers can't send without it)
3. No versioning strategy (what happens when you add fields?)

Refining based on production requirements:

A better design separates event metadata from entity data:

What emerged from refinement:

• Event metadata enables distributed tracing
• Optional fields with defaults enable safe evolution
• Documentation in schema serves as contract
• Namespace prevents naming collisions

Subject Naming Strategies

Schema Registry organizes schemas by subject. The default naming strategy is:

For topic task-created:

• Key schema subject: task-created-key
• Value schema subject: task-created-value

Alternative Strategies

Configure in producer:

Common Patterns and Anti-Patterns

Pattern: Envelope with Metadata

Wrap all events in a standard envelope:

Anti-Pattern: Overusing Union Types

Don't use unions to represent "any type":

If you need this flexibility, you've lost the schema's contract value.

Anti-Pattern: Deeply Nested Optional Objects

Flatten when possible, or use separate events for different entity states.

---

Reflect on Your Skill

You built a kafka-events skill in Chapter 1. Test and improve it based on what you learned.

Test Your Skill

Identify Gaps

Ask yourself:

• Did my skill explain schema compatibility types (backward, forward, full)?
• Did it show how to use Schema Registry and handle schema evolution?

Improve Your Skill

If you found gaps:

---

Try With AI

Apply schema design and evolution to your Task API events.

Setup: Open Claude Code or your preferred AI assistant in your project directory.

---

Prompt 1: Design an Event Schema

What you're learning: Schema design decisions---which fields are essential to the event's meaning (required) versus context that might not always be available (optional with defaults).

---

Prompt 2: Plan a Schema Evolution

What you're learning: Compatibility analysis---understanding which changes are safe and how to work around the restrictions when you need to add required fields to existing schemas.

---

Prompt 3: Debug a Compatibility Error

What you're learning: Compatibility debugging---understanding that renaming a field is effectively a delete+add operation, and how to handle migrations that require breaking changes (versioning strategies, new topics, dual-writes).

---

Safety Note: Schema changes affect all producers and consumers. Always test compatibility in a staging environment before production, and coordinate deployment order based on your compatibility mode (BACKWARD = upgrade consumers first).