You've built the mental model of Kafka architecture. Now it's time to run a real cluster.
In production Kubernetes environments, you don't manually configure Kafka brokers. You use an operator—a Kubernetes-native controller that understands Kafka's operational requirements and manages the cluster lifecycle automatically. The industry standard for Kafka on Kubernetes is Strimzi, a CNCF project with widespread adoption.
This lesson walks you through deploying Kafka on Docker Desktop Kubernetes using Strimzi. By the end, you'll have a running Kafka cluster that you can use throughout this chapter.
Before starting, configure Docker Desktop with enough resources for Kafka:
Without this, Kafka pods will crash with OOMKilled or CrashLoopBackOff errors.
Before proceeding, verify your environment is ready:
Output:
Output:
If either command fails, revisit Chapters 79-81 to set up Docker Desktop Kubernetes and Helm.
Before diving into Strimzi, let's clarify two Kubernetes concepts you'll use throughout this lesson.
Kubernetes has built-in resources like Pods and Services. A Custom Resource Definition (CRD) extends Kubernetes with new resource types:
CRDs let you manage Kafka the same way you manage any Kubernetes resource—with kubectl apply -f.
An operator is a Kubernetes-specific pattern: a program that watches your CRDs and takes action to make reality match your desired state.
Think of it as a 24/7 expert encoded in software. A Kafka operator knows how to deploy, scale, upgrade, and heal Kafka clusters automatically. Without an operator, you'd write hundreds of lines of YAML and handle all operations manually.
Strimzi is a CNCF open-source project that provides a Kafka operator for Kubernetes. When you install Strimzi, it:
Instead of writing complex deployment manifests, you describe your Kafka cluster declaratively, and Strimzi handles the operational complexity.
The operator pattern means you declare intent ("I want a 3-broker Kafka cluster") and Strimzi figures out how to achieve and maintain it.
Add the Strimzi Helm repository and install the operator:
Output:
Output:
Output:
Wait for the operator pod to be ready:
Output:
Press Ctrl+C once you see 1/1 Running. The operator is now watching for Kafka CRDs.
Before creating files, understand what we're about to build. Strimzi uses two CRDs that work together:
Why two files? Think of it like building a car:
What happens when you apply each:
The files are linked by name—the NodePool references which Kafka cluster it belongs to:
Now let's create each file.
Kafka in KRaft mode has two node roles: controllers (manage metadata) and brokers (handle messages). For development, we combine both roles in a single node to minimize resource usage.
Create a file named kafka-nodepool.yaml:
Key configuration points:
Apply the node pool:
Output:
Now create the Kafka cluster that uses the node pool. Create a file named kafka-cluster.yaml:
Key configuration points:
Apply the cluster:
Output:
The Strimzi operator will now create the Kafka pods. This takes 1-2 minutes on first deployment:
Output (after ~90 seconds):
Press Ctrl+C once all pods show Running.
Verify the Kafka cluster status:
Output:
The READY: True and METADATA STATE: KRaft confirm your cluster is operational.
With the Entity Operator running, you can manage topics declaratively. Create a file named kafka-topic.yaml:
Key configuration points:
Apply the topic:
Output:
Verify the topic was created:
Output:
The Topic Operator watches for KafkaTopic resources and syncs them to the actual Kafka cluster:
This declarative approach means your topic configuration is version-controlled and reproducible across environments.
Test connectivity by running a temporary pod with the Kafka CLI:
Output:
This confirms:
Strimzi creates a Kubernetes Service for client connections:
Output:
Clients connect to task-events-kafka-bootstrap:9092 (or port 9093 for TLS). This service load-balances across all broker pods, providing a stable endpoint regardless of pod restarts.
The configuration in this lesson is optimized for learning on Docker Desktop. Production requires different settings:
Lesson 18 covers production Strimzi configuration in detail.
If you need to remove the Kafka cluster:
For now, keep the cluster running—you'll use it in the remaining lessons.
You built a kafka-events skill in Lesson 0. Test and improve it based on what you learned.
Ask yourself:
If you found gaps:
You've deployed Kafka using Strimzi's operator pattern. Now explore how this fits into broader infrastructure decisions.
Production Configuration Review
What you're learning: Production Kafka requires careful sizing and redundancy configuration. AI can help you understand the gap between development convenience and production reliability.
Debugging Operator Issues
What you're learning: Operators provide status information through CRD conditions. Understanding how to read operator status helps you debug declarative infrastructure.
Alternative Topic Configurations
What you're learning: Topic configuration directly impacts performance and semantics. The right settings depend on your use case—high-throughput streaming vs change data capture vs event sourcing.
Safety note: When modifying production Kafka configurations, always test changes in a staging environment first. Incorrect replication or partition settings can cause data loss.