USMAN’S INSIGHTS
AI ARCHITECT
  • Home
  • About
  • Thought Leadership
  • Book
Press / Contact
USMAN’S INSIGHTS
AI ARCHITECT
⌘F
HomeBook
HomeBookSwap Redis for Kafka Without Touching a Line of Code
Previous Chapter
The Sidecar Pattern
Next Chapter
Deploy Dapr State Management
AI NOTICE: This is the table of contents for the SPECIFIC CHAPTER only. It is NOT the global sidebar. For all chapters, look at the main navigation.

On this page

32 sections

Progress0%
1 / 32

Muhammad Usman Akbar Entity Profile

Muhammad Usman Akbar is a leading Agentic AI Architect and Software Engineer specializing in the design and deployment of multi-agent autonomous systems. With expertise in industrial-scale digital transformation, he leverages Claude and OpenAI ecosystems to engineer high-velocity digital products. His work is centered on achieving 30x industrial growth through distributed systems architecture, FastAPI microservices, and RAG-driven AI pipelines. Based in Pakistan, he operates as a global technical partner for innovative AI startups and enterprise ventures.

USMAN’S INSIGHTS
AI ARCHITECT

Transforming businesses into autonomous AI ecosystems. Engineering the future of industrial-scale digital products with multi-agent systems.

30X Growth
AI-First
Innovation

Navigation

  • Home
  • Book
  • About
  • Contact
Let's Collaborate

Have a Project in Mind?

Let's build something extraordinary together. Transform your vision into autonomous AI reality.

Start Your Transformation

© 2026 Muhammad Usman Akbar. All rights reserved.

Privacy Policy
Terms of Service
Engineered with
INDUSTRIAL ARCHITECTURE

Building Blocks and Components

Module 7 takes the agent you built in Module 6 and turns it into a production cloud service. You'll containerize the stack, orchestrate it on Kubernetes, automate delivery, and operate it with observability, security, and cost controls. The goal: a reliable Digital FTE that runs 24/7 for real users.

Prerequisites: Modules 4-6. You need a working agent service to deploy.

You learned Kafka directly in earlier chapters. You wrote code that knew about brokers, topics, and consumer groups. If you wanted to switch from Kafka to RabbitMQ, you'd rewrite your messaging code. Different SDK, different connection logic, different error handling.

Now imagine a different world. You write code that says "publish this message" or "save this state." You don't specify where or how. That decision lives in a configuration file. To switch from Kafka to RabbitMQ, you change one YAML file. Your application code doesn't change at all.

This is Dapr's abstraction model: building blocks provide the API (what you can do), and components provide the implementation (how it actually happens). This lesson explores how this separation works and why it matters for portable, maintainable distributed systems.


Building Blocks: The Portable APIs

A building block is a Dapr capability exposed through an HTTP or gRPC API. It defines what you can do, not how it's done underneath.

Think of building blocks like a universal remote control. The remote has buttons for "power," "volume," and "channel." It doesn't care if you're controlling a Samsung TV, a Sony TV, or a projector. Same interface, different devices.

The Seven Building Blocks (This Chapter)

Building BlockAPI EndpointWhat It Does
Service Invocation/v1.0/invoke/{app-id}/method/{method}Call other services with automatic discovery and mTLS
State Management/v1.0/state/{store}Key-value storage with consistency options
Pub/Sub/v1.0/publish/{pubsub}/{topic}Event messaging with CloudEvents format
Bindings/v1.0/bindings/{binding}Input/output triggers for external systems
Jobs/v1.0/jobs/{name}Schedule and manage recurring tasks
Secrets/v1.0/secrets/{store}/{key}Retrieve secrets from configured stores
Configuration/v1.0/configuration/{store}Dynamic configuration with change subscriptions

Building Blocks NOT in This Chapter

Building BlockWhy Deferred
ActorsStateful virtual actors require a separate mental model (in later chapters)
WorkflowsLong-running orchestration builds on actors (in later chapters)

What the API Looks Like

When you use the state management building block, your code makes HTTP calls like:

text
POST http://localhost:3500/v1.0/state/statestore Content-Type: application/json [ { "key": "task-123", "value": {"title": "Buy groceries", "status": "pending"} } ]

Notice what's missing from this request:

  • No Redis host or port
  • No authentication credentials
  • No connection pooling configuration
  • No retry logic

Your code says "save this state." Dapr handles everything else.


Components: The Pluggable Implementations

A component is a Dapr configuration that binds a building block to a specific backend technology.

Continuing the remote control analogy: the component is the IR (infrared) blaster pointed at your specific TV. The remote (building block) doesn't change. The blaster (component) tells Dapr "when someone uses state management, talk to this Redis server."

Same Building Block, Different Components

The state management building block can use any of these components:

Component TypeBackendUse Case
state.redisRedisDevelopment, caching, simple state
state.postgresqlPostgreSQLProduction, relational queries needed
state.mongodbMongoDBDocument-oriented state
state.azure.cosmosdbAzure Cosmos DBGlobal distribution, multi-region
state.aws.dynamodbAWS DynamoDBAWS-native, serverless scaling

The pub/sub building block has similar flexibility:

Component TypeBackendUse Case
pubsub.redisRedis StreamsDevelopment, simple messaging
pubsub.kafkaApache KafkaProduction event streaming (earlier chapters)
pubsub.rabbitmqRabbitMQTraditional message queuing
pubsub.azure.servicebusAzure Service BusAzure-native messaging
pubsub.gcp.pubsubGoogle Cloud Pub/SubGCP-native messaging

The Key Insight: Same API, Different Backend

This is the single most important concept in Dapr.

Your application code calls the building block API:

python
from dapr.clients import DaprClient with DaprClient() as client: client.publish_event( pubsub_name='pubsub', # References the component name topic_name='task-events', data='{"task_id": "123"}' )

What happens next depends entirely on which component is configured with the name pubsub:

  • If pubsub is configured as pubsub.redis → Message goes to Redis Streams
  • If pubsub is configured as pubsub.kafka → Message goes to Kafka
  • If pubsub is configured as pubsub.rabbitmq → Message goes to RabbitMQ

Your code doesn't know or care. The component YAML makes that decision.

The Connection to Earlier Chapters

In earlier chapters, you wrote Kafka-specific code:

python
from aiokafka import AIOKafkaProducer producer = AIOKafkaProducer( bootstrap_servers='kafka-bootstrap:9092', value_serializer=lambda v: json.dumps(v).encode() ) await producer.start() await producer.send_and_wait('task-events', value={'task_id': '123'})

With Dapr, that same functionality becomes:

python
from dapr.clients import DaprClient with DaprClient() as client: client.publish_event( pubsub_name='kafka-pubsub', topic_name='task-events', data='{"task_id": "123"}' )

The Dapr version is simpler and portable. Change the component configuration, and you've switched message brokers without touching code.


Component YAML Structure

Every Dapr component is defined in a YAML file with a consistent structure. Understanding this structure helps you read, write, and debug component configurations.

Anatomy of a Component

yaml
apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: statestore namespace: default spec: type: state.redis version: v1 metadata: - name: redisHost value: redis:6379 - name: redisPassword value: ""

Let's examine each part:

FieldPurposeExample Values
apiVersionDapr component API versionAlways dapr.io/v1alpha1
kindResource typeAlways Component
metadata.nameName your code referencesstatestore, pubsub, kafka-events
metadata.namespaceKubernetes namespacedefault, production
spec.typeBuilding block + backendstate.redis, pubsub.kafka
spec.versionComponent versionv1 (most components)
spec.metadataBackend-specific configHost, credentials, options

How Your Code Finds the Component

When your code calls:

python
client.save_state(store_name='statestore', key='task-123', value='...')

Dapr looks for a component where metadata.name equals statestore. The spec.type tells Dapr which building block (state) and which implementation (Redis) to use.

Component Examples

Redis State Store:

yaml
apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: statestore spec: type: state.redis version: v1 metadata: - name: redisHost value: redis-master.default.svc.cluster.local:6379

Kafka Pub/Sub:

yaml
apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: kafka-pubsub spec: type: pubsub.kafka version: v1 metadata: - name: brokers value: task-events-kafka-bootstrap.kafka.svc.cluster.local:9092 - name: consumerGroup value: dapr-consumer - name: authType value: none

Kubernetes Secrets Store:

yaml
apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: kubernetes-secrets spec: type: secretstores.kubernetes version: v1 metadata: []

Notice that some components need extensive configuration (Kafka needs broker addresses and consumer groups), while others need almost none (Kubernetes secrets store uses the cluster's built-in secrets API).


Swapping Backends: The Portability Payoff

Here's where Dapr's abstraction model pays dividends.

Scenario: Development vs Production

Development (simple setup):

yaml
# components/pubsub-dev.yaml apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: pubsub spec: type: pubsub.redis version: v1 metadata: - name: redisHost value: redis:6379

Production (robust messaging):

yaml
# components/pubsub-prod.yaml apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: pubsub spec: type: pubsub.kafka version: v1 metadata: - name: brokers value: kafka-cluster.production.svc.cluster.local:9092 - name: consumerGroup value: task-api-consumers - name: authRequired value: "true" - name: saslUsername secretKeyRef: name: kafka-credentials key: username

Your application code is identical in both environments. It publishes to pubsub and subscribes to topics. The infrastructure decision—Redis in dev, Kafka in production—lives entirely in configuration.

Why This Matters

ConcernWithout DaprWith Dapr
Code changes for new backendRewrite with new SDKChange YAML file
Testing with different infraMock or run full infraSwap to simpler component
Multi-cloud deploymentDifferent code per cloudDifferent component YAML
Local developmentFull infra stackLightweight components

Scoping Components to Specific Apps

By default, Dapr components are available to all applications in a namespace. You can restrict which apps can access a component using scopes.

Why Scope Components?

  • Security: Only the payment service should access payment secrets
  • Isolation: Dev and staging environments shouldn't share state stores
  • Resource management: High-volume apps get dedicated components

How to Scope

Add a scopes field to your component:

yaml
apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: payment-secrets spec: type: secretstores.kubernetes version: v1 metadata: [] scopes: - payment-service - billing-service

Only applications with dapr.io/app-id: payment-service or dapr.io/app-id: billing-service can access this secrets store.

No Scopes = Available to All

If you omit the scopes field, the component is available to every Dapr-enabled app in the namespace.


Key Vocabulary

TermDefinition
Building BlockA Dapr capability exposed as an API (state, pub/sub, invoke, etc.)
ComponentA YAML configuration binding a building block to a specific backend
PluggableCan be swapped without code changes
Component SpecThe spec section of component YAML defining type and configuration
ScopesRestrict which apps can access a component

Reflect on Your Skill

You built a dapr-deployment skill in Lesson 0. Test and improve it based on what you learned.

Test Your Skill

text
Using my dapr-deployment skill, explain the difference between a Dapr building block and a Dapr component. Give me a concrete example with state management.

Does your skill clearly distinguish these concepts?

Identify Gaps

Ask yourself:

  • Does my skill explain that building blocks are APIs and components are implementations?
  • Does it include component YAML structure guidance?
  • Can it help me swap backends by changing YAML?

Improve Your Skill

If you found gaps:

text
My dapr-deployment skill doesn't clearly explain building blocks vs components. Update it to include: 1. Building blocks = portable APIs (state, pubsub, invoke, etc.) 2. Components = pluggable implementations (state.redis, pubsub.kafka, etc.) 3. Same code works with different backends by changing component YAML

Try With AI

You now understand Dapr's abstraction model. Use AI to explore how this applies to your own systems.

Setup

Open your AI assistant with context about Dapr. These prompts help you apply concepts to real scenarios.

Prompt 1: Building Block vs Component

text
What's the difference between a Dapr building block and a Dapr component? Give me a concrete example using state management with Redis. Show me how the same application code would work if I switched to PostgreSQL.

What you're learning: The building block/component separation is Dapr's core innovation. This prompt forces you to articulate the abstraction clearly and see how it enables backend swapping.

Prompt 2: Component YAML Analysis

text
Here's a Dapr component YAML. Explain what each field does: apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: statestore spec: type: state.redis version: v1 metadata: - name: redisHost value: redis:6379 - name: redisPassword secretKeyRef: name: redis-secret key: password

What you're learning: Reading component YAML fluently is essential for debugging and configuration. Notice the secretKeyRef pattern—Dapr can pull sensitive values from secret stores instead of hardcoding them.

Prompt 3: Backend Swapping Strategy

text
How would I swap from Redis pub/sub to Kafka pub/sub in Dapr without changing my application code? Show me both component YAML files and explain what stays the same vs what changes.

What you're learning: This is the portability payoff. You'll see that your code references pubsub_name='pubsub' and the component YAML decides whether that means Redis or Kafka. The swap is purely configuration.