Chapter 52: Event-Driven Architecture with Kafka
You build the kafka-events skill first, then use each lesson to test and deepen it—from EDA fundamentals to production-grade operations. Kafka 4.0+ KRaft mode (no ZooKeeper) is the default.
Goals
- Understand event-driven architecture and Kafka’s core model (topics, partitions, consumer groups)
- Deploy Kafka with Strimzi in KRaft mode
- Build reliable producers/consumers with delivery guarantees and transactions
- Integrate Kafka with FastAPI using schemas (Avro + schema registry)
- Apply advanced patterns: Connect, CDC with Debezium, agent events, saga
- Operate Kafka: production config, monitoring, debugging
- Capture everything in a reusable Kafka skill
Lesson Progression
| # | Lesson | Focus |
|---|---|---|
| 0 | Build Your Kafka Skill | Scaffold from docs |
| 1-3 | EDA Foundations | Why events, EDA patterns, Kafka mental model |
| 4-8 | Kafka Core | Strimzi deploy, producers, consumers, groups |
| 9-13 | Production Patterns | Async FastAPI integration, schemas, delivery semantics, transactions, reliability config |
| 14-17 | Advanced Patterns | Connect, Debezium CDC, agent event design, saga orchestration |
| 18-19 | Operations | Production Strimzi, monitoring/debugging |
| 20 | AI-Assisted Development | Use AI to generate/tune configs |
| 21 | Capstone: Event-Driven Notifications | Production pipeline |
Each lesson ends with a skill reflection: test, find gaps, and improve.
Outcome & Method
You finish with a production-ready Kafka deployment, reliable producer/consumer code integrated with FastAPI, and a Kafka skill for future projects. The chapter follows the 4-Layer approach: foundations → production patterns → AI-assisted authoring → spec-driven capstone.
Prerequisites
- Chapters 49-51: container image and Kubernetes/Helm familiarity
- Ability to run a local Kubernetes cluster (e.g., Docker Desktop) for Strimzi
- Implement reliable producers: acks semantics, retries, idempotent producer, error handling
- Implement robust consumers: Consumer groups, rebalancing, offset management, lag monitoring
- Integrate with FastAPI: Async producers/consumers, lifespan events, background tasks
- Design event schemas: Avro with Schema Registry, schema evolution, breaking change prevention
- Apply delivery guarantees: At-least-once, at-most-once, exactly-once semantics and trade-offs
- Use transactions: Consume-process-produce pattern, zombie fencing, read_committed isolation
- Build data pipelines: Kafka Connect, Debezium CDC, outbox pattern for microservices
- Implement agent patterns: Task events, notification fanout, audit logs, saga pattern
- Run Kafka on Kubernetes: Strimzi operator, Kafka CRDs, KRaft mode, production configuration
- Debug production issues: Consumer lag, under-replicated partitions, rebalancing storms
Technology Choices
| Component | Choice | Rationale |
|---|---|---|
| Kafka Operator | Strimzi | CNCF project, industry standard for Kafka on K8s |
| Kafka Mode | KRaft (no ZooKeeper) | Kafka 4.0+ default, simpler architecture |
| Python Client | confluent-kafka-python | Best performance, native async, Schema Registry support |
| Schemas | Avro + Confluent Schema Registry | Industry standard, evolution support |
| Platform | Docker Desktop Kubernetes | Consistent with Chapters 49-51 |
| CDC | Debezium | Best-in-class change data capture |
What's NOT Covered
This chapter focuses on developer skills, not SRE operations:
- Docker Compose — we use Kubernetes throughout Part 7
- Multi-datacenter replication (MirrorMaker 2)
- Security deep dive (SASL, SSL, ACLs) — covered at overview level only
- Kafka Streams framework — separate advanced topic
- Broker hardware sizing and tuning
- ZooKeeper — removed in Kafka 4.0
Looking Ahead
This chapter teaches Kafka directly. Chapter 53 (Dapr) shows how to abstract pub/sub behind Dapr's API, making your code portable across message brokers while retaining the concepts you learned here.