Apache Kafka for Event-Driven Systems

Section 1: Event-Driven Architecture Concepts

• Why direct point-to-point integration breaks down as services grow and downstream systems fail
• Events, message queues and event streaming, and the differences that matter
• Real use cases: system integration, real-time processing and event logging
• When Kafka is the right choice and when a direct API call still fits better

Section 2: Apache Kafka Architecture

• Core components: brokers, topics, partitions, replicas, leaders and followers
• The append-only log and the offsets that let consumers replay data
• Replication and tolerating broker failure
• Retention periods and log compaction

Section 3: Installing and Operating a Cluster

• Standing up a Kafka cluster with Docker Compose and checking cluster health
• Command line tools for creating topics, producing and consuming messages
• Using a UI to inspect topics, partitions and consumer groups
• Lab: build a cluster, create your first topic and produce and consume from the CLI

Section 4: Designing Topics and Partitions

• Topic naming conventions and sizing partition counts for the expected load
• Choosing keys to distribute data while keeping related records in order
• How partition count affects throughput, parallelism and adding consumers later
• Lab: design and create topics for a business scenario and verify data distribution

Section 5: Producers and Delivery Guarantees

• Writing producers and the settings that matter: acks, retries and batching
• Delivery guarantees: at most once, at least once and exactly once
• The idempotent producer and transactions when correctness is critical
• Lab: write a producer and observe its behaviour when a broker stops responding

Section 6: Consumers and Consumer Groups

• Writing consumers and how consumer groups divide partitions between members
• Offset management: auto commit versus manual commit and the risk of duplicates or loss
• What triggers a rebalance, its impact and how to reduce the pain
• Lab: add and remove consumers from a group and watch partitions redistribute

Section 7: Schemas and Data Formats

• Common formats: JSON and Avro and the trade-offs of each
• Schema Registry and keeping compatibility as the data structure evolves
• Designing events that other systems can rely on for years
• Lab: define a schema for an event and evolve it without breaking consumers

Section 8: Kafka Connect for System Integration

• How Kafka Connect works and the difference between source and sink connectors
• Pulling data from databases into Kafka and pushing it out to destinations
• Change data capture for tracking database changes as events
• Lab: configure a connector that streams database rows into a topic with no code

Section 9: Stream Processing with Kafka Streams

• Processing data in motion: filtering, transforming and aggregating in real time
• Windowed aggregation and continuous counts and summaries
• Streams versus tables and joining two streams together
• Lab: build a stream processing job that summarizes topic data in real time

Section 10: Reliability and Performance

• Replication factor and minimum in-sync replicas so data is never lost
• Handling and recovering from broker failure, and testing it for real
• Tuning producer and consumer performance and managing consumer lag
• Lab: simulate a broker failure and observe how the system keeps working

Section 11: Security and Monitoring

• Encrypting connections and authenticating clients
• Least-privilege authorization for topic access
• Monitoring the cluster: consumer lag, broker health and the metrics that matter
• Planning cluster maintenance and upgrading without downtime

Section 12: Capstone and Real-World Adoption

• Capstone: design and build an event-driven system for one business process end to end
• Present your topic architecture and the reasoning behind the design
• How to adopt Kafka in the organization, starting small before scaling out
• Workshop: present what you built and review the design together