Producers

Kafka Architecture: Producers - 2025 Edition

🚀 What’s New in This 2025 Update

Major Updates and Changes

• Metadata Bootstrapping - KIP-1102 enables automatic metadata recovery
• Enhanced Protocol Resilience - Improved error handling and recovery
• Mandatory Modern Protocols - Requires broker 2.1+ for Java clients
• KRaft Performance Benefits - Reduced latency with ZooKeeper removal
• Strengthened Best Practices - Focus on idempotency and transactions
• Clear Upgrade Path - KIP-1124 migration guidance

Deprecated Features

• ❌ Pre-2.1 protocol versions - Old client protocols removed
• ❌ Legacy compatibility modes - Modern protocols required
• ❌ ZooKeeper-based metadata - Replaced by KRaft

Ready to build high-performance, resilient producers? Let’s master Kafka producer architecture in the modern era.

Continue reading

Kafka Architecture: Producers

Kafka Producer Architecture - Picking the partition of records

This article covers some lower level details of Kafka producer architecture. It is a continuation of the Kafka Architecture and Kafka Topic Architecture articles.

This article covers Kafka Producer Architecture with a discussion of how a partition is chosen, producer cadence, and partitioning strategies.

Kafka Producers

Kafka producers send records to topics. The records are sometimes referred to as messages.
The producer picks which partition to send a record to per topic. The producer can send records round-robin. The producer could implement priority systems based on sending records to certain partitions based on the priority of the record.

Continue reading