Kafka Log Compaction

Kafka Architecture: Log Compaction - 2025 Edition

🚀 What’s New in This 2025 Update

Major Updates and Changes

• KRaft-Managed Compaction - All compaction under KRaft control
• Tiered Storage Integration - Compaction across local and remote tiers
• Diskless Topics - Object storage compaction (KIP-1165)
• Performance Optimizations - Reduced I/O with cloud-native design
• Simplified Operations - No ZooKeeper coordination needed
• Enhanced Monitoring - Better visibility into compaction progress

Deprecated Features

• ❌ ZooKeeper-based coordination - Fully removed
• ❌ Legacy message formats - v0 and v1 no longer supported
• ❌ Old compaction metrics - Updated for KRaft

Ready to master Kafka’s powerful state management feature? Let’s explore how log compaction enables event sourcing and stateful processing at scale.

Continue reading

Kafka Architecture: Log Compaction

Kafka Architecture: Log Compaction

This post really picks off from our series on Kafka architecture which includes Kafka topics architecture, Kafka producer architecture, Kafka consumer architecture and Kafka ecosystem architecture.

This article is heavily inspired by the Kafka section on design around log compaction. You can think of it as the cliff notes about Kafka design around log compaction.

Kafka can delete older records based on time or size of a log. Kafka also supports log compaction for record key compaction. Log compaction means that Kafka will keep the latest version of a record and delete the older versions during a log compaction.

Continue reading