Kafka

Kafka Tutorial

Kafka Tutorial This comprehensive Kafka tutorial covers Kafka architecture and design. The Kafka tutorial has example Java Kafka producers and Kafka consumers. The Kafka tutorial also covers Avro and Schema Registry. Kafka Tutorial Part 1: What is Kafka? Kafka Tutorial Part 2: Kafka Architecture Kafka Tutorial Part 3: Kafka Topic Architecture Kafka Tutorial Part 4: Kafka Consumer Architecture Kafka Tutorial Part 5: Kafka Producer Architecture Kafka Tutorial Part 6: Using Kafka from the command line Kafka Tutorial Part 7: Kafka Broker Failover and Consumer Failover Kafka Tutorial Part 8: Kafka Ecosystem Kafka Tutorial Part 9: Kafka Low-Level Design Kafka Tutorial Part 10: Kafka Log Compaction Architecture Kafka Tutorial Part 11: Writing a Kafka Producer example in Java Kafka Tutorial Part 12: Writing a Kafka Consumer example in Java Kafka Tutorial Part 13: Writing Advanced Kafka Producer with Java examples Kafka Tutorial Part 14: Writing Advanced Kafka Consumer with Java examples Kafka Tutorial Part 15: Kafka and Avro Kafka Tutorial Part 16: Kafka and Schema Registry Kafka Tutorial

Continue reading

Kafka Architecture: Log Compaction

Kafka Architecture: Log Compaction This post really picks off from our series on Kafka architecture which includes Kafka topics architecture, Kafka producer architecture, Kafka consumer architecture and Kafka ecosystem architecture. This article is heavily inspired by the Kafka section on design around log compaction. You can think of it as the cliff notes about Kafka design around log compaction. Kafka can delete older records based on time or size of a log.

Continue reading

Kafka Architecture: Low Level

If you are not sure what Kafka is, see What is Kafka?. Kafka Architecture: Low-Level Design This post really picks off from our series on Kafka architecture which includes Kafka topics architecture, Kafka producer architecture, Kafka consumer architecture and Kafka ecosystem architecture. This article is heavily inspired by the Kafka section on design. You can think of it as the cliff notes. Kafka Design Motivation LinkedIn engineering built Kafka to support real-time analytics.

Continue reading

Kafka Architecture: Consumers

Kafka Consumer Architecture - Consumer Groups and subscriptions This article covers some lower level details of Kafka consumer architecture. It is a continuation of the Kafka Architecture, Kafka Topic Architecture, and Kafka Producer Architecture articles. This article covers Kafka Consumer Architecture with a discussion consumer groups and how record processing is shared among a consumer group as well as failover for Kafka consumers. Kafka Consumer Groups You group consumers into a consumer group by use case or function of the group.

Continue reading

Kafka Architecture: Producers

Kafka Producer Architecture - Picking the partition of records This article covers some lower level details of Kafka producer architecture. It is a continuation of the Kafka Architecture and Kafka Topic Architecture articles. This article covers Kafka Producer Architecture with a discussion of how a partition is chosen, producer cadence, and partitioning strategies. Kafka Producers Kafka producers send records to topics. The records are sometimes referred to as messages.

Continue reading

Kafka Topic Architecture

Kafka Topic Architecture - Replication, Failover and Parallel Processing This article covers some lower level details of Kafka topic architecture. It is a continuation of the Kafka Architecture article. This article covers Kafka Topic’s Architecture with a discussion of how partitions are used for fail-over and parallel processing. Kafka Topics, Logs, Partitions Recall that a Kafka topic is a named stream of records. Kafka stores topics in logs. A topic log is broken up into partitions.

Continue reading

Kafka vs. JMS

Kafka vs JMS, SQS, RabbitMQ Messaging Is Kafka a queue or a publish and subscribe system? Yes. It can be both. Kafka is like a queue for consumer groups, which we cover later. Basically, Kafka is a queue system per consumer group so it can do load balancing like JMS, RabbitMQ, etc. Kafka is like topics in JMS, RabbitMQ, and other MOM systems for multiple consumer groups. Kafka has topics and producers publish to the topics and the subscribers (Consumer Groups) read from the topics.

Continue reading

Kafka Architecture

If you are not sure what Kafka is, see What is Kafka?. Kafka Architecture Kafka consists of Records, Topics, Consumers, Producers, Brokers, Logs, Partitions, and Clusters. Records can have key (optional), value and timestamp. Kafka Records are immutable. A Kafka Topic is a stream of records ("/orders", "/user-signups"). You can think of a Topic as a feed name. A topic has a Log which is the topic’s storage on disk.

Continue reading

The Kafka Ecosystem - Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry

The Kafka Ecosystem - Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry The core of Kafka is the brokers, topics, logs, partitions, and cluster. The core also consists of related tools like MirrorMaker. The aforementioned is Kafka as it exists in Apache. The Kafka ecosystem consists of Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry. Most of the additional pieces of the Kafka ecosystem comes from Confluent and is not part of Apache.

Continue reading

What is Apache Kafka?

What is Kafka? Kafka’s growth is exploding, more than 1⁄3 of all Fortune 500 companies use Kafka. These companies includes the top ten travel companies, 7 of top ten banks, 8 of top ten insurance companies, 9 of top ten telecom companies, and much more. LinkedIn, Microsoft and Netflix process four comma messages a day with Kafka (1,000,000,000,000). Kafka is used for real-time streams of data, used to collect big data or to do real time analysis or both).

Continue reading


Kafka Tutorial
Akka Consulting
Cassandra Training
AWS Cassandra Database Support
Kafka Support Pricing
Cassandra Database Support Pricing
Non-stop Cassandra
Watchdog
Advantages of using Cloudurable™
Cassandra Consulting
Cloudurable™| Guide to AWS Cassandra Deploy
Cloudurable™| AWS Cassandra Guidelines and Notes
Free guide to deploying Cassandra on AWS
Kafka Training
Kafka Consulting
DynamoDB Training
DynamoDB Consulting
Kinesis Training
Kinesis Consulting
Kafka Tutorial PDF