April 27, 2017
Running a Kafka Broker
Starting brokers in Kafka is pretty straightforward, here are some simple quick start instructions. But as developers, we want to do at least a little more than just the basics. For instance my first needs were to start multiple brokers on the same machine, and also to enable JMX.
Out of the box, you can simply rely on the supplied server.properties Each broker needs a unique id and needs a unique port. These are the corresponding properties from server.properties:
9092 is the internal default, so the listeners line can be commented out. In older versions, you used to set the port via the
To start the broker, run this command (assumes Kafka is installed at /opt/kafka):
sudo /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Kafka comes with a ton of JMX MBeans, but you need to configure the broker to have access to those beans from a JMX tool such as JConsole or even Kafka’s own remote JMX inspection tool. Since JMX is started as part of the Java environment, the JMX config needs to be part of the jvm command line.
The broker is started from
kafka-server-start.sh which eventually calls
kafka-run-class.sh, which contains the following:
# JMX settings if [ -z "$KAFKA_JMX_OPTS" ]; then KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false " fi # JMX port to use if [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT " fi
You could modify your version of
kafka-run-class.sh or you can put the values into the environment. I just set the
JMX_PORT in the environment. You can apply any additional configuration
you want. See the Oracle [JMX Technology] Agent documentation for more information on JMX.
sudo JMX_PORT=9990 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &
To run multiple brokers on the same machine, the broker.id and port must be different for each instance.
Originally, when I wanted to have two brokers, I just made two different server.properties, changed the id and port in the second file and executed the command with the proper server.properties file.
And I even made different scripts so I didn’t have to type so much, kstart0.sh and kstart1.sh.
But then I wanted to turn on JMX, and JMX doesn’t get turned on by a server.properties setting, so I reworked the scripts and leveraged the
--override command line option.
Here is my kstart.sh script
#!/bin/bash if [ $# -lt 1 ]; then echo "USAGE: kstart.sh id where id is 0..9" exit 1 fi sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=$1 --override listeners=PLAINTEXT://:909$1 --override port=909$1&
You would run the script like
kstart.sh 0 or
kstart.sh 1 The script takes advantage of
--override which overrides the server.properties setting. Let’s break down the command line:
run kafka-server-start with JMX_PORT in the enviroment, using the $1 parameter for the last digit in the port number
sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh
The kafka-server-start needs you to tell it exactly where the properties file lives.
override the server.properties broker.id using the $1 parameter
override the server.properties listeners using the $1 parameter
override the port. I don’t think this is needed in the latest version anymore, but I just added it for completeness
- What is Kafka?
- Kafka Architecture
- Kafka Topic Architecture
- Kafka Consumer Architecture
- Kafka Producer Architecture
- Kafka Architecture and low level design
- Kafka and Schema Registry
- Kafka and Avro
- Kafka Ecosystem
- Kafka vs. JMS
- Kafka versus Kinesis
- Kafka Tutorial: Using Kafka from the command line
- Kafka Tutorial: Kafka Broker Failover and Consumer Failover
- Kafka Tutorial
- Kafka Tutorial: Writing a Kafka Producer example in Java
- Kafka Tutorial: Writing a Kafka Consumer example in Java
- Kafka Architecture: Log Compaction
- Subscription Kafka Streaming Platform support (Support subscription pricing for Cassandra and Kafka).
- Kafka Quickstart Mentoring Consulting
- Kafka Architectural Analysis Consulting
- Training and mentoring for Cassandra and Kafka
AWS Cassandra Database Support
Kafka Support Pricing
Cassandra Database Support Pricing
Advantages of using Cloudurable™
Cloudurable™| Guide to AWS Cassandra Deploy
Cloudurable™| AWS Cassandra Guidelines and Notes
Free guide to deploying Cassandra on AWS
Kafka Tutorial PDF