Kafka Broker Startup Scripts

April 27, 2017

                                                                           

Running a Kafka Broker

Starting brokers in Kafka is pretty straightforward, here are some simple quick start instructions. But as developers, we want to do at least a little more than just the basics. For instance my first needs were to start multiple brokers on the same machine, and also to enable JMX.

Out of the box, you can simply rely on the supplied server.properties Each broker needs a unique id and needs a unique port. These are the corresponding properties from server.properties:

broker.id=0
#listeners=PLAINTEXT://:9092

9092 is the internal default, so the listeners line can be commented out. In older versions, you used to set the port via the port= property.

To start the broker, run this command (assumes Kafka is installed at /opt/kafka):

sudo /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

JMX

Kafka comes with a ton of JMX MBeans, but you need to configure the broker to have access to those beans from a JMX tool such as JConsole or even Kafka’s own remote JMX inspection tool. Since JMX is started as part of the Java environment, the JMX config needs to be part of the jvm command line.

The broker is started from kafka-server-start.sh which eventually calls kafka-run-class.sh, which contains the following:

# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
fi

# JMX port to use
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

You could modify your version of kafka-run-class.sh or you can put the values into the environment. I just set the JMX_PORT in the environment. You can apply any additional configuration you want. See the Oracle [JMX Technology] Agent documentation for more information on JMX.

sudo JMX_PORT=9990 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &

Multiple Brokers

To run multiple brokers on the same machine, the broker.id and port must be different for each instance. Originally, when I wanted to have two brokers, I just made two different server.properties, changed the id and port in the second file and executed the command with the proper server.properties file. And I even made different scripts so I didn’t have to type so much, kstart0.sh and kstart1.sh. But then I wanted to turn on JMX, and JMX doesn’t get turned on by a server.properties setting, so I reworked the scripts and leveraged the
--override command line option. Here is my kstart.sh script

#!/bin/bash

if [ $# -lt 1 ];
then
  echo "USAGE: kstart.sh id where id is 0..9"
  exit 1
fi

sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=$1 --override listeners=PLAINTEXT://:909$1  --override port=909$1&

You would run the script like kstart.sh 0 or kstart.sh 1 The script takes advantage of --override which overrides the server.properties setting. Let’s break down the command line:

run kafka-server-start with JMX_PORT in the enviroment, using the $1 parameter for the last digit in the port number

sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh

The kafka-server-start needs you to tell it exactly where the properties file lives.

/opt/kafka/config/server.properties

override the server.properties broker.id using the $1 parameter

--override broker.id=$1

override the server.properties listeners using the $1 parameter

--override listeners=PLAINTEXT://:909$1

override the port. I don’t think this is needed in the latest version anymore, but I just added it for completeness

--override port=909$1



About Cloudurable

We hope you enjoyed this article. Please provide feedback. Cloudurable provides Kafka training, Kafka consulting, Kafka support and helps setting up Kafka clusters in AWS.

Cloudurable™

Cloudurable™ provides:

Contact us

For more details on the subscription support or pricing please contact us or call ((415) 758-1113) or write info@cloudurable.com.

                                                                           

Kafka Tutorial
Akka Consulting
Cassandra Training
AWS Cassandra Database Support
Kafka Support Pricing
Cassandra Database Support Pricing
Non-stop Cassandra
Watchdog
Advantages of using Cloudurable™
Cassandra Consulting
Cloudurable™| Guide to AWS Cassandra Deploy
Cloudurable™| AWS Cassandra Guidelines and Notes
Free guide to deploying Cassandra on AWS
Kafka Training
Kafka Consulting
DynamoDB Training
DynamoDB Consulting
Kinesis Training
Kinesis Consulting
Kafka Tutorial PDF