Kafka Broker Startup Scripts

Tweet                                                                            

Running a Kafka Broker

Starting brokers in Kafka is pretty straightforward, here are some simple quick start instructions. But as developers, we want to do at least a little more than just the basics. For instance my first needs were to start multiple brokers on the same machine, and also to enable JMX.

Out of the box, you can simply rely on the supplied server.properties Each broker needs a unique id and needs a unique port. These are the corresponding properties from server.properties:

broker.id=0
#listeners=PLAINTEXT://:9092

9092 is the internal default, so the listeners line can be commented out. In older versions, you used to set the port via the port= property.

To start the broker, run this command (assumes Kafka is installed at /opt/kafka):

sudo /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

JMX

Kafka comes with a ton of JMX MBeans, but you need to configure the broker to have access to those beans from a JMX tool such as JConsole or even Kafka’s own remote JMX inspection tool. Since JMX is started as part of the Java environment, the JMX config needs to be part of the jvm command line.

The broker is started from kafka-server-start.sh which eventually calls kafka-run-class.sh, which contains the following:

# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
fi

# JMX port to use
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

You could modify your version of kafka-run-class.sh or you can put the values into the environment. I just set the JMX_PORT in the environment. You can apply any additional configuration you want. See the Oracle [JMX Technology] Agent documentation for more information on JMX.

sudo JMX_PORT=9990 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &

Multiple Brokers

To run multiple brokers on the same machine, the broker.id and port must be different for each instance. Originally, when I wanted to have two brokers, I just made two different server.properties, changed the id and port in the second file and executed the command with the proper server.properties file. And I even made different scripts so I didn’t have to type so much, kstart0.sh and kstart1.sh. But then I wanted to turn on JMX, and JMX doesn’t get turned on by a server.properties setting, so I reworked the scripts and leveraged the
--override command line option. Here is my kstart.sh script

#!/bin/bash

if [ $# -lt 1 ];
then
  echo "USAGE: kstart.sh id where id is 0..9"
  exit 1
fi

sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=$1 --override listeners=PLAINTEXT://:909$1  --override port=909$1&

You would run the script like kstart.sh 0 or kstart.sh 1 The script takes advantage of --override which overrides the server.properties setting. Let’s break down the command line:

run kafka-server-start with JMX_PORT in the enviroment, using the $1 parameter for the last digit in the port number

sudo JMX_PORT=999$1 /opt/kafka/bin/kafka-server-start.sh

The kafka-server-start needs you to tell it exactly where the properties file lives.

/opt/kafka/config/server.properties

override the server.properties broker.id using the $1 parameter

--override broker.id=$1

override the server.properties listeners using the $1 parameter

--override listeners=PLAINTEXT://:909$1

override the port. I don’t think this is needed in the latest version anymore, but I just added it for completeness

--override port=909$1

Related content

• What is Kafka?
• Kafka Architecture
• Kafka Topic Architecture
• Kafka Consumer Architecture
• Kafka Producer Architecture
• Kafka Architecture and low level design
• Kafka and Schema Registry
• Kafka and Avro
• Kafka Ecosystem
• Kafka vs. JMS
• Kafka versus Kinesis
• Kafka Tutorial: Using Kafka from the command line
• Kafka Tutorial: Kafka Broker Failover and Consumer Failover
• Kafka Tutorial
• Kafka Tutorial: Writing a Kafka Producer example in Java
• Kafka Tutorial: Writing a Kafka Consumer example in Java
• Kafka Architecture: Log Compaction

About Cloudurable

We hope you enjoyed this article. Please provide feedback. Cloudurable provides Kafka training, Kafka consulting, Kafka support and helps setting up Kafka clusters in AWS.

Cloudurable™

Cloudurable™ provides:

• Subscription Kafka Streaming Platform support (Support subscription pricing for Cassandra and Kafka).
• Kafka Quickstart Mentoring Consulting
• Kafka Architectural Analysis Consulting
• Training and mentoring for Cassandra and Kafka

Contact us

For more details on the subscription support or pricing please contact us or call ((415) 758-1113) or write info@cloudurable.com.

Check out our new GoLang course. We provide onsite Go Lang training which is instructor led.

Tweet