DevOps | Cloud | Analytics | Open Source | Programming





How to Setup a Multi Node Kafka Cluster or Brokers ?


In this post , we will see - How to Setup Multi Node Kafka Cluster or Brokers ? As we all know the Real strength and True purpose of Kafka System is inculcated in a Multi Node setup. An multi node Kafka Architecture takes care of Data Replication and Safety as well as efficient Topic Partitions.

Let us follow the below steps to start a Multi node Kafka Cluster or Broker system . For this case we will create a 3-Node Kafka Cluster. Our system Cluster comprises of 3 nodes -

  • Node A - Kafka Broker A
  • Node B - Kafka Broker B
  • Node C - Kafka Broker C
  Step 1:

It is assumed that you has already downloaded the Kafka libraries from the Apache Kafka or Confluent Kafka in all the 3 Machines . For sake of simplicity , we will assume the Kafka is downloaded and unzipped in /usr/Kafka directory . So all our Kafka Files and executables will be in /usr/Kafka in ALL THE 3-MACHINES.  Use your respective Kafka directory. Zookeeper also comes as part of the Kafka Download. So it will also be within /usr/Kafka Directory.   Step 2: Zookeeper Configuration

We need to chnage the zookeeper config file -   zookeeper.properties in all the 3-NODES ~/kafka/config/zookeeper.properties Note there are many details which could be changed and tried but below is a MANDATORY for our case . This basically introduces the 3-Brokers nodes to the Zookeeper environment.

  • Broker A  ~/kafka/config/zookeeper.properties

\# list of servers
server.<Node1\_UNIQUE\_ID\_1>=0.0.0.0:2888:3888
server.<Node2\_UNIQUE\_ID\_2>=<Ip of Broker B>:2888:3888
server.<Node3\_UNIQUE\_ID\_3>=<ip of Broker C>:2888:3888
Note the Unique Id details following server. parameter. It is very important that this id is Unique across all the 3-Nodes in the Cluster.  

  • Broker B  ~/kafka/config/zookeeper.properties

\# list of servers
server.<Node1\_UNIQUE\_ID\_1>=<ip of Broker A>:3888
server.<Node2\_UNIQUE\_ID\_2>=0.0.0.0:2888:3888
server.<Node3\_UNIQUE\_ID\_3>=<ip of Broker B>:2888:3888
  Broker C  ~/kafka/config/zookeeper.properties


\# list of servers
server.<Node1\_UNIQUE\_ID\_1>=<ip of Broker A>:2888:3888
server.<Node2\_UNIQUE\_ID\_2>=<Ip of Broker B>:2888:3888
server.<Node3\_UNIQUE\_ID\_3>=0.0.0.0:2888:3888
If you have even more Nodes or Servers in your cluster , you can follow and repeat the similar steps for them as well   Step 3: Kafka Configuration

Next we need to configure the Kafka configuration in all the 3-Kafka Brokers or Nodes.

  • We first have to locate the Config Property file . Inside that we will find a server.properties file

cd  /usr/Kafka/config
  • Broker A  server.properties
Within that file , change the below -


broker.id=1 \-----> THIS SHOULD BE UNIQUE FOR EACH BROKER
\# add all 3 zookeeper instances ip here
zookeeper.connect=<BrokerA\_IP>:2181,<BrokerB\_IP>:2181,<BrokerC\_IP>:2181
 

  • Modify Broker B Server.properties file

Within that file , change the below -

broker.id=2 -----> THIS SHOULD BE UNIQUE FOR EACH BROKER
# add all 3 zookeeper instances ip here
zookeeper.connect=<BrokerA\_IP>:2181,<BrokerB\_IP>:2181,<BrokerC\_IP>:2181
 

  • Modify Broker C Server.properties file - serverC.properties

Within that file , change the below -

broker.id=3 -----> THIS SHOULD BE UNIQUE FOR EACH BROKER
# add all 3 zookeeper instances ip here
zookeeper.connect=<BrokerA\_IP>:2181,<BrokerB\_IP>:2181,<BrokerC\_IP>:2181
  Step 4: Start the Zookeeper & Kafka

Run the below commands to start the Zookeeper & Kafka in all the 3 Nodes.  

  • Broker A

cd ~/kafka

$ bin/zookeeper-server-start.sh ~/config/zookeeper.properties
 
$ bin/kafka-server-start.sh ~/config/server.properties
 

  • Broker B

cd ~/kafka

$ bin/zookeeper-server-start.sh ~/config/zookeeper.properties
 
$ bin/kafka-server-start.sh ~/config/server.properties
 

  • Broker C

cd ~/kafka

$ bin/zookeeper-server-start.sh ~/config/zookeeper.properties
 
$ bin/kafka-server-start.sh ~/config/server.properties
  Hope this helps to set-up a Multi node Kafka Cluster or Broker system.     Other Reads -