DevOps | Cloud | Analytics | Open Source | Programming





List of Kafka Commands Cheatsheet


List of Kafka Commands Cheatsheet. In this post we will explore the common kafka commands , kafka consumer group command , kafka command line , kafka consumer command , kafka console consumer command, kafka console producer command .

(We Keep Updating This List - So You can BookMark it)

   

  Start Zookeeper Services -


bin/zookeeper-server-start.sh config/zookeeper.properties
  Start Kafka Server -


bin/kafka-server-start.sh config/server.properties
  To Retrieve Replica Assignment Information - This cli script is used to query a list of replicas per log directory on a broker. It provides information for optimizing replica assignment across various brokers in the cluster.


kafka-log-dirs.sh --describe --bootstrap-server hostname:port --broker-list broker 1, broker 2 --topic-list topic 1, topic 2
 

2. Kafka Consumer Commands:

Start Kafka Console Consumer -


bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --max-messages 100
\[record-earliest-offset\]
\[record-earliest-offset+1\]
  Write To Topics from Kafka Consumer - Consume from Kafka Broker , process the message and then Write back to Some Other Topic. These need to be part of the Consumer code logic. Below is just a reference using python "kafka" package.


producer.send(Topic1, key = msg.key, value = outmsg1).add\_callback(<SEND\_SUCCESS\_FUNCTION>).add\_errback(<SEND\_ERROR\_FUNCTION>)
producer.flush()
consumer.commit()
  Kafka Consumer Group -


kafka-consumer-groups.sh --bootstrap-server \[BORKER1\],\[BROKER2\]... --describe --group \[GROUP\_ID\]
  List All the Kafka Consumers - Below command List Consumers that use the Java consumer API i.e. non-Zookeeper-based consumers


bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
  Below command List Consumers that use Zookeeper i.e. those not using the Java consumer API


bin/kafka-consumer-groups.sh --list --zookeeper localhost:2181
  **Get Detail Info about Your Consumer Group - **


bin/kafka-consumer-groups.sh --describe --group <YOUR\_CONSUMER-GROUP-NAME> --bootstrap-server localhost:9092
  View Only 10  Messages on the Terminal -


./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --max-messages 10
 

Create Kafka Topic -


bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic awesome
  Create a Kafka topic if  topic doesn’t exist -


bin/kafka-topics.sh --zookeeper $ZK\_HOSTS --create --topic test --partitions 3 --replication-factor 3 --**if-not-exists**
  Delete Kafka Topic -


bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
  List Kafka Topic -


bin/kafka-topics.sh --list --zookeeper localhost:2181
For Kafka 2.2 and above , you can also use -


bin/kafka-topics.sh --list --bootstrap-server <BROKER-LIST>
  Topic Properties - This command gives three information -

  • Partition count
  • Replication factor:  '1' for no redundancy and higher for more redundancy.
  • Replicas and in-sync replicas (ISR):  Broker ID’s with partitions and which replicas are current.

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
  Increase Topic Partitions - Topic having key, any partition logic or messages order might be impacted . So double check in such scenarios.


bin/kafka-topics.sh --zookeeper $ZK\_HOSTS --alter --topic test --partitions 5
  Purge a Running Kafka Topic - This is not quite straightforward. So refer our other post for how to do that here - How to Purge a Running Kafka Topic ?    

4. Kafka Spark Application Commands :

Spark-Submit with Kafka Jars - Below command is for submitting a spark python application - spark.py which processes Kafka Topic "test" and writes to postgres database. Hence you would need additional jars which needs to be provided through the --jars arguement.


./bin/spark-submit \\

  --packages org.apache.spark:spark-sql-kafka-0-10\_2.11:2.4.1 \\

  --jars spark-sql-kafka-0-10\_2.11-2.4.1.jar,kafka-clients-0.10.1.0.jar, postgresql-42.2.10.jar \\

  spark.py \\   

  localhost:9092\\

  subscribe test
 

5. Kafka Producer Commands :

  Produce a Message to Kafka Topic -


bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
 

6. Kafka Docker Commands :

  Start the Kafka Docker -


docker-compose up -d
  Attach to the Kafka Broker -


docker exec -it kafka bash
 

  List all available Kafka brokers in a cluster


./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Alternative1 - You can also use -


echo dump | nc -q 2 localhost 2181 | grep brokers
Alternative2 - You can also use - (For Newer versions of Kafka 5.3+)


kafka-broker-api-versions --bootstrap-server BROKER | grep 9092
  Hope you enjoyed this post.   Kafka Command Cheatsheet      

Other Reads -

 

kafka create topic, kafka commands , kafka consumer group command , kafka command line , kafka consumer command , kafka consumer group, kafka console consumer , kafka console producer , kafka offset , kafka consumer , Kafka producer