How to Send Large Messages in Kafka ?

---

Many time while trying to send large messages over Kafka it errors out with an exception – “MessageSizeTooLargeException”. These mostly occurs on the Producer side.

Fixes:

There are couple of configuration properties , you can try making changes and see it that works.

Producer: Increase max.request.size to send the larger message.

Kafka Broker side: replica.fetch.max.bytes – Size used to replicate the messages within the brokers as an practice of Replication.  The Brokers obey this size as a rule to create replicas of a message.  If this size is set as very small, the message is not replicated(fully committed). Hence message will not be committed. And so the consumer will not see the message.

Broker side: message.max.bytes – Maximum size of the message that a Broker can receive from a producer. Try setting this size on a higher side and see if the error exception goes away.

Broker side (per topic): max.message.bytes – Maximum size of the message allowed (by Broker) to be appended to a topic. Defaults is  broker's message.max.bytes. It can be lower than the broker's message.max.bytes.

Consumer: Increase max.partition.fetch.bytes to receive larger messages

Kafka Consumer side: fetch.message.max.bytes -  Maximum size of a message that can be consumed by the Kafka consumer. Hope these helps to resolve the Kafka exception.  

Other Interesting Reads -

• How to Send Large Messages in Kafka ?

• Fix Spark Error – “org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0”

• How to Handle Bad or Corrupt records in Apache Spark ?

• How to use Broadcast Variable in Spark ?

• How to log an error in Python ?

• How to Code Custom Exception Handling in Python ?

• How to Handle Errors and Exceptions in Python ?

• How To Fix – “Ssl: Certificate_Verify_Failed” Error in Python ?

 

kafka broker message size too large, handle large messages in apache kafka, kafka large message, Kafka default max message size, Kafka message size best practice, Kafka max message size configuration, Kafka message size too large, Kafka message compression, Kafka message size performance, Kafka message chunking, Tuning Kafka for large messages, What is the maximum size of the message does Kafka server can receive, How do I acknowledge Kafka messages, How long is Kafka messages, kafka message, kafka queue, kafka message queue, kafka message broker, kafka multiple consumers same topic,  kafka large messages, Kafka default max message size, Kafka message size best practice, Kafka max message size configuration, Kafka message size too large, Kafka message compression, Kafka message size performance, Kafka message chunking, Tuning Kafka for large messages, MessageSizeTooLargeException