How to Send Large Messages in Kafka ?
Many time while trying to send large messages over Kafka it errors out with an exception – “MessageSizeTooLargeException”. These mostly occurs on the Producer side.
There are couple of configuration properties , you can try making changes and see it that works.
Increase max.request.size to send the larger message.
Kafka Broker side
: replica.fetch.max.bytes – Size used to replicate the messages within the brokers as an practice of Replication. The Brokers obey this size as a rule to create replicas of a message. If this size is set as very small, the message is not replicated(fully committed). Hence message will not be committed. And so the consumer will not see the message.
: message.max.bytes – Maximum size of the message that a Broker can receive from a producer. Try setting this size on a higher side and see if the error exception goes away.
(per topic): max.message.bytes – Maximum size of the message allowed (by Broker) to be appended to a topic. Defaults is broker's message.max.bytes. It can be lower than the broker's message.max.bytes.
: Increase max.partition.fetch.bytes to receive larger messages
Kafka Consumer side
: fetch.message.max.bytes - Maximum size of a message that can be consumed by the Kafka consumer. Hope these helps to resolve the Kafka exception.
Other Interesting Reads -
kafka broker message size too large, handle large messages in apache kafka, kafka large message, Kafka default max message size, Kafka message size best practice, Kafka max message size configuration, Kafka message size too large, Kafka message compression, Kafka message size performance, Kafka message chunking, Tuning Kafka for large messages, What is the maximum size of the message does Kafka server can receive, How do I acknowledge Kafka messages, How long is Kafka messages, kafka message, kafka queue, kafka message queue, kafka message broker, kafka multiple consumers same topic, kafka large messages, Kafka default max message size, Kafka message size best practice, Kafka max message size configuration, Kafka message size too large, Kafka message compression, Kafka message size performance, Kafka message chunking, Tuning Kafka for large messages, MessageSizeTooLargeException