DevOps | Cloud | Analytics | Open Source | Programming





How To Fix - Message Duplicates Issue in Kafka Broker ?



In this post , we will see How To Fix - Message Duplicates Issue in Kafka Broker . During Kafka runtime , sometimes due to network failure or any other reasons , the sync between Producer and Kafka broker gets broken . Thereby impacting the confusion whether the last messages were delivered (and replicated) or not. And once you bring up and restart the system and Resend the messages , you might notice Duplicate Messages in Kafka Broker. To handle this issue , we can take help of the Kafka Idempotent Producer feature . This feature is supported in version Kafka 0.11 . The idempotent producer feature ensures that Producer messages are not Duplicated . As a matter of fact , if your Kafka configuration or set-up uses Complete in-sync replicas or acks=all , then you Must Potentially Enable this Feature. This Simplifies the complete situation of Debugging when messages are lost. And saves a lot of time of the debugger or the developer. To enable the Idempotent Producer feature , do the following -

  • In the producer configuration  , add the below flag

enable.idempotence=true

 

  • Also Add very HIGH values for the Flag -

message.send.max.retries

 

  • Since Idempotent Producer is used in case of full-set in-sync is expected , you also need to set the following flag

Acks=All

 

  • Leave below field without using any value so that Default value is taken up Automatically . Generally the value is taken between 0 to 5. But for safe side , its better to keep it default and let the system choose the appropriate one.

max.in.flight.requests.per.connection

  Hope this post helps you To Fix - Message Duplicates Issue in Kafka Broker ?   **Additional Read - **