DevOps | Cloud | Analytics | Open Source | Programming

How To Fix - Message Duplicates Issue in Kafka Broker ?

In this post , we will see How To Fix - Message Duplicates Issue in Kafka Broker . During Kafka runtime , sometimes due to network failure or any other reasons , the sync between Producer and Kafka broker gets broken . Thereby impacting the confusion whether the last messages were delivered (and replicated) or not. And once you bring up and restart the system and Resend the messages , you might notice Duplicate Messages in Kafka Broker. To handle this issue , we can take help of the Kafka Idempotent Producer feature . This feature is supported in version Kafka 0.11 . The idempotent producer feature ensures that Producer messages are not Duplicated . As a matter of fact , if your Kafka configuration or set-up uses Complete in-sync replicas or acks=all , then you Must Potentially Enable this Feature. This Simplifies the complete situation of Debugging when messages are lost. And saves a lot of time of the debugger or the developer. To enable the Idempotent Producer feature , do the following -

  • In the producer configuration  , add the below flag



  • Also Add very HIGH values for the Flag -



  • Since Idempotent Producer is used in case of full-set in-sync is expected , you also need to set the following flag



  • Leave below field without using any value so that Default value is taken up Automatically . Generally the value is taken between 0 to 5. But for safe side , its better to keep it default and let the system choose the appropriate one.

  Hope this post helps you To Fix - Message Duplicates Issue in Kafka Broker ?   **Additional Read - **