How To Fix - Message Duplicates Issue in Kafka Broker ?
In this post , we will see How To Fix - Message Duplicates Issue in Kafka Broker . During Kafka runtime , sometimes due to network failure or any other reasons , the sync between Producer and Kafka broker gets broken . Thereby impacting the confusion whether the last messages were delivered (and replicated) or not. And once you bring up and restart the system and Resend the messages , you might notice Duplicate Messages in Kafka Broker. To handle this issue , we can take help of the Kafka Idempotent Producer feature . This feature is supported in version Kafka 0.11 . The idempotent producer feature ensures that Producer messages are not Duplicated . As a matter of fact , if your Kafka configuration or set-up uses Complete in-sync replicas or acks=all , then you Must Potentially Enable this Feature. This Simplifies the complete situation of Debugging when messages are lost. And saves a lot of time of the debugger or the developer. To enable the Idempotent Producer feature , do the following -
- In the producer configuration , add the below flag
enable.idempotence=true
- Also Add very HIGH values for the Flag -
message.send.max.retries
- Since Idempotent Producer is used in case of full-set in-sync is expected , you also need to set the following flag
Acks=All
- Leave below field without using any value so that Default value is taken up Automatically . Generally the value is taken between 0 to 5. But for safe side , its better to keep it default and let the system choose the appropriate one.
max.in.flight.requests.per.connection