DevOps | Cloud | Analytics | Open Source | Programming





How To Mask - Confidential Info in Kafka Connect Logs ?


We will see the options that we have as to How To Mask - Confidential Info in Kafka Connect Logs. While executing Kafka connector it logs the user credentials. This is not ideal and as such we should try to keep such information off from the Logs .  We can do that using below options -  

Option 1:

We can mask the confidential information using the connection property files. The connection property , within config, has user & password field which can be used to fill-in the login credentials for Kafka connect. This would avoid logging these information . See the below example as to how to use this -


curl -X POST http://localhost:8083/Kafka\_connectors -H "Content-Type: application/json" -d \\
'{ 
   "name": <SOURCE\_NAME>,
   "config":{ 
      "connector.class": <CONNECTOR\_CLASS\_NAME> ,
      "connection.url": <JDBC\_URL>,
      "connection.user": <USER\_NAME> , <----- This will not be logged
      "connection.password": <PASSWORD>, <------ This will not be logged
      "topic.prefix": <TOPIC\_NAME\_PREFIX>,
   }
}'
Post execution , the log looks like below -


connection.password = null
 

Option 2:

This is based on the Kafka - KIP-297 . The idea is to make a Custom ConfigProvider which would be used for masking.

  • Set up a Credential file e.g.    customdetails/user_details.properties

USER\_NAME="user1"
USER\_PASSWORD="<jdjjdjfjdwuwujdjkfkfkf>"
  • Add these properties to connect-distributed.properties file.

config.providers='file'
config.providers.file.class='org.apache.kafka.common.config.provider.FileConfigProvider'
  • Restart the Kafka Connect Worker
 

  • Now in the connector config , use the custom details .See example below -
 


 "config":{ 
      "connector.class": <CONNECTOR\_CLASS\_NAME> ,
      "connection.url": <JDBC\_URL>,
      "connection.user": "${file:/customdetails/user\_details.properties:USER\_NAME}" , <----- This will not be logged
 "connection.password": "${file:/customdetails/user\_details.properties:USER\_PASSWORD}" , <------ This will not be logged
      "topic.prefix": <TOPIC\_NAME\_PREFIX>,
   }
  Hope this helps you How To Mask - Confidential Info in Kafka Connect Logs.   Additional Read -

 

Mask - Confidential Info in Kafka Connect Logs, kafka data masking, confluent kafka tutorial, ksql, confluent kafka query, ksql use cases, ksql stream processing, kafka community, ksql recipes, confluent kafka tutorial, kafka filter, kafka streams, ksql, confluent kafka query, ksql filter, kafka community, ksql concat, Mask data, Data Masking in Kafka, Data Masking