DevOps | Cloud | Analytics | Open Source | Programming





How to Stream Data with Azure Event Hubs from Kafka (Java) ?


In this post, we will see How to Stream Data with Event Hubs using Kafka Protocol. Both Azure Event Hubs and Apache Kafka a data streaming platform and event ingestion service But Event Hubs comes without on-premise option. Azure Event Hubs for Kafka supports Apache Kafka version 1.0 and later.

Azure Event Hubs Kafka endpoints enable users to connect to Azure Event Hubs through the Kafka protocol. Users can connect to Azure Event Hub ecosystem by making some changes to the Kafka application. and reap the benefits of the Azure ecosystem. We will explain How to do this using Java. Also check at the end for details regarding error that might come while using this.  

Using Java as Client Language:

Before proceeding, ensure that you have the below things in place -

  • Azure Account
  • Java JDK 1.7+
  • Maven Binary
  • Git
 

Step 1:

  • Create an Azure Event Hubs namespace and then Kafka endpoint for the namespace will be automatically enabled.
  • Subsequently you can stream events into event hubs from the applications using Kafka protocol.
 

Step 2:

  • Go to - https://github.com/Azure/azure-event-hubs-for-kafka
  • Clone the repo
 

Step 3: Producer Side

  • Change the Producer configs
  • Go to below dir in the Repo.
azure-event-hubs-for-kafka/quickstart/java/producer/src/main/resources/producer.config

  • This opens up the config. Note the specs of the config.
    • bootstrap.servers: Cluster endpoint.
    • security.protocol: Protocol used to communicate with brokers. Here we use SASL_SSL. SASL for authentication and SSL for encryption.
    • sasl.mechanism: Indicates how we authenticate. Using "PLAIN", we use a regular username/password based authentication process.
    • sasl.username: The username to use. For Confluent Cloud, it will be an API key mechanism, where the username is the API key, and the password is the API key’s secret.
    • sasl.password: As per above, this is the API key’s secret.
 

  • From the config you would require to modify - to the your actual namespace as shown below.

bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL\_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="**<your\_Event\_Hubs\_namespace>**";
  One example of actual config settings


bootstrap.servers=dev-we-eh-feed.servicebus.windows.net:9093
client.id=mirror\_maker\_producer
request.timeout.ms=60000
sasl.mechanism=PLAIN
security.protocol=SASL\_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://dev-we-eh-feed.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=\*\*\*\*\*\*";
  OAuth:


bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL\_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
sasl.login.callback.handler.class=CustomAuthenticateCallbackHandler;
 

  • As a next step, you can execute the producer side code, let it run and start streaming events to Azure Event Hubs.
 

Step 4:  Consumer Side

  • Change the Consumer configs
  • Go to below dir in the Repo. azure-event-hubs-for-kafka/quickstart/java/consumer/src/main/resources/consumer.config
 

  • This opens up the config. Note the specs of the config.
    • bootstrap.servers: Cluster endpoint.
    • security.protocol: Protocol used to communicate with brokers. Here we use SASL_SSL. SASL for authentication and SSL for encryption.
    • sasl.mechanism: Indicates how we authenticate. Using "PLAIN", we use a regular username/password based authentication process.
    • sasl.username: The username to use. For Confluent Cloud, it will be an API key mechanism, where the username is the API key, and the password is the API key’s secret.
    • sasl.password: As per above, this is the API key’s secret.
 

  • From the config you would require to modify - to the your actual namespace.

bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL\_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="**<your\_Event\_Hubs\_namespace>**";
  For OAuth:


bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL\_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
sasl.login.callback.handler.class=CustomAuthenticateCallbackHandler;
 

  • As a next step, you can execute the producer side code, let it run and start streaming events from event hub using your Kafka clients
   

Measures to Handle Failures:

**To handle issues relating to connection, you could check the below pointers - **

  • Check whether Azure Eventhub is created in Basic plan Event Hub or Standard plan Event Hub. The Basic plan does not support interaction via Kafka protocol. You might have to upgrade to a Standard plan Event Hub.
  • SharedAccessKey gives error most of the time
  • Authentication Failure due to invalid credentials with Kafka brokers older than 1.0.0
  • Firewall blocking Kafka TLS traffic - restriciting to ONLY HTTPS traffic
  • Transient network issue. (org.apache.kafka.clients.NetworkClient:759)
   

Other Interesting Reads -

   


azure event hub kafka ,azure event hub kafka connect ,azure event hub kafka limitations ,azure event hub kafka surface ,azure event hub kafka configuration ,azure event hub kafka version ,azure event hub kafka pricing ,azure event hub kafka broker ,azure event hub kafka github ,azure event hub kafka connection string , ,azure event hub data stream kafka ,azure event hub with kafka ,azure event hubs for apache kafka ,azure event hub streaming ,azure event hub for kafka ,azure event hub and stream analytics ,azure event hub spark streaming ,is azure event hub based on kafka ,azure event hub data replication ,event hub vs stream analytics ,azure event hub kafka connect ,azure event hub kafka limitations ,azure event hub kafka enabled ,azure event hub kafka python ,azure event hub kafka configuration ,azure event hub kafka port ,azure event hub kafka consumer group ,azure event hub kafka mirrormaker ,azure event hub kafka api ,azure event hub vs apache kafka ,azure event hubs for apache kafka pricing ,azure event hub kafka broker ,azure event hub kafka compatibility ,azure event hub kafka c# ,azure event hub vs confluent kafka ,azure event hub kafka example ,azure event hubs for kafka ecosystems ,azure event hub kafka java example ,azure event hub kafka github ,azure event hub vs hdinsight kafka ,azure event hubs kafka interface ,azure event hub kafka java ,kafka integration with azure event hub ,is azure event hub kafka ,azure event hub or kafka ,azure event hub kafka protocol ,azure event hub kafka pricing ,azure event hub vs kafka performance ,azure event hub kafka schema registry ,azure event hub kafka streams ,azure event hub kafka spring ,azure event hub kafka topic ,azure event hub kafka tutorial ,azure event hub kafka timeout ,azure event hub uses kafka ,does azure event hub use kafka ,azure event hub kafka version ,azure event hub vs kafka ,kafka and event hub ,azure event hubs connector for apache spark ,azure event hubs from apache kafka ,azure event hub kafka ,azure event hub kafka connect ,azure event hub kafka limitations ,azure event hub kafka surface ,azure event hub kafka configuration ,azure event hub kafka version ,azure event hub kafka pricing ,azure event hub kafka broker ,azure event hub kafka github ,azure event hub kafka api ,azure event hub vs apache kafka ,azure event hubs for apache kafka pricing ,difference between azure event hub and kafka ,difference between azure event hub and apache kafka ,spring boot azure event hub kafka ,is azure event hub based on kafka ,azure service bus vs event hub vs kafka ,azure event hub kafka consumer ,azure event hub kafka consumer group ,azure event hub kafka c# ,azure event hub vs confluent kafka ,confluent kafka azure event hub ,does azure event hub use kafka ,is azure event hub kafka ,why use azure event hub ,azure event hub kafka endpoint ,azure event hub kafka enabled ,azure event hub kafka java example ,azure event hubs for kafka ecosystems ,azure event hub for kafka ,kafka connector for azure event hub ,azure event hub vs hdinsight kafka ,azure event hubs kafka interface ,azure event hub kafka java ,azure event hubs for apache kafka ,azure event hub kafka mirrormaker ,azure event hub or kafka ,azure event hub kafka protocol ,azure event hub kafka python ,azure event hub kafka port ,azure event hub vs kafka performance ,azure event hub kafka schema registry ,azure event hub kafka streams ,azure event hub kafka spring ,kafka connect azure event hub sink ,azure event hub kafka topic ,azure event hub kafka timeout ,azure event hub kafka tutorial ,kafka connect to azure event hub ,azure event hub uses kafka ,azure event hub vs kafka ,azure event hub with kafka ,azure event hub kafka connection string ,azure event hub kafka connector ,kafka integration with azure event hub ,azure event hub exactly once ,azure event hub partition example ,azure event hub costing ,azure event hub ppt , ,azure event hub kafka java example ,azure event hub kafka kafka ,azure event hub kafka limitations ,azure event hub kafka mirrormaker ,azure event hub kafka node ,azure event hub kafka not working ,azure event hub kafka port ,azure event hub kafka pricing ,azure event hub kafka protocol ,azure event hub kafka python ,azure event hub kafka query ,azure event hub kafka questions ,azure event hub kafka queue ,azure event hub kafka schema registry ,azure event hub kafka spring ,azure event hub kafka streams ,azure event hub kafka surface ,azure event hub kafka timeout ,azure event hub kafka topic ,azure event hub kafka tutorial ,azure event hub kafka version ,azure event hub kafka xampp ,azure event hub kafka yaml ,azure event hub kafka yml ,azure event hub kafka youtube ,azure event hub kafka yum ,azure event hub kafka zabbix ,azure event hub kafka zip ,azure event hub kafka zip file ,azure event hub kafka zookeeper ,azure event hub or kafka ,azure event hub uses kafka ,azure event hub vs apache kafka ,azure event hub vs confluent kafka ,azure event hub vs hdinsight kafka ,azure event hub vs kafka ,azure event hub vs kafka performance ,azure event hub with kafka ,azure event hubs for apache kafka ,azure event hubs for apache kafka pricing ,azure event hubs for kafka ecosystems ,azure event hubs kafka interface ,azure hdinsight kafka vs event hub ,azure service bus vs event hub vs kafka ,confluent kafka azure event hub ,difference between azure event hub and apache kafka ,difference between azure event hub and kafka ,does azure event hub use kafka ,how azure event hub works ,is azure event hub based on kafka ,is azure event hub kafka ,kafka connect azure event hub sink ,kafka connect to azure event hub ,kafka connector for azure event hub ,kafka mirrormaker azure event hub ,kafka to azure event hub ,spring boot azure event hub kafka ,what is event hub in azure ,what is event hub in kafka ,why azure event hub ,why use azure event hub ,azure event hub and stream analytics ,azure event hub data stream kafka access ,azure event hub data stream kafka application.yml ,azure event hub data stream kafka automation ,azure event hub data stream kafka avro ,azure event hub data stream kafka backend ,azure event hub data stream kafka binder ,azure event hub data stream kafka body ,azure event hub data stream kafka build ,azure event hub data stream kafka configuration ,azure event hub data stream kafka connect ,azure event hub data stream kafka consumer ,azure event hub data stream kafka dependency ,azure event hub data stream kafka describe ,azure event hub data stream kafka documentation ,azure event hub data stream kafka download ,azure event hub data stream kafka example ,azure event hub data stream kafka filter ,azure event hub data stream kafka flow ,azure event hub data stream kafka github ,azure event hub data stream kafka group ,azure event hub data stream kafka guide ,azure event hub data stream kafka handle ,azure event hub data stream kafka handler ,azure event hub data stream kafka header ,azure event hub data stream kafka health ,azure event hub data stream kafka include ,azure event hub data stream kafka interview ,azure event hub data stream kafka issues ,azure event hub data stream kafka java ,azure event hub data stream kafka job ,azure event hub data stream kafka js ,azure event hub data stream kafka json ,azure event hub data stream kafka kafka ,azure event hub data stream kafka keys ,azure event hub data stream kafka knowledge ,azure event hub data stream kafka library ,azure event hub data stream kafka list ,azure event hub data stream kafka load ,azure event hub data stream kafka load balancer ,azure event hub data stream kafka login ,azure event hub data stream kafka marketplace ,azure event hub data stream kafka maven ,azure event hub data stream kafka message ,azure event hub data stream kafka metadata ,azure event hub data stream kafka metrics ,azure event hub data stream kafka model ,azure event hub data stream kafka net ,azure event hub data stream kafka offset ,azure event hub data stream kafka producer ,azure event hub data stream kafka python ,azure event hub data stream kafka que es ,azure event hub data stream kafka query ,azure event hub data stream kafka questions ,azure event hub data stream kafka report ,azure event hub data stream kafka rest ,azure event hub data stream kafka retry ,azure event hub data stream kafka run ,azure event hub data stream kafka sample ,azure event hub data stream kafka stream ,azure event hub data stream kafka topic ,azure event hub data stream kafka transaction ,azure event hub data stream kafka trigger ,azure event hub data stream kafka tutorial ,azure event hub data stream kafka ui ,azure event hub data stream kafka variable ,azure event hub data stream kafka version ,azure event hub data stream kafka workflow ,azure event hub data stream kafka x4 ,azure event hub data stream kafka xls ,azure event hub data stream kafka xml ,azure event hub data stream kafka yaml ,azure event hub data stream kafka youtube ,azure event hub data stream kafka zip ,azure event hub data stream kafka zookeeper ,azure event hub for kafka ,azure event hub partition example ,azure event hub spark streaming ,azure event hub streaming ,azure event hub with kafka ,azure event hubs for apache kafka ,difference between azure event hub and kafka ,difference between event hub and kafka ,event hub to kafka ,is azure event hub based on kafka ,why azure event hub