Building an Apache Kafka Messaging Consumer on Bluemix

The Message Hub service on Bluemix is based on Apache Kafka, which is a fast, scalable, and durable real-time messaging engine. For developers there are Java, Node and REST APIs to leverage Kafka. Yesterday I blogged about how to write a simple Kafka producer in Node.js. This article covers the counterpart – a Kafka consumer.

To understand how consumers work in Kafka I recommend reading the Kafka documentation. Essentially you define consumer groups with potentially multiple consumer instances. Only one instance per group receives one particular message. In order to scale you can simply increase the number of consumer instances. In Kafka each container instance has it’s own partition from which it receives messages for a certain topic (in the right order). However since the Message Hub only supports one partition in the current beta status, you cannot use this functionality yet.

The consumer code below is again a slightly modified version of the sample Node.js chat application from my colleague Niall Weedon. I separated the producer from the consumer code and removed the chat code.

The consumer instance is created in line 87 by passing in the consumer group name, the consumer instance id (maps to Bluemix Cloud Foundry application instance_id) and an offset to define which messages to read. The code in lines 66ff shows how the consumer actually polls for updates on a certain interval and reads the messages (as batch) in line 69. To make sure that other consumer instances receive messages when a particular consumer instance goes down, you need to tell this Kafka in line 125.

You can run the consumer either locally or on Bluemix. To run it on Bluemix, create a Bluemix Node.js application, add the Message Hub service and execute these commands from the project’s root directory.

cf login
cf push <mykafkaconsumer>

To run the consumer locally, execute these commands.

npm install
node app.js <message_hub_rest_endpoint> <message_hub_api_key>

In order to test the consumer you need to also run the producer as explained in the previous article. Both need to use the same Message Hub instance. Here is a screenshot of the producer and consumer running locally.