Overview
If you’ve been following my blog, you’re probably well aware of my penchant for node.js and the few blog posts I’ve already posted on the subject. More recently, I wanted to explore RabbitMQ, a messaging platform similar to JMS, that can be easily leveraged with node.js as an alternative to saving messages with HTTP POST using a REST interface. In this blog post, I’m going to extend the DogTag application I blogged about in DogTag App in 30 minutes – Part 1: Node.js and MongoDB and show you how you can use RabbitMQ to push messages into MongoDB using Node.js on the server and a simple Groovy client as the publisher of those messages.
What is RabbitMQ?
RabbitMQ provides robust messaging that is easy to use and supported by a large number of developer platforms as well as operating systems. Its low barrier to entry makes it quite suitable to use with node.js as you will be amazed at the remarkable little code that is required to establish a RabbitMQ exchange hosted within a node.js application. RabbitMQ is similar to JMS in that it supports many of the same paradigms you may have seen in this or other messaging platforms (e.g., Queues, Topics, Routing, Pub/Sub, RPC, etc.). Primary language support includes Python and Java, but there are so many others supported as well.
In this blog post, I’ll be using a NPM package aptly named node-amqp. AMQP stands for “Advanced Message Queuing Protocol”. AMQP is different from API-level approaches, like JMS, because it is a binary wire-level protocol that defines how data will be formatted for transmission on the network. This results in better interoperability between systems based on a standard supported through the Oasis standards body. AMQP started off as a beta but released a 1.0 version in 2011 and has since grown in popularity (see here for comparison).
RabbitMQ and Node.js
In example I’m about to show you, I’m basically providing a simple means to batch several messages to save new “dog tags” in the DogTags application I referenced earlier. I am going to post the messages to a queue and have the server save them into the database. Let’s take a look at the JavaScript code necessary to do this in node.js:
var amqp = require('amqp');
var conn = amqp.createConnection({url: mq_url});
conn.on('ready', function () {
var exchange = conn.exchange('queue-dogtag', {'type': 'fanout', durable: false}, function () {
var queue = conn.queue('', function () {
console.log('Queue ' + exchange.name + ' is open');
queue.bind(exchange.name, '');
queue.subscribe(function (msg) {
console.log('Subscribed to msg: ' + JSON.stringify(msg));
api.saveMsg(msg);
});
});
queue.on('queueBindOk', function () {
console.log('Queue bound successfully.');
});
});
});
On lines 1-2, you can see how I’m including the amqp package from NPM and establishing a connection with a provided URL (e.g., amqp://guest:guest@localhost:5672). Once the connection is established we create what’s called an exchange on line 4. Here you define the type of exchange, the name and whether it is durable or not. For further details on this and other settings, I will refer you to the docs here. You may be asking why not just send directly to a queue? While this is possible, it’s not something that is practiced given the messaging model of RabbitMQ; namely, that you should never send any messages directly to a queue. Instead, the producer should send messages to an exchange which knows exactly what to do with them. In my case, I chose “fanout” as my exchange type, which simply broadcasts all the messages to all the queues it is aware of, which in my case is just the one queue defined on line 5. Other exchanges you may want to look into include Direct, Headers and Topic exchanges which are described in detail here.
Once I have defined the exchange and the queue, it is time to bind the queue to the exchange name and subscribe to any messages published to the queue. I do this on line 8-11. It is then a simple matter of processing each message and using the api object I’ve established to save the message to MongoDB and to record an appropriate log message. As an added measure, I also think it is worthwhile to see if the queue is bound to the exchange prior to any processing and report the results the log. I do this on lines 14-16.
The Groovy Publisher
So now that we have the server-side component in place ready to subscribe to any messages, we need to create a publisher to publish messages to the exchange. To do this, I chose to write a CLI tool in Groovy and leverage Spring MQ to publish the message. Groovy is great language to create CLI tools and since it is maintained by the folks at SpringSource, it was just a natural fit to use the SpringMQ framework which can generate AMQP messages without much fuss. Of course, as a prerequisite, you’ll have to grab the latest Groovy binaries so you can run the script. Let’s take a look at the code:
@Grab(group = 'org.springframework.amqp', module = 'spring-amqp', version = '1.1.1.RELEASE')
@Grab(group = 'org.springframework.amqp', module = 'spring-rabbit', version = '1.1.1.RELEASE')
@Grab(group = 'com.rabbitmq', module = 'amqp-client', version = '2.8.7')
@Grab(group = 'org.codehaus.jackson', module = 'jackson-mapper-asl', version = '1.9.9')
import org.springframework.amqp.rabbit.connection.*
import org.springframework.amqp.rabbit.core.RabbitTemplate
import org.springframework.amqp.support.converter.JsonMessageConverter
class RabbitMqUploader {
static void main(args) {
println("Starting...")
def cli = new CliBuilder(usage: 'uploader.groovy [filename.csv]')
def factory = new CachingConnectionFactory();
factory.setUsername("username");
factory.setPassword("password");
factory.setVirtualHost("ve2b460e7e6b24b5da88c935ff63c4e86");
factory.setHost("hostname")
factory.setPort(10001)
RabbitTemplate templ = new RabbitTemplate(factory);
templ.setExchange("queue-dogtag");
templ.setMessageConverter(new JsonMessageConverter());
def options = cli.parse(args)
if (!options) {
// Means no parameters were provided
cli.usage()
return
}
def extraArguments = options.arguments()
def filename = ''
if (extraArguments) {
if (extraArguments.size() > 0) {
filename = extraArguments[0]
}
//load and split the file
new File(filename).splitEachLine("\n") { row ->
def field = row[0].split(',')
def dt = new DogTag()
dt.name = field[0]
dt.description = field[1]
dt.longitude = field[2]
dt.latitude = field[3]
dt.phone = field[4]
templ.convertAndSend(dt)
println('Sent >>>' + row)
}
}
factory.destroy()
}
}
So after our discussion in the previous section, you can follow the code here to see how simple it is to publish a message to the RabbitMQ exchange we set up. One thing I simply love about Groovy is that I don’t need to grab any dependencies or build any jars or define any classpaths. For dependencies, I simply use the @Grab annotation to pull those required for SpringMQ, RabbitMQ and the JSON parser down to the client so that they can run the groovy script without any set-up. In lines 15-20, I set-up the configuration to point to my host (which I recommend using a free aPaaS like OpenShift or Nodejitsu to host your Node.js app). Lines 22-24, I define the RabbitMQ template based on the config, set the exchange name and establish a JSON converter since MongoDB likes JSON format. Finally, in lines 34-55, I parse a CSV file provided as a command line parameter and send each line as an AMQP message to our server.
To run this program, I simply run: groovy RabbitMqUploader.groovy ./dogtags.csv.
Conclusion
With a few simple changes to my original application, it was easy to augment it to support RabbitMQ and define an exchange to consume messages. With Groovy, it was easy to write an autonomous, concise client to establish a connection to my node.js server and broadcast messages over the AMQP wire protocol for the server to process. I expect to use this approach in the future where perhaps a REST interface may not be best suited and using well established messaging paradigms make more sense. I encourage you to explore this interesting messaging protocol for your future needs as well.
