I’m currently working on a project where I need to be able to transfer a large number of requests via JSON over web services. I need to take some of that data, do some aggregation with it, and store it in a persistent store. In order to allow the data to be reliably processed in a number of different ways, I wanted to place the incoming data into multiple queues and have it processed and then stored. Something like this:
At this point you might be wondering why I’m not using MSMQ, since most of the work I do is on Windows. Well, a few different reasons. First and foremost, I am running all of my infrastructure inside of EC2, so I want the option of running the queues on a Linux box. Secondly, because the web service front end is a very thin layer that simply transforms the incoming JSON into message to drop on a queue, I want the option to run those in whatever language I want and be confident that I will have no issues connecting to the queues. Third, I want to be able to persist or not persist messages to disk and have my queues operate entirely in memory or on disk. This is especially useful in EC2 where hard disk performance is somewhat lacking.