Saturday, February 03, 2007

Apache ActiveMQ: Version 5.0 Broker Changes

ActiveMQ has now graduated out of the incubator, which is going to enable us to get on track on releasing more often.

We are currently going through a stage of consolidation of features to ensure we have a solid foundation to build on going forward, and these features will be available on 5.0 - the first milestone release will be in the next couple of weeks.

New features will include:

New dispatch model for Persistent messages.

Prior releases to 5.0 have kept references for all the messages that could be dispatched to an active Durable Topic Consumer or a Queue in memory. Whilst a reference itself was not large, it did impose a limit on the maximum number of messages that could be pending delivery.

A typical approach for messaging systems dispatching persistent messages is to pull them in batches from long term storage when a client is ready to consume them, using a cursor to maintain the next to dispatch position. This is a robust and very scalable approach, but not the most performant for cases when the consumer(s) can keep up with the producer(s) of messages.

Hence ActiveMQ 5.0 has a hybrid approach, allowing messages to pass from producer to consumer directly (after the messages have been persisted) but switches back to using cursors if the consumer(s) fall behind.

New Default Message Store

Hiram and I have been working on different storage options for a while now, but we've finally settled on an implementation that uses the best bits of the journal and Kaha to provide a storming messaging persistent engine, which scares the pants of me - its sooo fast!

Slow Consumers


Version 5.0 will a combination of outstanding messages and memory consumption to decide when a consumer can be sent more messages (message prefetch). This will allow us to determine when consumption is slow, regardless of the message size.

Spooling to disk for non-persistent slow consumers has also been optionally introduced. There is some more tuning thats required to be done to get this right, but should be ready this week.


Other features that will be incorporated in the 5.x release:

* automatic recovery for the 'shared nothing' master/slave (about to start on this)
* bi-directional network bridges

2 comments:

robottaway said...

Congrats on the graduation! I'm glad to hear that your group is working on the persistence of durable subscriptions. I think that will solve some of the production issues I have had with your product. Thanks for all the hard work.

regards,
Rob O.

Nigel said...

Rob

Well done on the sale - now get on with some serious blogging (and drinking; those Iona guys are based in Dublin after all...).

Cheers Nigel