Friday, July 23, 2010

Messaging Basics - Reliability

There does generally seem to be confusion amongst users about how reliable messaging works, so  I thought it would be worth going through some messaging basics again. Firstly, assume you use a hub and spoke architecture, with a broker(s) in the middle to provide location transparency and handle load for you. There are going to be a number of network hops between your Message Producer and Message Consumer:

Everything is asynchronous, we have location transparency, everything is very, very fast. But, it isn't reliable. If you care about message delivery you have to think about where things may go wrong, and even though the odds are in your favour that your messaging application will work 99.99% of the time, its always the time that it fails that you should design for.  The questions you need to think about are:

  • what happens if the broker runs out of resource (memory/sockets/disk space)?
  • what happens if my clients loose network connectivity?
  • What happens if the hard disks crash on the broker's machine
  • what happens in a power outage ?
  • etc, etc.

Depending on the size of the message and the quality of the network, a  good producer can send tens of messages per millisecond - and it can take sometime (seconds or longer) before your producer realizes there's a problem - your producer client could have just lost thousands of messages and your application wouldn't even know about till its too late. So unless you have smarts built into the client to handle failure conditions (like ActiveMQ does), using asynchronous message publish to a broker isn't always the best thing to do. Its for this reason that the majority of message queue systems support a synchronous publish to the broker, so they can detect failure and resend the message. If you look at JMS, beanstalkd,Gearman, MQTT or STOMP synchronous publish to the broker is on by default. Its a sad fact that AMQP hasn't supported this basic messaging construct before 1.0 of the specification, where at least its optional. Its for this reason the ActiveMQ has not adopted AMQP till this point - you cannot have a JMS compliant product without synchronous message publish.  JMS is rightly strict about how the semantics work  - in that a receipt would not be returned to the publisher until the message has been received by the store and securely written to storage if its persistent - i.e. guaranteed delivery:


 For performance, you could cache messages, and write them to disk lazily if they haven't been consumed, however the very real downside is that the broker could disappear before the message has been stored. Your application publisher would believe the message was delivered, but this wouldn't actually be true if the broker disappeared.
BTW - some people confuse transactions with guaranteed delivery - for messaging, transactions should be used for logically grouping messages together. If you use JMS, never use a transaction to send a single message - its unnecessary.

I'm going to be talking about reliability and performance at an upcoming FuseSource Webinar next week - drop by if you have the time ;)

    8 comments:

    alexis said...

    Rob,

    I'm a bit lost. Are you saying that an AMQP broker cannot provide reliable delivery?

    alexis

    Rob Davies said...

    Alexis,

    As you know - AMQP can provide reliability by the use of transactions. I'm not sure why AMQP before 1.0 (and there are no implementations of this yet) didn't support synchronous send. However, most applications don't use transactions, they just assume reliability of messaging - a fair assumption as most messaging systems by default use a synchronous publish to the broker. What's worse is that AMQP has confused transactions with guaranteed delivery - when in fact it should be used to logically group messages together.

    alexis said...

    Rob

    Thanks for your reply. I appreciate your acknowledgement that AMQP is capable of providing reliability. Indeed this is implemented in both RabbitMQ and Qpid.

    I don't think AMQP confuses transactions with guaranteed delivery -- it's very explicit that they are groups of actions. And, a synchronous ack does require grouping the publication with the ack. In AMQP 0-8 and 0-9-1 this is achieved by explicitly using a transaction.

    In AMQP 0-10 there is an additional synchronous publish which is simpler in some cases.

    If the only thing stopping you implementing AMQP 0-9-1 is that you want to see an explicit, non-transaction scoped, ack on publish, then we invite you to help us define it. We would love to work with you to add it to the spec.

    Best wishes

    alexis

    Rob Davies said...

    Hi Alexis,

    The point I was trying to make that by default (and most users just use defaults) - you don't have reliability with out a synchronous publish.

    Right now we are looking at implementing the AMQP 1.0 spec at some point - along with some other open protocols.

    thanks,

    Rob

    Anonymous said...

    Hi Rob

    I am very excited about:

    "Right now we are looking at implementing the AMQP 1.0 spec at some point - along with some other open protocols."

    how far away is it?
    (also will it also have "bindings" for client platforms other than Java - i.e. .NET/.NETCF is of most interest here ;)


    cheers,
    Oleg Kiorsak
    http://www.linkedin.com/in/kiorsak


    PS: I did not mean to post as "Anonymous", but I don't like the idea of typoing in my gmail credentials all other the place (esp. when Safari could not verify the HTTPS connection)

    Kshitiz said...

    Hi Rob,

    I am trying to archive/store all the JMS messages that pass through ActiveMQ, even after their delivery to consumers.

    How can I do it? I have tried persistent="true" and also archiveDataLogs="true" but nothing seems to work !

    Many thanks,
    Kshitiz

    Kshitiz said...

    Hi Rob,

    We are trying to store all the messages that are passed through ActiveMQ, even the delivered ones.

    We have tried persistent="true" and archiveDataLogs="true", but nothings seems to work.

    please advice!

    Many thanks,
    Kshitiz

    Rob Davies said...

    Messages are usually always deleted after they have been delivered and acknowledged. However - if you are using KahaDB you can enable archiving which will store deleted messages in a separate directory.

    cheers,

    Rob