Rob Davies on Open Source Integration

Apache ActiveMQ Apollo 1.0 is released!

2012-02-03T06:43:00.000-08:00

Apache ActiveMQ Apollo 1.0 has gone GA. Hiram Chirino does a great summary of features and functionality on his blog.

About 2 years ago, FuseSource started to think about the requirements of enterprise messaging systems for the next 10-15 years. There has been an exponential growth in the amount of data used by organisations over the past 25 years, and that trend is set to continue for the next 25. Once applications held megabytes of data, now they can hold petabytes (10^15). Within 5 years that will be zetabytes (10^21). Speed of accessing that data is increasing, especially with the use of BigData (Apache Hadoop, Cassandra, MongoDB etc) - and so is the requirement for filling data stores with the vast volumes of data required.
Message-oriented middleware is all about transferring data reliably and as quickly as possible from one application to another. So it made sense for us to look at the requirements of the next generation of ActiveMQ with that in mind. We realised we would need to provide the following:

Extreme throughput
Scale to millions of dynamic destinations for both point-2-point (Queues) and publish/subscribe (Topics).
Scale to 100,000 connections or more
Extensive protocol support (STOMP/OpenWire/MQTT/AMQP).

Apache ActiveMQ is performant, and provides extensive capabilities - but in its current architecture, its not going to meet the performance and scaling we required. ActiveMQ 5 is really stable - after being deployed in real world production environments, its become very battle hardened, its not something we radically want to change. However, if you examine some of the internals of the current broker, you realise that there are too many points of synchronisation for ActiveMQ to scale up for the next generation of messaging problems that we will start to encounter in just a few years time.

It was decided early on that to help with scaling and concurrency, we should use a functional language - which would help reduce side effects (and hence the need for synchronisation), and enable Apollo to maximise the utilisation of multi-core architectures. Currently, the Apollo broker is written in the Scala programming language. At the core of Apollo is the Grand Central Dispatch (GCD) technology to help optimise support for multi-core.

The implementation of GCD, has been implemented by Hiram - and is called HawtDispatch. ActiveMQ Apollo currently supports the STOMP protocol, but it does support JMS - using JMS over STOMP - there are performance comparisons here, which demonstrates Apollo's performance, even using a text based protocol its considerably faster than ActiveMQ 5 - which uses efficient data encoding based on OpenWire.

Right now ActiveMQ Apollo is a very performant message broker, that you should use if raw performance is the overriding criteria of your architecture. Later releases will be adding support for more protocols (MQTT, OpenWire, AMQP) and the enterprise features that will bring it in line with ActiveMQ 5. When this work is complete FuseSource will also provide a hardened supported version of ActiveMQ Apollo.

CamelOne is back!

2012-01-26T23:01:00.000-08:00

FuseSource is hosting the CamelOne event this year in Boston, MA, May 15-18th 2012. Its the best conference to go to to hear about Apache Camel, ActiveMQ, ServiceMix, Karaf and CXF.

Its going to be awesome!

Its also going to be a great opportunity for us to talk about all the really cool, secret technology we've been developing at FuseSource for the past year too :)

To get a flavour of technical content at this event - you can view last year's sessions ...

ActiveMQ IS Ready For Prime Time

2011-06-27T05:23:00.000-07:00

The ActiveMQ project has been around since 2005 - and has been a top level project at the Apache Software Foundation for a good part of that. The aims of the ActiveMQ project has been to provide a world class enterprise messaging solution, with brokers being able to provide connectivity from smart, IP enabled devices all the way through to high availability at the enterprise back end. ActiveMQ provides cross language clients - including Java/C/C++/C#/Perl/PHP/Python etc - for a full list see here. Open connectivity has always been a goal, which why as well as OpenWire and STOMP, ActiveMQ will be supporting AMQP 1.0 when its finalised.

FuseSource ships is own distributions of Apache ActiveMQ (called FuseMQ), Camel, ServiceMix (FuseESB) and CXF. The code base is exactly the same as Apache, however we need to be able to ship mission critical fixes and enhancements within hours for production systems. We also ensure that these distinct projects work well together on different platforms, and put the releases through more extensive system tests. I mention this because some of the case studies that I'll refer to mention FuseMQ - which is is the same code as ActiveMQ - and can be used as a drop-in replacement - its just better tested.

The Java/C++/C# clients support seamless failover if connectivity is lost to a broker - and brokers can be configured in highly available clusters. In addition, ActiveMQ supports connectivity between brokers over wide-area networks, using store and forward network topologies. This means that ActiveMQ is not only used to provide connectivity between remote data centres, but is also being used to provide connectivity between unreliable communications (dial-up, satellites). This is particularly important for large retailers, to provide real time reliable connectivity for order placement, stick tracking and monitoring. There's a case study on SpecSaver's here - but I know some of the largest retailers in the US (as they are FuseSource customers) are also heavily using ActiveMQ for very similar deployments.

Enterprise Messaging has traditionally been used to enable large transactional systems for enterprise deployment. Can ActiveMQ do that ? Yes it can - FuseMQ is deployed as part of FuseESB - see this NY Times article. Messaging is also used for high volume real time updates - can ActiveMQ do that - yep (CERN, RiotGames or JPL). If you want an example of really mission critical deployment of ActiveMQ - look at the FAA case study on how its been utilised in the Next Generation Transport System.

In order for ActiveMQ to be utilised in so many varied deployments, it has to be very flexible, so that it can configured to run in the best way possible for a particular deployment. This asset can also be ActiveMQ's achilles heel. As a developer, I often expect things to just work, and figure out afterwards why they don't, however ActiveMQ is one of those highly configurable tools that does require some upfront knowledge, or background reading (or at least reading the F.A.Q) before you can get the best out of it.

As FuseSource CTO, I see ActiveMQ being a unique foundation stone to the thousands of successful enterprise integration projects we support our many hundreds of enterprise customers on. ActiveMQ is not only ready for prime time, but with the innovation, flexibility and world beating performance being developed in ActiveMQ Apollo, it will continue to dominate the enterprise messaging space for many years to come.

CamelOne - the one event to go to for open source integration

2011-03-30T11:52:00.000-07:00

This is the first ever CamelOne event - designed for users of most popular integration projects, Apache Camel, ServiceMix, ActiveMQ and CXF. Why call it CamelOne? - cause Camel is the integration glue that holds them all together. Get more info here.

ActiveMQ in Action is in Print

2011-03-25T01:01:00.000-07:00

Writing a book is easy - finding the time to do is the hard part. ActiveMQ in Action is now in print - if you are using ActiveMQ in anyway, you really should buy this book - it will save you time and money in the long run.

FuseSource launches beta for Fuse IDE for Apache Camel

2011-01-31T03:15:00.000-08:00

Fuse IDE will eventually come in a number of different flavours, for different aspects of of integration (Web and Restful services, messaging as well as supporting remote configuration). Initially the first flavour will be for Enterprise Integration Patterns, with direct support for defining integration routes within the excellent Apache Camel. James Strachan has already blogged about this, but we will also be adding the ability to debug camel routes in the near future.

At FuseSource we recommend you always start any integration project first with Apache Camel, but if you want high availability, distributed routes or simply a managed container to deploy your integration application in, then deploy your camel routes inside ServiceMix.

The focus for FuseSource in 2011 is building even more value for our users, as well as continuing to extend the envelope for innovation through our Apache integration projects of Camel, ActiveMQ, CXF and ServiceMix.

Making development of integration applications even easier with Eclipse based tooling is just one thread of development we are following, as we build on our success of last year.

FuseSource in Numbers

2010-10-28T05:31:00.000-07:00

1 million+ transactions every minute running on Fuse software
200,000 downloads a month
200+ enterprise customers and growing rapidly
24 hr Enterprise support - every day
20 + Apache committers and contributors
15 + years of Enterprise Integration experience (ok slightly more - but we lie about our age)
#1 provider of Open Source Integration and Messaging

FuseSource software is based on Apache ServiceMix, ActiveMQ, Camel and CXF.

We wrote the code!

Podcast on Apache ActiveMQ and FuseSource

2010-10-28T04:24:00.000-07:00

I recently got the chance to ramble to James Denman from SearchSoa about Apache ActiveMQ, ActiveMQ in Action and a little bit about the FuseSource launch. You can listen to it on the SearchSoa site here. There's also a FuseSource webinar on ActiveMQ in the Enterprise - if you want to get a flavour of some of the topics covered in the ActiveMQ in Action book.

ActiveMQ in Action in Production!

2010-10-27T04:45:00.000-07:00

The ActiveMQ in Action book, that has been a background part of my working life for the past couple of years has finally gone into production. Its actually bang up to date - and covers features that are applicable to the latest Apache ActiveMQ 5.4 releases.

To celebrate - Manning are giving 40% of your purchase of ActiveMQ in Action - just use the coupon code *activemq40* at the time of checkout.

FuseSource has launched!

2010-10-25T04:36:00.000-07:00

FuseSource is now operationally independent - and has been spun out as a separate legal entity from the Progress Software Corporation - to emerge as the world leader in open source integration and messaging solutions for the Enterprise. Although FuseSource will remain a 100% owned subsidiary of Progress Software, it does mean we are bucking the current trend of consolidation in the open source industry and been given the freedom to concentrate on our own destiny.

FuseSource provides enterprise support for its open source core products, Apache ServiceMix, ActiveMQ, Camel and CXF, as well as providing products to better enable production deployment and management around those projects.

We actually started off over 5 years ago, as LogicBlaze - a start-up intent on providing the best open source integration solutions for developers. We had already been working on ActiveMQ and had just started on ServiceMix. We moved both of these projects to Apache, because we wanted to provide truly open solutions for our users and customers. Open source development should never be about simply the src code - its the community that matters, and the Apache Software Foundation simply brings about the best environment for open source. The Apache licence ensures that anyone can use their projects without fear (in contrast to copyleft licences) and the diversity in the community of Apache developers ensures that an Apache project will never be held hostage to the fortune of any single company or individual.

As a VC funded start up LogicBlaze came to the point in 2007 when it needed a new round of funding to grow the business, and at that time, although we had lots of customers, to gain a foot hold into providing enterprise integration for large organisations we really needed to partner with a larger organisation who had the enterprise pedigree to back us up. We had a lot of enterprises looking at our solutions, but they all wanted the assurance that we were still going to be around in 5-10 years before committing to a subscription. This was at a point in time just before open source was seen as a viable option for anything other than operating systems or project based databases for large organisations. A lot has happened in the last few years to change that view, but this was back in the day. This convinced me of two things: The Apache Software foundation was the right place to home open source integration solutions; and we really needed to be part of a larger organisation to start making inroads into large enterprises. Ideally for us, we wanted to be a subsidiary, to be independent, to be able to decide our own sales and marketing strategy, but have the financial clout and backing of a plc.

Actually at that time there weren't many companies we would have been happy to join, and we pretty much convinced that we would have to continue as a venture backed start-up - but we just happened to meet the folks at IONA Technologies - and there was an instant cultural fit. IONA had its own open source story, which ended up being Apache CXF which they were using to supplement their own licence based integration solution. We realised there would be potentially a good technical fit as well as a cultural one - and we were acquired in April 2007.We were looking at the potential of what we could grow together - and with IONA's CXF added to the project portfolio, we created a new way to glue everything together called Apache Camel - which is probably now the most widely used integration glue anywhere - and we were renamed as the Fuse group.

What we didn't realise was the extent of the financial storm that was about to hit the economy, and it 2008 become a very hard time for a lot software corporations, including IONA, and they were acquired by Progress Software in September 2008. I think its fair to say that Progress weren't entirely sure what to do with FUSE and how to market it effectively along side their own products - and Progress owned a lot of products! Progress have been going through a lot of changes in the last 2 years, as they have re-invented themselves away from being the owner of lots of disparate application and data integration solutions, into a cohesive solution for operational responsiveness. I think its fair to say that Progress wasn't sure where Fuse should fit in, and we weren't sure if we should try and fit into the Progress story.

However, some hard facts shone through about the Fuse business. Fuse has managed over 100% growth year on year, has more than 200 large enterprise customers world wide (3 in the top Global 10) and more than 1million downloads of our software a year. Progress realised that Fuse should be seen as a strategic asset, especially with the big inroads we have made into large enterprises in past couple of years - its just that our business and culture is somewhat orthogonal to Progress Software.

I give much kudos to the Progress executive team for deciding the best way forward is for Fuse to become a separate legal entity, but wholly owned subsidiary - known as FuseSource. We have a great management team, exceptionally talented technical team and superb and supportive customers - and we have the full backing and financial support of Progress to continue to grow our business. We have a really innovative roadmap, not just for the Apache projects we contribute to, but also for our Fuse based products - which we will start rolling out through 2011!

Enterprise ActiveMQ Webinar

2010-08-12T04:57:00.000-07:00

I'll be talking about using ActiveMQ in the Enterprise on Thursday 19th August in a live webinar.

As well as covering failover, high availability and scaling - I will also cover features in the ActiveMQ 5.4 and give some insight into the new architecture and features for ActiveMQ Apollo - the messaging system currently being built to meet the high demands of messaging for the next decade.

Messaging Basics - Reliability

2010-07-23T05:51:00.000-07:00

There does generally seem to be confusion amongst users about how reliable messaging works, so I thought it would be worth going through some messaging basics again. Firstly, assume you use a hub and spoke architecture, with a broker(s) in the middle to provide location transparency and handle load for you. There are going to be a number of network hops between your Message Producer and Message Consumer:

Everything is asynchronous, we have location transparency, everything is very, very fast. But, it isn't reliable. If you care about message delivery you have to think about where things may go wrong, and even though the odds are in your favour that your messaging application will work 99.99% of the time, its always the time that it fails that you should design for. The questions you need to think about are:

what happens if the broker runs out of resource (memory/sockets/disk space)?
what happens if my clients loose network connectivity?
What happens if the hard disks crash on the broker's machine
what happens in a power outage ?
etc, etc.

Depending on the size of the message and the quality of the network, a good producer can send tens of messages per millisecond - and it can take sometime (seconds or longer) before your producer realizes there's a problem - your producer client could have just lost thousands of messages and your application wouldn't even know about till its too late. So unless you have smarts built into the client to handle failure conditions (like ActiveMQ does), using asynchronous message publish to a broker isn't always the best thing to do. Its for this reason that the majority of message queue systems support a synchronous publish to the broker, so they can detect failure and resend the message. If you look at JMS, beanstalkd,Gearman, MQTT or STOMP synchronous publish to the broker is on by default. Its a sad fact that AMQP hasn't supported this basic messaging construct before 1.0 of the specification, where at least its optional. Its for this reason the ActiveMQ has not adopted AMQP till this point - you cannot have a JMS compliant product without synchronous message publish. JMS is rightly strict about how the semantics work - in that a receipt would not be returned to the publisher until the message has been received by the store and securely written to storage if its persistent - i.e. guaranteed delivery:

For performance, you could cache messages, and write them to disk lazily if they haven't been consumed, however the very real downside is that the broker could disappear before the message has been stored. Your application publisher would believe the message was delivered, but this wouldn't actually be true if the broker disappeared.
BTW - some people confuse transactions with guaranteed delivery - for messaging, transactions should be used for logically grouping messages together. If you use JMS, never use a transaction to send a single message - its unnecessary.

I'm going to be talking about reliability and performance at an upcoming FuseSource Webinar next week - drop by if you have the time ;)

Extending Apache ActiveMQ with Apache Camel

2010-07-01T04:36:00.000-07:00

Apache Camel - the excellent integration engine - can be embedded in an ActiveMQ broker to turn it into a full integration hub. Camel has probably more integration connectors with 3rd party apps than any other ESB - couple that with its ease of use and you get an understanding of why its so popular!

Camel actually started off as a sub-project of ActiveMQ - so the integration between the two is excellent (although it will be even better in ActiveMQ 6).

I blogged a while ago about the new Statistics Plugin - that you can use to request a message from the ActiveMQ broker about its running statistics (just in case you don't want to use JMX). Now it would kinda nice to automatically send these messages out from the ActiveMQ message broker periodically. Well its really easy to do that ...

Firstly enable the statics plugin in the ActiveMQ broker configuration and also import a camel config - e.g.

<beans>
   <broker brokerName="testBroker" xmlns="http://activemq.apache.org/schema/core">
   <transportConnectors>
   <transportConnector uri="tcp://localhost:61616"/>
   </transportConnectors>
  </broker>
  <import resource="camel.xml"/>
</beans>


Then in the camel XML configuration define an embedded ActiveMQ Connection -

<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
   <property name="connectionFactory">
   <bean class="org.apache.activemq.ActiveMQConnectionFactory">
   <property name="brokerURL" value="vm://testBroker?create=false&waitForStart=1000" />
   <property name="userName" value="DEFAULT_VALUE"/>
   <property name="password" value="DEFAULT_VALUE"/>
   </bean>
   </property>
</bean>

Note: the vm:// transport (uses a broker's name to locate it in the same JVM)

Then just add a Camel route in the Camel configuration which uses a Timer to request statistics from the broker and publish them on a Topic every second - e.g:

<route>
   <from uri="timer://foo?fixedRate=true&period=1000"/>
   <inOut uri="activemq:queue:ActiveMQ.Statistics.DestinationTest.Queue"/>
   <to uri="activemq:topic:Statistics.Topic"/>
</route>

Simple!

We are going to be on the road doing the popular Fuse days again from September - but in the meantime - you can find out more about Apache Camel, ActiveMQ and ServiceMix from our Fuse webinars.

ActiveMQ 5.4: JMS Timer and CRON Scheduler

2010-02-16T01:07:00.000-08:00

I've added a persistent scheduler engine to ActiveMQ, so that messages that are scheduled for delivery will always survive restarts. The scheduling of some message in the future can be useful - but deciding a scheduling syntax that is flexible, but can also fit into Message header properties is a little tricky. I've decided the best approach is to use a combination of different concepts - simple delay, repeats and CRON.

There are a few choices of around scheduling messages to deliver in the future(inspiration from java.util.Timer) - a simple delay - this one will deliver the message in a minute:

long time = 60 * 1000;
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, time);

producer.send(message);

Delivery at a fixed rate - this example will delivery 10 messages, waiting 10 seconds between each message - note the repeat count is 9 (9 + first one = 10).
long delay = 30 * 1000;
long period = 10 * 1000;
int repeat = 10;
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, delay);
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_PERIOD, period);
message.setIntProperty(ScheduledMessage.AMQ_SCHEDULED_REPEAT, COUNT repeat);
producer.send(message);

For more advanced messaging you can use CRON scheduling - this one will deliver a message at 2AM on the 12 day of every month:

message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, "0 2 12 * *");
producer.send(message);

You can combine delays and repeats with CRON. For example, you may wish to send 10 messages, every 30 minutes, with a delay of 10 seconds between each one.

message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, "30 * * * *");
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, 10*1000);
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_PERIOD, 10*1000);
message.setIntProperty(ScheduledMessage.AMQ_SCHEDULED_REPEAT, 9);

As I quickly realised whilst testing - having a persistent message scheduler that survives re-starts really needs management support - so that you can delete those pesky jobs that you accidentally scheduled ;). So there's JMX support - and the ActiveMQ Web Console has been updated too:

A funny thing happened to my ActiveMQ Performance

2010-02-14T22:24:00.000-08:00

In ActiveMQ we have a message cursoring system to ensure that there is no memory restriction on the size of the queues we use. However, we do cache messages to improve performances - as depicted below:

The cursor cache is limited by memory constraints that are defined in the ActiveMQ configuration. Usually the cache is valid, but if memory constraints hit, - or consumers lag to far behind producers, the cache is cleared down. However for every message dispatched through the cursor after the cache is cleared, the cursor will check if the Queue is empty in the store to determine if it can start using the cache again.

This all sounds eminently sensible. However, the code to check if the Queue was empty in the store called for a count on the outstanding messages. In order to determine the size of the Queue, the BTree index used for KahaDB would iterate through every BTree node and, er count them. ActiveMQ would do this for every message dispatched after the cache became invalid.

Just to labour the point, for every message dispatched, every BTree node for the BTree index would be paged into memory in the same transaction and counted ....

Although this is an issue with some serious side-affects - like being a memory hog when using KahaDB and seriously affecting performance - it does highlight that for distributed systems there are always going to be behaviour that you can't always anticipate.

The reason I found this issue so interesting is because it wasn't obvious - its an edge case that happens under only under certain load conditions with certain configurations - but was very straightforward to fix - which always makes me smile :)

This issue only affects KahaDB and is fixed in trunk, the ActiveMQ 5.3.1 branch and Fuse Message Broker versions - and for those interested - the code change is here.

Apache ActiveMQ 2009 in Numbers

2010-01-04T05:03:00.000-08:00

So Claus started this with his blog on Apache Camel - 2009 in numbers - so I'd thought it would be interesting to do the same with ActiveMQ (does not include ActiveMQ C#, C++ or Stomp sub-projects). The numbers are not as good as Camel - but still good:

Number of posts to the ActiveMQ user forum in 2009: 4440
Number of posts to the ActiveMQ dev forum in 2009: 4447
Number of commits in 2009: 2893
Number of downloads 2009: - hard to know (because of Apache mirrors) - but about 60k a month

Number of tickets created in 2009 - 505 (295 of these are resolved)
Number of resolved issues in 2009 - 467

We didn't quiet manage to resolve more issues than were raised - but this is because as a community, until very recently we have been very lax at accepting issues - so a lot of issues raised don't even have the version, let alone the environment or a test case! If a test case is attached - we can usually resolve an issue very quickly - its reproducing the issue that always takes the time.

Query Statistics for Apache ActiveMQ

2009-10-15T05:13:00.000-07:00

Apache ActiveMQ has extensive support for JMX, but if you want to find out the running state of ActiveMQ from your application or from the one of the number of cross-language clients - your options are a bit limited.

ActiveMQ supports Broker plugins, which allows the default functionality to be extended, and new with version 5.3 of Apache ActiveMQ is a Statistics plugin, which enables statistics about the running broker, or Queues and Topics to be queried. To configure ActiveMQ to use the statistics plugin - just add the following to the ActiveMQ XML configuration:

...

</plugins>

The statistics plugin looks for messages sent to particular destinations. To query the running statistics of a the message broker, send an empty message to a Destination (Queue or Topic) named ActiveMQ.Statistics.Broker, and set the replyTo (jmsReplyTo if using JMS) field with the Destination you want to receive the result on. The statistics plugin will send a MapMessage filled with the statistics for the running ActiveMQ broker - e.g.

Queue replyTo = session.createTemporaryQueue();

MessageConsumer consumer = session.createConsumer(replyTo);

Queue query = session.createQueue("ActiveMQ.Statistics.Broker");

MessageProducer producer = session.createProducer(query);

Message msg = session.createMessage();

msg.setJMSReplyTo(replyTo);

producer.send(msg);

MapMessage reply = (MapMessage) consumer.receive();

for (Enumeration e = reply.getMapNames();e.hasMoreElements();) {

String name = e.nextElement().toString();

System.err.println(name+"="+reply.getObject(name));

}

You should see output like this:

vm=vm://localhost
memoryUsage=0
storeUsage=3330
tempPercentUsage=0
ssl=
openwire=tcp://localhost:50059
brokerId=ID:bigmac-50057-1253605065511-0:0
consumerCount=2
brokerName=localhost
expiredCount=0
dispatchCount=1
maxEnqueueTime=5.0
storePercentUsage=0
dequeueCount=0
inflightCount=1
messagesCached=0
tempLimit=107374182400
averageEnqueueTime=5.0
stomp+ssl=
memoryPercentUsage=0
size=10
tempUsage=0
producerCount=1
minEnqueueTime=5.0
dataDirectory=/Users/rajdavies/dev/projects/activemq/activemq-core/activemq-data
enqueueCount=10
stomp=
storeLimit=107374182400
memoryLimit=67108864

Similarly, if you want to query the statistics on a Destination, send a message to the Destination name, prepended with ActiveMQ.Statistics.Destination. For example, to retrieve the statistics on a Queue named test.foo send an empty message to the Queue ActiveMQ.Statistics.DestinationTest.Foo.

e.g.

Queue replyTo = session.createTemporaryQueue();
MessageConsumer consumer = session.createConsumer(replyTo);
Queue testQueue = session.createQueue("Test.Queue");
MessageProducer producer = session.createProducer(null);
Queue query = session.createQueue("ActiveMQ.Statistics.Destination" + testQueue.getQueueName());

Message msg = session.createMessage();

producer.send(testQueue,msg)
   msg.setJMSReplyTo(replyTo);
   producer.send(query,msg);
MapMessage reply = (MapMessage) consumer.receive();
assertNotNull(reply);
assertTrue(reply.getMapNames().hasMoreElements());

for (Enumeration e = reply.getMapNames();e.hasMoreElements();) {
            String name = e.nextElement().toString();
            System.err.println(name+"="+reply.getObject(name));
   }

You should see:

memoryUsage=0
dequeueCount=0
inflightCount=0
messagesCached=0
averageEnqueueTime=0.0
destinationName=queue://Test.Queue
size=1
memoryPercentUsage=0
producerCount=0
consumerCount=0
minEnqueueTime=0.0
maxEnqueueTime=0.0
dispatchCount=0
expiredCount=0
enqueueCount=1
memoryLimit=67108864

You can also use wildcards too, and receive a separate message for every destination matched.

Apache ActiveMQ 5.3 Released!

2009-10-13T06:05:00.000-07:00

ActiveMQ 5.3 is finally released - and probably the biggest lesson is that going forward, we need to release far more frequently!

Here are some highlights:

We are making it a lot more obvious how to scales brokers vertically, to 10s of thousands of Queues/Topics - by including more example configurations.
New incarnation of KahaDB one of the persistent options we support. This provides vastly improved recovery times and scalability.
We've upgraded to the latest and greatest Apache Camel 2.0 - allowing for enterprise integration patterns to be run right inside the worlds most popular message broker
Support for getting broker statistics from languages other than Java
Message Expiration support directly in-store (any message store - and we support a few)
More extensive advisory messages
Enhancements to Master/Slave - including more flexibility for starting slave brokers
NIO and SSL support for the mostly widely supported messaging transport - STOMP
over 300 issues resolved since version 5.2

Apache ActiveMQ is used in thousands of production environments for delivering high performance and reliable messaging under extreme load. Being an Apache project, it is supported by a diverse community of developers with extensive knowledge of building messaging solutions. Why not go and give it a spin ? - try the latest download!

Fuse Forge is now open!

2009-07-01T21:44:00.001-07:00

The FUSE team has been hard at work over the last few months and here is an update on the great new capabilities that we're rolling out. We are entering a new phase in our community evolution by opening up FUSE Forge. The Forge is a collaborative environment for the development of open source projects related to FUSE and the Apache projects that they leverage. On FUSE Forge you can create and run a project, participate as a committer in one or more projects, or just browse the ever-growing library of components designed for FUSE. In the spirit of open source transparency the FUSE team has moved all internal development to the forge.

New Project Highlights:

• FUSE Depot - FUSE Depot is a web-based tool for provisioning distributed applications on Apache Tomcat and FUSE ESB, our certified distribution of Apache ServiceMix. With FUSE Depot you can assemble bundles and libraries into applications, assign those applications to containers, and schedule deployments all from a central web console.

• RestMQ - RestMQ defines an open, easy to integrate, loosely coupled, technology independent and interoperable messaging protocol so that you can use your web browser (or curl on the command line) to reliably send and consume messages.

• Management - A set of FUSE management extensions using JMX and REST, visualized using FUSE HQ.

• Actional Diagnostics - Actional Diagnostics helps developers build, test and deliver Web services for SOA, REST and POX by reducing the complexity of XML and making testing easy and accessible early in the product life cycle.

For more information on FUSE Forge see the FAQ!

Scaling ActiveMQ

2009-05-26T06:10:00.000-07:00

Scaling ActiveMQ has come up a lot recently on the ActiveMQ users lists. One of the strengths of Apache ActiveMQ is its flexibility - but sometimes that's its curse too. Out of the box, ActiveMQ is configured to handle hundreds of very deep (contain 100 of millions of messages) - but its not the best configuration to scale queues horizontally (thousands of Queues on the same broker).

I've added a couple of FAQ entries to the Apache ActiveMQ site:

Something we need to address in the upcoming 5.3 release, is to provide more example configurations - to enable users to get a good head start on picking the right configuration for their use cases.

Work Starting on Mixing ActiveMQ and SonicMQ

2009-02-12T22:23:00.000-08:00

So we are getting some of the SonicMQ experts to dig in and help work on the next major version of ActiveMQ.

The aim is to take performance and scalability to the next level - while ensuring that stability and reliability increases. The SonicMQ guys have some great test environments which will enable us to provide even better quality code out of the door.

There's also some cool features that SonicMQ has like its Dynamic Routing Architecture and clustering that we are going to look to repurpose.

As ActiveMQ 6.0 is going to be a new architecture - it also enables us to look at supporting some more weird and wonderful wire protocols too (the invasive ones which force you to rewrite your broker to use them).

The development work is being done at Apache - with Hiram setting the pace at the moment. Please feel free to jump in and help - :)

90's Retro is in!

2009-02-06T02:48:00.000-08:00

Sometimes you just get distracted on some of the most mundane things - like design's of Logo's Web Site etc. Though when you start working for slightly larger organizations, you soon realize that there is a lot of things you don't know.

For example, I didn't realize that the 90's retro was back? I just came across this thread (pretty old I admit) about what constitutes a 90's look:

things like a swish, CAPITALIZATION , drop downs etc.

So its refreshing that we have decided to be a head of the curve and will be adopting the 90's look again! Just of to get a mullet ....

Progress FUSE Open Source Team still Growing ...

2008-12-13T07:34:00.000-08:00

Its been a hectic couple of months as we try and find our feet in Progress - but the Progress folks have done their best to make us feel part of their family.

Progress has some great SOA technology, and we've been having some good cross-pollination of ideas with the Sonic folks and pulling together development plans so we can leverage Apama, Actional, Mindreef etc.

We've been growing the number of users and committers from Progress of our Apache projects - ActiveMQ, Camel, ServiceMix, CXF etc. The size of the team was already good from the IONA days - but we are always looking at extending our breadth and depth of Apache committers.

In the past couple of weeks we've been able to strengthen our ServiceMix team by dragging back Chris Custine and our ActiveMQ team have some great additions in Dejan Bosanac (who is one of the authors of ActiveMQ in Action) and Tim Bish (ActiveMQ CPP lead).

We have another one or two Apache commiters joining in the next few weeks too!

Btw - a common misconception of our FUSE Products is that you can only get them on subscription - which simply isn't true. They are drop in replacements for the Apache projects (same code mainly), same license, same cost (its FREE) - the only difference being we have more control over the releases. Our enhancements and patches go into Apache first (we always want the Apache projects to gain the benefits of our investment) and then we pull in the deltas into the FUSE repos - selectively taking the deltas to build branches we release from. This enables us to support our customers (users who want to have the benefit of our support in a timely manner) in development and production in the same way a closed source organization would :). Why not give them a try ? A lot (tens of thousands) of folks already have, as the gain the benefit of stable, regular releases. And if you don't like them - as they are just drop in replacements - just go back to the Apache versions.

IONA FUSE Open Source Group now offically part of Progress

2008-09-15T23:54:00.000-07:00

Progress completed its acquisition of IONA Technologies last Friday. The FUSE open source group will be running as its own business unit as part of the Progress Software Corporation - so we are still operating as before - but with considerably more investment.

Progress saw the number and quality of enterprise customers we have using our integration products, saw that we consistently met and exceeded our sales targets and realized our potential to grow significantly.

So although we currently employee a lot of open source developers - we will be recruiting even more - and have some exciting new software projects in development for release in 2009 :)

Most useful iphone app I'm using to date - is Vicinity

2008-08-07T03:26:00.000-07:00

I'm behind most folks in upgrading to an iphone 3G - but the most useful iphone app I'm currently using is vicinity - its locates all the useful things you need if you on the road and find yourself in a strange place (restaurant, cafes, bars, stores, taxis, banks etc) - it integrates very well Maps too - I highly recommend it!

There's a nice overview here.

Early Access to ActiveMQ in Action

2008-08-06T01:00:00.000-07:00

There's early access to some chapters from the book ActiveMQ in Action - on the Manning Early Access Program - MEAP.

The book is aimed to be a practical guide to ActiveMQ, explaining the best practices for configuring brokers and clients, common gotchas - how many times do we need to explain that one ? ;) - broker persistence models, topologies, tuning, security and integration with containers such as JBoss, GlassFish and Geronimo.

SpecJMS2007 using ActiveMQ

2008-05-30T04:51:00.000-07:00

There has been a need for an industry standard benchmark for message-oriented middleware that use the JMS API for a long time. SpecJMS2007, which came out October 2007 is the first - and we've been waiting for a JMS vendor to be the first to dip their toe and share their results.

An independent benchmark is important, because it negates the chance to skew home groan tests to a vendor's strengths. So I thought there would be a lot of interest in this - but it seems no one wants to be first - a bit like a 70's disco where folks are all scared to be the first to be on the dance floor - luckily that's a problem I've never had :)

So IONA Technologies are currently running SpecJMS2007 against Apache ActiveMQ - and we will be giving a webinar on Tuesday June 10th, about SpecJMS, ActiveMQ and some of the performance tuning you can do get maximize throughput and how you can use ActiveMQ 5.x as an EAI system, now that it embeds Apache Camel.

If you can't make June 10th - the webinar will be available as part of Open Source SOA for the Enterprise series.

The actual results need to be verfied by Spec before they publish them - but I'll link to them once they've been verified.

BTW - I've found myself getting a little lazy on reading blogs, using twitter etc - I guess there's too much to follow. Which is why I've only just come across this blog - which gives a good summary on using routing and mediation with ActiveMQ.

IONA provides enterprise versions, support, services and training for ActiveMQ as well as ServiceMix, Camel and CXF - hence our desire to publish this work.

Apache CXF graduates out of the Incubator!

2008-04-17T03:48:00.000-07:00

Congratulations to Dan Kulp and the rest of the community for graduating CXF out of the incubator at Apache!

This is a big deal, its taken 20 months to finally prove that there is a vibrant and diverse community behind Apache CXF.

There's often this mis-conception that graduating from the incubator at Apache is related to the code's maturity (CXF has always been a very stable and mature product btw).

However, the Apache Software Foundation isn't about just the business friendly license, Apache projects have business friendly communities too. To get out of the incubator the project needs to have a diverse number of committers and develop code in a collaborative and meritocratic way. This ensures that a project is a vibrant entity in its own right, outside the influence of one company or an individual developer.

Things I'm looking forward to this Spring ...

2008-04-15T01:38:00.000-07:00

We have been keeping our heads down, ignoring (and sometimes laughing at) the gossip around Iona OS group, and keeping our focus on our Apache projects these past few months.

Things bubbling up and to keep an eye out for:

ServiceMix 4.0 - whilst keeping JBI very much a first class citizen, there is tighter integration with Apache Camel - making integration even easier - whilst allowing room for JBI 2.0 and SCA to be co-located in the future. Milestone 1 should be released in Apache very, very soon. Guillaume and Bruce did a good talk about this recently at ApacheCon.

ActiveMQ 5.1 - lots, and lots of late nights and weekends ironing out some complex use cases from our users - which has really helped us harden this enterprise class messaging system - thx Hiram! Look out for this release over the next couple of days.

CXF - both graduating from Apache incubator (fingers crossed Dan!) - and releasing 2.1 - which will be supporting JAX-WS 2.1, JAXB 2.1 etc.

and Camel just did a 1.3 release with tons of new features (yay James!).

Things I'm not looking forward to:

Leeds Utd missing promotion - I know its going to happen :(

Introduction to Apache Camel

2008-03-08T05:15:00.000-08:00

Thought I'd best link to this article written by James Strachan and myself (Jim isn't the camel btw!)

Great ActiveMQ and ServiceMix Training

2008-02-12T01:02:00.000-08:00

Iona is running some public training courses in both London and the Boston area:

London,UK March 11-14

Waltham, MA March 17-20

This is a good opportunity for architects and developers to gain hands on experience of using a JBI based ESB (ServiceMix) and the most popular messaging system (ActiveMQ).

Don't get confused by the FUSE branding, Iona builds and fully tests Apache ServiceMix (FUSE ESB), Apache ActiveMQ(FUSE Message Broker) as well as Apache Camel (FUSE Mediation Router) and Apache CXF (FUSE Services Framework).

These are essentially the Apache versions, but with some more system tests and documentation.

Apache ActiveMQ 5.0 Released!!!

2007-12-16T10:12:00.000-08:00

It's been a while coming - but version 5.0 of the Apache ActiveMQ message broker is now available to download.

Perhaps the biggest single change is the use of cursors to page messages from disk or store into the broker - which have taken a lot of time to bed in. This new feature even had an impact on the default message store - we had to write a new one!!

To improve flexibility and to increase the scope on integration cases, Apache ActiveMQ 5.0 also supports Enterprise Integration Patterns via Apache Camel.

You can also get a fully tested production ready version, from Iona.

Erlang Server For Stomp

2007-12-12T03:16:00.000-08:00

So Jeff Xiong has just anounced the erlstomp project - an Erlang broker that uses the stomp protocol. Stomp is a simple text oriented wire format for messaging - which is really starting to get a lot of traction because of its so easy to implement new clients with.

Existing clients include:

C, C++, C#, Java, Delphi, Flash, Ruby, Smalltalk, Perl, PHP etc.

This brings the number of stomp supporting brokers to 5 (including ActiveMQ)!!

Great Open Source Enterprise Integration

2007-08-17T03:08:00.000-07:00

James just posted a great summary of upcoming features in the Apache Camel 1.1 on his blog Camel is a fantastic integration framework - and its now been embedded in ActiveMQ 5.0 (released real soon - honest).

We now have available from Apache the four core foundation stones for Service Oriented integration:

Bad, Bad Journalism

2007-05-23T23:59:00.000-07:00

Typically bad dis of JBI 1.0 - quote from an unnamed source: "JBI 1.0 was not ready for prime time; it focused on vendor needs and not the user, and has been a miserable failure." - yeah right. A miserable failure for some vendors.

Some open source JBI 1.0 implementations continue to be the most widely used, standard based esbs on the planet!

BTW - if you want an idea of what JBI 2.0 will probably look like - check out the CAMEL!

What's that coming over the hill ? - Apache Camel!

2007-05-08T06:46:00.000-07:00

Apache Camel a sub project of Apache ActiveMQ - is a declarative rules based framework for integration based on enterprise integration patterns - or EIP. Camel is designed to be very easy to use, and suports generics, automatic type conversion and already has a large number of integration components.

So why Camel ? We saw a real need for an easy to use, easily embedded integration package that is Apache licenced and really complementary to ServiceMix.

Camel version 1 should be ready to ship tomorrow - once James recovers from his hangover ;)

Apache ActiveMQ: Version 5.0 Broker Changes

2007-02-03T04:29:00.000-08:00

ActiveMQ has now graduated out of the incubator, which is going to enable us to get on track on releasing more often.

We are currently going through a stage of consolidation of features to ensure we have a solid foundation to build on going forward, and these features will be available on 5.0 - the first milestone release will be in the next couple of weeks.

New features will include:

New dispatch model for Persistent messages.

Prior releases to 5.0 have kept references for all the messages that could be dispatched to an active Durable Topic Consumer or a Queue in memory. Whilst a reference itself was not large, it did impose a limit on the maximum number of messages that could be pending delivery.

A typical approach for messaging systems dispatching persistent messages is to pull them in batches from long term storage when a client is ready to consume them, using a cursor to maintain the next to dispatch position. This is a robust and very scalable approach, but not the most performant for cases when the consumer(s) can keep up with the producer(s) of messages.

Hence ActiveMQ 5.0 has a hybrid approach, allowing messages to pass from producer to consumer directly (after the messages have been persisted) but switches back to using cursors if the consumer(s) fall behind.

New Default Message Store

Hiram and I have been working on different storage options for a while now, but we've finally settled on an implementation that uses the best bits of the journal and Kaha to provide a storming messaging persistent engine, which scares the pants of me - its sooo fast!

Slow Consumers

Version 5.0 will a combination of outstanding messages and memory consumption to decide when a consumer can be sent more messages (message prefetch). This will allow us to determine when consumption is slow, regardless of the message size.

Spooling to disk for non-persistent slow consumers has also been optionally introduced. There is some more tuning thats required to be done to get this right, but should be ready this week.

Other features that will be incorporated in the 5.x release:

* automatic recovery for the 'shared nothing' master/slave (about to start on this)
* bi-directional network bridges

open source SOA platform released!

2006-07-31T00:58:00.000-07:00

We've been busy getting the latest release of FUSE FUSE 1.2 out of the door - the best new feature is perhaps the Eclipse Tooling - check out the tutorials

We have also included single-sign on security and are now using the Liferay portal!!

Filters for JMX over JMS in Lingo

2006-05-01T03:40:00.000-07:00

Just going through some code in lingo and realised that I'd only half put in support for specifiying group and server names for the JMS destination used for communicating between the JMX Connector and Connector server - Doh!

By default any MBean Server using the Lingo Connector will be logically grouped and results of operations and attribute values will be averaged - which can be handy for looking at server stats - e.g. average memory usage across a cluster.

As Lingo's JMX over JMS uses wildcards, by using the destinationGroupName and destinationServerName properties, you can restrict the view from the connector.

On the server side - it would be good practice to always specify the group name and server name, then depending on what the client side connector wants to view, specify exact matches or wild cards.

File Store Persistence for ActiveMQ

2006-04-29T22:52:00.000-07:00

I've recently been working on a file based persistent store (called Kaha) for the ActiveMQ messaging project. There were a couple of reasons for doing this:

firstly we are about to re-architect the message dispatching for persistent messages, so having something with a simple interface (Collections) would enable some rapid prototyping
It's got to be compatible with the Apache licence - ruling out some well known alternatives
I wanted to see if I could provide a faster persistent engine than the current ones we use in ActiveMQ.

For messaging systems, the general pattern for persistence is that messages are short lived in the store - being written and deleted extremely quickly. Messages only tend to be read for message consumers that haven't been active for a while (or slow consumers - when messages are read from the store so as not to constrain other consumers).

Persisted messages are generally read in insertion order, but also need to be accessed by their id so they can be deleted (message acknowledgements can be received out of order - e.g. for Queues). So using a mixture of persistent Maps and Lists from the developers view is ideal.

The Kaha persistence adaptor is available in the latest snapshot of ActiveMQ, though we are not going to push this as a new feature until the 4.1 release. But if you're brave enough - give it a try :)