In order to improve customer service and make its support engineers more efficient, US manufacturer Smith Electric Vehicles last year installed telemetry sensors into all of the trucks that it sells.
The idea was that data about the vehicles and their performance would be beamed back over the GSM mobile network to the company’s Missouri headquarters, where engineers could see how the vehicles were running in real time. That would allow the company to spot potential faults quickly and to dispatch parts and engineers in a timely fashion, keeping customers’ vehicles on the road.
It needed a system to process and integrate the large quantities of data that these sensors would create. The company decided that the only viable option was to adopt message queuing technology, which allows data to pass between systems without the need for those systems to connect synchronously.
However, commercial systems capable of handling the volumes of data they needed to analyse were simply too expensive. High-end message queuing software from the market leaders typically costs about $100,000 per server in licence fees, plus it requires consultancy services to implement and expensive in-house skills to operate.
“Message queuing is incredibly expensive,” says Sam Lambert, database engineer for the company’s telemetry team. “If you look at the market leaders, it’s unaffordable for most ordinary companies.
“It also seems to be a ‘dark art’ as well,” he adds. “People can work in IT for 30 years and never touch upon it.”
The company therefore looked at a number of alternatives. It considered Amazon Web Services’ cloud-based Simple Queue Service, which is paid for on a per-transaction basis. Organisations that have adopted SQS include NASA’s Jet Propulsion Laboratory, which uses it as part of the cloud infrastructure that operates its extra-terrestrial explorers.
However, Lambert was not convinced that SQS would be able to handle the volumes of data that Smith needed to process – roughly ten terabytes a month – especially during the spikes that occur when vehicles start up in the morning.
“We felt that it was too low on performance and that it would not be able to scale,” he says. “The feature set was quite small as well.”
More promising was software based on an emerging messaging standard called the Advanced Message Queuing Protocol (AMQP), an open, vendor-neutral Internet protocol designed for interoperable business messaging.
The development of AMQP has been led by a number of the world’s largest banks (typically the biggest consumers of message queuing software) and users include Bank of America Merrill Lynch, JP Morgan, Goldman Sachs, Credit Suisse and Barclays, as well as Germany’s Deutsche Börse stock exchange. Industry backers include Microsoft, Cisco Systems, Red Hat and VMware, all of which participated in the official version 1.0 launch in mid-October 2011.
This high-profile support was one of the attractions to AMQP for Smith Electric Vehicles, says Lambert. In the end, Smith opted for a cloud-hosted, AMQP-based messaging service from a start-up called StormMQ. This allows Lambert to concentrate on the core application and leave the running and maintenance of the messaging software to specialists, he explains.
The birth of a standard
The AMQP initiative was founded by John O’Hara, currently head of architecture and strategy for global commodities technology at Bank of America Merrill Lynch, in 2003 when he was working at JP Morgan.
O’Hara’s aim was to overcome a time-consuming challenge that he encountered time and again in his working life. With thousands of systems with tens of thousands of connections between them, he had become frustrated by the number of different messaging systems the bank had to run, the number of times a messaging system needed to be built for new applications and the number of times those systems needed to be upgraded when changes were made to just one application.
“Lots of banks have written their own messaging solutions, including JP Morgan and Bank of America Merrill Lynch. Indeed, you name the bank and they have probably written their own systems,” says O’Hara. Those banks also have a plethora of licences for proprietary messaging software from enterprise middleware vendors such as Oracle, Tibco and IBM.
As O’Hara saw it, open standardisation was the only solution. “It occurred to me that it needed to be bigger than any one company. So I sought to partner with other people to devise a specification.”
The early years of AMQP’s life involved building a consortium of companies to support the development of the technology on both the user and vendor sides. That meant developing a specification that could handle high-end messaging without being too heavyweight for mobile and embedded applications.
When O’Hara left JP Morgan in 2006, Integration Services group vice president Robert Godfrey continued the bank’s commitment to AMQP. “Establishing the ubiquity of AMQP means that we will have the ability to use best-of-breed solutions and have them work together without having to write thousands of lines of ‘glue code’,” says Godfrey. “The cost of training people to administer things will also hopefully be reduced if we use a common toolset.”
Deutsche Börse, the German stock exchange, is currently developing applications that use AMQP to disseminate critical risk information to clients.
According to the head of the stock exchange’s middleware practice, Dr Andreas Moravec, one of the characteristics that makes AMQP desirable is that it uses the Internet’s transport layer, via the Transmission Control Protocol (TCP), as opposed to the application layer where protocols such as HTTP operate. “It is therefore much more efficient and reliable than messaging over HTTP and other higher-layer protocols,” he explains.
The fact that it is independent of any particular application means that the protocol could be used to distribute all manner of financial information more widely, Moravec explains, and at much lower cost.
This could have a dramatic impact on the world’s financial markets. For example, stock market platforms are currently only available to companies that can afford the software to receive data in real time from the exchange. AMQP could blow open this oligopoly by drastically lowering the technical and cost barriers to entry.
AMQP in industry
The first version of the AMQP technical standard has been submitted to the OASIS global standards body, and on 1 November a technical committee was appointed to inspect the specification.
The hope is that it will be able to go through the process relatively quickly because so much work has already been done on the standard in the field. “We wanted to release the final protocol specification only after we had seen a lot of working implementations,” explains Deutsche Börse’s Dr Moravec.
Formal standardisation should accelerate adoption, says John Barr, research director for high-performance computing at analyst The 451 Group. “Many companies wouldn’t consider something as new as AMQP unless it was a formal standard,” he says.
So far, though, adoption outside the banking sector has been limited. However, support from suppliers such as VMware, whose RabbitMQ open-source messaging platform is based on the standard, may help to push things along.
RabbitMQ has won some high-profile adopters. Google’s Rocksteady project, a self-built, open source system to identify network performance faults as they happen, uses RabbitMQ as its messaging layer.
The Indian government is using it in the largest online identity management project in the world, which will provide the country’s 1.2 billion citizens with a unique identity number. RabbitMQ helps to decouple sub-components of the project’s various applications, enabling it to scale. Publishing company Pearson is also using RabbitMQ with eCollege, a suite of online training services that support around eight million students. The company uses AMQP as the integration fabric for two of those services – e-learning platform LearningStudio and collaboration system OpenClass.
The various subsystems that make up these services used to be integrated using daily batch processes. The trouble with batch processing, says eCollege software engineer Chris Chew, is that it quickly escalates from a once-a-day activity. “You get to the point where you are running the batch processes all the time,” he says.
That is time consuming and it affects performance, Chew explains. “To be able to send ‘bits’ of information between systems when events happen allows a much more even processing flow.
“Computers, software and systems – especially systems across the Internet – are increasingly operating in that type of event-triggered fashion,” he adds.
Pearson evaluated a number of possible messaging platforms to provide this event-driven integration. These included the Java Message Service (JMS), which was rejected because the organisation uses a number of different programming environments.
What attracted the company to RabbitMQ was the open source community surrounding the platform. “We really liked the support community,” recalls Chew. “As a project, it’s the most together I’ve ever seen, and the software is fast.”
Other implementations of AMQP include open source software vendor Red Hat’s Enterprise MRG (messaging, real-time, and grid) infrastructure platform and Apache Qpid.
Microsoft – not traditionally known for its support of open source standards – has also been an active participant. At the launch of AMQP in October 2011, the software giant demonstrated a system that allowed an Excel spreadsheet to be populated automatically with data hosted in its Azure Cloud platform using the messaging technology. The demonstration was intended to show how it might be possible for companies to run real-time accounts without having to spend time and money on bespoke middleware.
Disruptive potential
According to Deutsche Börse’s Dr Moravec, the advent of cloud computing is one reason why AMQP is so timely, as it is driving demand for high-performance messaging across systems, across networks and across organisational boundaries. “Until the emergence of cloud computing, most messaging systems were contained within large organisations, which had little motivation to move from established proprietary providers or from home-grown solutions,” he explains.
The same argument applies to mobility, Moravec adds. AMQP could well be used as the integration layer for the voluminous streams of data now being produced by smartphones and sensors, and is a strong candidate to become the basis for the ‘Internet of things’.
“It is very likely that AMQP will become the norm for any middleware messaging requirement,” says Moravec. What does this mean for the established messaging market leaders: IBM, Tibco and Oracle? By one token, it could be seen as a threat to their dominance of the $1 billion messaging market, providing a cheap alternative to their high-cost platforms.
But Moravec sees it as more of an opportunity for these providers to enhance their own products by allowing integration with a wider array of third parties.
“IBM, Tibco and Oracle provide a broad range of services around their messaging capabilities,” he explains. “In the longer term, AMQP provides them with a method to communicate with service providers that do not license their products, thus increasing their capabilities to provide enhanced services and applications to their customers.”
So far, these companies have barely acknowledged AMQP. But with support from such high-profile organisations as Bank of America Merrill Lynch and JP Morgan, it is not something they can ignore forever.
