Ever since computers were first used to solve business problems, there has been a great gulf between users and programmers. And that has always meant that, whenever a business process needs to be changed, or a new system or partner introduced, then complicated and often expensive IT issues will soon arise.

But that may be about to change. Today, analysts and a handful of groundbreaking vendors predict that service-oriented architecture (SOA) and business process management systems (BPMS) will enable business managers and users to create the software they need almost as quickly and effortlessly as they build new spreadsheets. And they might even be able to do so without too much help from the IT department.

How is this paradigm-busting change to be brought about? The story centres on middleware – poignantly described by Novell’s vice chairman Chris Stone, when he was CEO of the Object Management Group (OMG), as “software that no one wants to pay for”.

Middleware is packaged software for building distributed systems. Back in the 1970s, seasoned programmers struggled for months to get networked applications working properly. Nowadays, provided the environment is not too complex or the requirements too demanding, standard middleware can do the job in days or hours – and, if used properly, does it reliably and efficiently. Since the great majority of corporate applications are now distributed, it is no exaggeration to say that middleware has become an indispensable part of modern software.

Unfortunately, this is where the trouble starts. Networked and distributed systems are very complicated indeed, and this complexity is inevitably reflected in the vast array of middleware tools and methods. Synchronous or asynchronous, API or message-oriented, standard or proprietary: the variety is immense. And that is before the taking into account the added complexities of transaction support, guaranteed delivery, security, quality of service (QoS), error handling, lifecycle management and a host of other options.

Naturally enough, non-technical business users are baffled and repelled by talk of MOM, RPC, DCE, CORBA, COM+, MQ, JMI, SOAP and .NET remoting. When Stone said that no one wants to pay for middleware, he realised that the average business manager does not see why all that “back-end stuff” is not taken care of automatically, behind the scenes.

The 1990s witnessed significant progress towards the automation of middleware. Standardisation took great strides; for example, most middleware is now layered on top of the ubiquitous Internet protocols such as TCP/IP. The latest generation – web services – is being rigorously specified by a variety of standards-setting consortia.

Meanwhile, development tools, also used for both building and integrating distributed systems, are being progressed. Today most types of middleware can be automatically managed and integrated through popular development environments (IDEs), such as Microsoft Visual Studio .NET, Borland Delphi and the open-source Eclipse and NetBeans. Today’s distributed systems are usually assembled from heavyweight building blocks, such as application servers, database servers, queues, buses and hubs, and support for all of these is built into Microsoft’s .NET and Sun Microsystem’s standard, Java 2 Enterprise Edition (J2EE).

This rapid progress in the automation of middleware is all very well and good; but so far there has been little to suggest that the whole messy business is going away any time soon. On the contrary, the middleware scene is becoming steadily more ramified – to the point where it is a full-time job just keeping track of all the new specifications and tools. What realistic prospect is there of a reversal in this trend?

The first attempt to break the mould came in the late 1990s, in the shape of enterprise application integration (EAI). Corporate applications were (and still are) notoriously incompatible, due to their piecemeal construction over the years by individual departments and functions. Package vendors, by developing for specific functions or customers, and often growing by acquisition, compounded the problem.

The result: a forest of IT ‘silos’ or ‘stovepipes’, each internally consistent but quite unable to interoperate with the others. Instead of radically re-architecting them all – which would be hopelessly difficult and expensive – EAI proposed to link them together by enabling them to exchange simple messages. Typical EAI products also provided ‘hubs’, central servers that co-ordinated and shaped the flow of messages, often with the help of a workflow engine.

EAI products helped, but their focus on synchronising data, rather than building flexible cross-departmental processes, did not entirely solve most customers’ problems. And they proved expensive and sometimes difficult to use.

Two related concepts are the enterprise service bus (ESB) and the integration broker. An ESB is any form of middleware that can be deployed throughout (and between) organisations, and that provides queue management, guaranteed delivery and controllable quality of service. The ideal integration broker (sometimes called a message broker) combines the roles of EAI hub, schema translator and business rules engine with other desirable functions, such as security, remote management and transaction auditing.

The latest wave of middleware makes full use of the emerging, and fully open, web services standards. While there is nothing revolutionary about the idea of exchanging XML messages over HTTP (the Internet transport standard) and other protocols, web services exert a unique appeal – if only because they have achieved universal acceptance within the industry. Web services enables applications to locate each other and to be hooked together without extensive programming, and, to a degree, to understand each other’s syntax and content.

SOA generalises the ideas behind web services, and according to some experts, opens up the possibility of applications composed of independently-built remote services. A number of vendors, including Kenamea, SAP, Siebel, SeeBeyond, Sybase, Vergil, Vignette and webMethods, are already marketing “composite application builders” to automate their construction. There is also an emerging market for composite applications, such as SAP’s xApps.

It is easy enough to set up simple, ad-hoc web services. Most business applications, however, require the ability to invoke multiple services one after another – or even simultaneously – and to execute conditional logic. “If the applicant is credit-worthy,” for instance, “then process application; otherwise send explanatory message and end transaction”.

Several overlapping standards specifications address the need for what is sometimes called ‘choreography’ and sometimes ‘orchestration’: BPEL4WS, WSCI, BPML and ebXML BPSS are the four best known. They all provide facilities for sequencing service invocations in accordance with business rules, and dovetail with other specifications that control transactions, security and provisioning. ERP and CRM vendors, such as PeopleSoft, SAP and Siebel, and integration specialists like SeeBeyond and webMethods, have been quick to see the potential of SOA.

The role of BPM

Exciting as SOA and composite applications look, BPM is still more radical. According to Howard Smith and Peter Fingar, authors of ‘Business Process Management: the Third Wave,’ it is an entirely new discipline based on the rigorous mathematics of Pi Calculus, which isolates business processes from software applications and allows them to be directly specified by the business.

Other experts share their enthusiasm: for instance Clive Finkelstein, who pioneered Information Engineering in the 1980s, describes BPM as “an approach that uses service-oriented architecture (SOA) as a bridge between business integration and technology integration”.

BPM purists insist that it has the potential to replace application development altogether. Instead of writing code, they say, managers and business professionals should be able to define business processes for which they are responsible, and have them execute immediately on a special-purpose BPMS. Lest there be any lingering doubt as to what they are proposing, Smith and Fingar urge: “Don’t bridge the Business-IT Divide: Obliterate it!”

At present, however, this remains largely futuristic. Specialist BPM suppliers mostly focus on the development of new application using repositories of processes, or on orchestrating existing processes. But most concede that there is still some low level coding to be done somewhere.

Given that middleware innovations have been appearing more and more rapidly of late, what are the odds that the current crop of standards will be the last? Very low indeed. That is one of the reasons for the OMG’s latest initiative, Model-Driven Architecture (MDA).

In a nutshell, this permits developers to create platform independent models (PIMs) that distil their enterprise’s business logic and data structures, without committing to deployment on J2EE, .NET, CORBA or any other specific middleware platform. A single PIM might be translated into a platform specific model (PSM) for either J2EE or .NET, allowing rapid and trouble-free migration should the need arise.

MDA is perfectly compatible with everything else discussed in this article, with the single exception of the more advanced vision for BPM. In Smith’s words: “Although MDA and BPM are both architectures for enterprise software systems the comparison stops there… MDA is designed to enable the computer-assisted generation of applications and components, i.e. software. By contrast, the purpose of BPM is to avoid writing, or generating, more software…”

This is an exciting time for middleware specialists, but – by the same token – a confusing and difficult time for middleware users. Important, possibly committal, decisions must be taken, but there can be no certainty about how the future will develop. The apparent imminence of revolutionary improvements such as BPM merely serves to raise the stakes. Which horse to back? There is no shortage of promising technology; the question is how much it is possible to accomplish with that technology.

Ad-hoc web services, composite applications and even in-house SOA are eminently feasible right now, although many of the necessary specifications have not yet been completed, let alone implemented and productised.

Full-fledged inter-enterprise SOA, equipped for dynamic service discovery and able to accommodate continuous modification of constituent services, still faces serious technical and non-technical obstacles. Facilities for cross-enterprise security and service management, for example, are in their infancy. Nevertheless the way forward is tolerably clear, and this type of application should become generally possible within a few years.