Bill Wohl is angry. As vice president of public relations at enterprise applications company SAP, he spends most of his time – and millions of dollars of SAP’s money – crafting and refining the company’s image as a supplier of software products that deliver demonstrable, sustainable value to thousands of customers. But his work is being undone, not by competitors, but by a handful of artful outsiders. A survey of SAP customers released in March 2003 by IT market analysts at Nucleus Research challenges the company’s whole payback story.

Their research – based on a straw poll of customer references highlighted as ‘success stories’ on SAP’s web site – found that more than half of those interviewed had, to date, failed to achieve any kind of positive return on investment (ROI) from their deployments. That sample may be unrepresentative, and indeed, the average respondent had only been using SAP software for 2.8 years. But, almost half of them also reported that their implementations of various SAP applications has exceeded the initial deployment budget.

Wohl’s reaction is blistering. “There is no poorer example of the marketing of ROI in the software industry today than the conduct of Nucleus Research,” he says. Analysts at Nucleus, he adds, “do the minimum of work to achieve maximum exposure.” Their approach is “unfortunate and irresponsible”.

His reaction is – to some extent – understandable. ROI is big business for SAP, just as it is for all technology companies. It has become the focal point of sales and marketing efforts as technology spending has been squeezed, and as IT budget holders have had to demonstrate much more clearly how and where IT delivers business benefits.

Indeed, IT managers and the ‘sponsors’ of IT projects within organisations are acutely aware that they are seldom given the go-ahead for any significant implementation without solid evidence that it will deliver increased productivity, revenue growth or cost reductions. In short, the business needs to be convinced of tangible ROI, and that means suppliers need to be well armed with convincing ROI arguments.

“In the past 12 to 18 months, I would say that between 50% and 70% of sales relied on us coming up with some sort of ROI calculation,” says Neil Sholay, industry marketing manager at application server company BEA Systems. “There were plenty of examples where [an ROI calculation] tipped the balance in our favour.” He cites British Telecommunications and Lehman Brothers as just two cases.

Figmental projections?

Meaningful and accurate ROI figures are, however, notoriously difficult to determine, and the area of technology ROI is, as a result, wide open to abuse. As Paul Strassmann, former CIO of the US Department of Defense and renowned IT economist, writes in his book The Value of Computers, Information and Knowledge: “Asking for a direct tie-in between increased funding for computers and a commitment to deliver provable

savings is an invitation to prepare figmental projections that demean both those who produce them as well as those that accept them.”

That does not detract from the fact that organisations still need to find reliable ROI estimates – and who can they trust to help them do so?

Software suppliers are certainly falling over themselves to assist. Their web sites and marketing collateral are awash with ROI justifications, projections and case studies of customer projects designed to create the impression that ROI is a universal experience. Many (from IBM and Sun to Microsoft and Siebel) try to provide reassurance with ‘ROI calculators’ or ‘value diagnostic tools’ – some are rudimentary spreadsheets, others are complex, deeply analytical templates. By plugging in details of the planned project and the size and nature of their company, prospective customers can get an estimate of the return the organisation can expect to achieve.

Many of these tools, however, have significant shortcomings, says Enrico Camerinelli, an analyst with IT market research company the Meta Group. “Often, companies get distracted by ‘blue smoke’ evaluation tools provided by vendors that look at how to measure ROI, but not at how to obtain and optimise it.” The final goal of any ROI model is not just to come up with a mathematical calculation of a project’s investment return, he adds, “but to [enable an organisation to] design a business case and plan for the highest degree of business success.”

Rebecca Wettemann, vice president of research at Nucleus Research – the company so maligned by Bill Wohl at SAP – goes further. “Any kind of automated ROI is generally bad news. Anything that’s based on average ROI is bad news. Vendors play marketing tricks with these numbers all the time,” she says.

As a measure of vendors’ faith in their marketing potential, a recent search on Google.com of the term ‘ROI calculator’ returned 94,200 hits. Just two years ago, the same search pointed to just over 15,000 such web pages.

Vendor ROI calculators, Wettemann explains, enerally use one of two different methodologies: average ROI and cumulative ROI.

Average ROI is based on the gains a group of current users reported after deploying a given technology. It provides some indication of the level of satisfaction in the current customer base, she says, but cannot reliably predict the returns a specific company is likely to see.

“Just as no right-minded CEO would decide to invest or not invest in building a new factory based on the average returns of peers that have built factories, no technology buyer should ever set local expectations based on a report of the average ROI achieved by existing users,” she says.

Cumulative ROI (cROI), on the other hand, aims to represent the sum of the gains made by a specific company over a certain length of time. This is achieved by adding together the benefits from all years under consideration, then dividing this total by the project’s initial cost.

cROI, she says, is also deeply flawed for a number of reasons. First, it can be used to refer to any length of time – even to the whole lifecycle of the product – which means it frequently yields a number that is several times greater than the real investment number. “For example, if a vendor’s promise of a 200% cROI depends on a five-year span, it is really offering 40% ROI per year – and even less once the time value of money is taken into account.”

In addition, because it aggregates returns unreasonably, cROI gives an exaggerated portrayal of returns and gives no sense of what will happen to the bottom line when the company closes its books at the end of the year. Finally, because returns from a technology deployment tend to grow over time, cROI also gives a misleading impression of what will happen in the crucial first year, and of the likely payback period.

“Average ROI reports may satisfy the lazy, and cumulative ROI may impress the foolish, but only a well-grounded analysis based on annual ROI and payback period will suffice when money is on the line,” she concludes.

Suspicious minds

End-users are justifiably suspicious of the ROI figures suppliers produce. “These ROI calculators are just sales tools held out by sales people eager for commission. We’re not taken in by them,” says Owen Williams, IT director at global property consultants Knight Frank. His company, he adds, chooses instead to rely

on “our own reasonable, informed view based on a rigorous questioning of the assumptions of suppliers”.

Knight Frank, says Williams, has developed ROI models in-house that have evolved and matured over time and give the organisation relatively accurate projections of potential benefits and savings of IT spend. But not every company has the expertise or confidence to trust its own estimates.

That has given rise to a whole sub-industry for the IT industry analysis companies. Organisations such as IDC, Meta and Gartner have all developed practices and sophisticated tools to perform ROI analyses on behalf of clients. Nucleus Research, in particular, sprung to life in 2000 with the sole purpose of providing a standard financial methodology and a range of tools, data and research that subscription customers access in order to make ROI calculations.

Such services don’t come cheap, and fees are set in line with the degree of perceived risk. While until the last few years, many IT decision-makers were only required to give only a vague business justification for technology spend – if any – making an accurate assessment of ROI is now of paramount importance, and getting it wildly wrong is a career-threatening error.

Nucleus’ customers, for example, are IT directors and CIOs at large enterprises such as BT, drugs giant Pfizer and aerospace company Lockheed Martin. And it is safe to assume that fees reflect that. “Larger projects are more complex, and more interesting financial risks issues surround them,” says Wettemann of Nucleus Research

Calculation station

Where does the complexity lie, and can third-party assistance really offset the risk? After all, an ROI calculation, in its simplest form, merely reflects the benefits of a project divided by the initial cost of the project. The answer centres on timescales and the measurement of both tangible and intangible aspects of the project.

Since technology rarely covers its costs in the first year, advises Wettemann, accurate calculations of ROI use a three-year timescale. “With a three-year horizon, the ROI calculation is now the average net benefit (that is, the benefit less any additional cost) per year, divided by three, then divided by the initial cost, times 100.” Unlike culmulative ROI calculations, this average annual ROI calculation is “comparable to other investments, the cost of capital, or simply putting the money in the bank,” she says.

But predicting benefits is the difficult part. Some benefits – such as increased sales – are easier to quantify – but still require some element of prediction and the elimination of non-IT factors. For example, when considering introducing new online services, financial services company Alliance &Leicester takes into account such metrics as increased volume of loans made, reduced customer churn and decreases in outstanding balances on credit cards, says Andrew Robinson, the company’s ecommerce director.

Other factors are less easily gauged in raw numeric terms – in particular, increases in productivity. Generally, these estimates are made by calculating the time an organisation expects its employees to save because of the introduction of a new technology; for example, if a company believes a new sales application, for example, that allows 100 representatives to file orders from the field, will knock 30 hours off each person’s annual administration time, then overall, it will clearly save the company 3,000 hours per year – with all the overhead reduction that that entails. However, time savings do not always translate into productivity gains – they are subject to a slew of human factors, such as management and job satisfaction.

A way around this problem is through the introduction of correction factors. Nucleus analysts, for example, have developed a database of correction factors to account for what they call “inefficient transfer of time”, or the phenomenon in which time saved by employees does not amount to an equal increase in time worked. A productivity correction factor, explains Wettemann, is a number of less than 1 and more than 0 that is used to correct the estimate of time saved to account for the inefficient transfer of time.

“Multiplying the time saved by the correction factor enables you to quantify the actual returns from increased productivity,” she explains. For example, sales staff that receive commission, and are thus more motivated to use saved time for additional work, require a correction factor of 0.7 to 0.9. Production line workers require a correction factor close to 1.0 – but only “if all the workers on a line save the same amount of time and the foreman or line manager is watching”. Employees in the marketing department, says Wettemann, are less likely to use time saved effectively, making a correction factor of 0.5 or less “not unusual”. If an organisation is unsure about the correction to apply, it can use 0.5 as an average or a more conservative measure, she says.

Ankle biters

Like Knight Frank, Alliance &Leicester has its own in-house ROI methodologies that enable it to estimate IT ROI with some degree of accuracy. But on certain projects, it will turn to outside consultancies as a ‘reality check’. “We engage consultants to assess whether the project we have in mind is feasible, whether we are likely to see the benefits we’re predicting – and then to lead the project for us,” he says.

Because the consultants are setting goals that they themselves will have to stick to, he adds, their ROI expectations are realistic, even overly cautious: “They don’t want to find that the project turns on them and bites them on the ankles.”

IT services companies are generally more attuned to the potential pitfalls inherent in a major implementation, and in recent years, have begun to offer customers risk/reward-based contracts in which they are penalised for project overruns or an absence of expected benefits – for example, the outsourcing contract struck between Barclays Bank and IBM Global Services.

By contrast, software companies selling a package into an organisation have more to lose in the short term by alerting sales prospects to potential pitfalls that may dent their chances of achieving the projected ROI. The situation may be changing, however. In very large projects, users are starting to ask software suppliers for the similar risk/reward terms that they have come to expect from services companies.

According to Sholay at BEA Systems, the company is increasingly asked by customers in the telecommunications industry to offer some guarantees, even where the BEA software is actually being implemented by a third party. “Accenture and EDS carry most of the risk, but BEA still takes a hit [in terms of licence fee] if the project goes awry,” he says. Often, BEA is asked to prove benefits early on in a project in order to get the next portion of business from that customer, he adds.

Pain reduction

As that suggests, a way of “minimising the pain of upfront costs” – and reducing the risk of missing ROI targets – according to Robinson of Alliance &Leicester, is to take an incremental approach to IT projects, trialling a piece of technology for a set period of time. That is an approach the bank has used with some success in a number of recent projects.

In the case of an online mortgage application service, he says, the company was unsure as to the extent customers would be prepared to use it. Robinson’s team simply put an HTML-based application form on the organisation’s web site and waited to see what would happen. “This wasn’t accompanied by any online or offline marketing, and wasn’t integrated with any back-end systems. But the metrics we got enabled us to build an ROI case and take the project further,” he says.

In another example, a customer relationship management system for the company’s call centres, a trial of the software convinced Alliance & Leicester that the package was too inflexible and that customisation, implementation and training costs would outweigh any benefits of installing it.

By carefully balancing the risks and rewards of a project, companies like Alliance &Leicester and Knight Frank are able to formulate a business case for each project that inspires confidence in IT’s role. In experienced hands, moreover, ROI calculations can provide valuable insight into how a project may be modified in order to deliver a more attractive return.

For example, organisations may move costs out of the initial year of deployment by staggering spending on training or consultancy. Or they may be in a better position to negotiate a discount with a technology provider if they can convince that vendor that a steep discount is necessary to make the ROI case and seal the sale. Or they may decide to initially roll out the technology to a smaller division or group that is likely to deliver higher returns.

In any case, realism is arguably the strongest tool these companies can employ. “You will never get an ROI calculation entirely right, because you are trying to predict the future. But it can allow you to proceed with far more confidence,” says Alliance &Leicester’s Robinson.