For most of the last 10 years IT users have been fighting a losing battle against server sprawl – the apparently irresistible proliferation of commodity servers that are cheap and easy to buy, but costly and complex to own and maintain.

Now though, some IT organisations think that the end of the server sprawl war is in sight, and that the final victory will be theirs. In most cases, the companies that think this way are the early adopters of virtualisation software for Intel x86-based machines. Given their apparent success in rationalising server estates, who can blame them?

Using products such as VMware’s ESX Server, or Citrix’s XenSource Server, pioneering users of virtualisation technology have already culled thousands of obsolete or poorly-utilised commodity processors from the server estates – scrapping them in favour of fewer, optimally-configured servers, sometimes running dozens of multiple virtual machines.

The impact that this commodity server virtualisation has had on these organisations is frequently both immediate and dramatic. Little wonder, then, that some early adopters of virtualisation software believe they have finally found a compelling solution to the problem of escalating server estates.

Systems management in a virtual world

VIRTUALISATION is already having a profound impact on IT practices, but it may yet have an even deeper and more disruptive impact on the market for systems management software,

Hewlett-Packard, BMC, IBM, Microsoft and CA dominate today’s market. All – except Microsoft – owe their leadership position to a customer base largely concerned with operating mainframes and proprietary Unix systems. This sector has long accounted for the bulk of systems management revenues, while owners of commodity server farms have relied on “manual” management methods. Now though, with systems management widely expected to be an essential element in the increasingly virtualised world of commodity processors, this is all about to change.

Today, IDC credits VMware with 80% of all Intel virtualisation software revenues. Technologically, and in terms of breadth of product offering, VMware’s closest competitors are XenSource, now part of Citrix, and Virtual Iron. There is also a robust community of more specialised vendors such as PlateSpin (provisioning) and 3Leaf System (virtual I/O appliances and systems management).

Currently, there is little overlap between the virtualisation specialists, who don’t manage physical assets, and the traditional players whose virtual machine management capabilities are less well developed. However, the exception to this rule may turn out to be Microsoft.

Despite lagging VMware in the delivery of underlying hypervisor technology, Microsoft has a significant customer base for its operating system-level Microsoft Virtual Server 2005 product. The company’s stated goal is to provide the industry’s most complete set of management tools for both physical and virtual resources to complement its Viridian hypervisor product when it launches some time in 2008.

In the meantime, VMware continues to boost its virtual machine management (VMM) capabilities, by development and by the acquisition of companies such as Dunes – which takes a content management approach to VMM. Established players like HP are similarly trying to fill out their own VMM offerings, either through partnership with virtualisation specialists or through acquisition.

Who will win this classic confrontation between technology innovators and market incumbents?

With comfortably less than 10% of the world’s Intel-based machines yet virtualised, there is still 90% of a fast-growing market to play for. Technological prowess will have its part to play, but so will economic and market muscle. Shrewd observers like to note that the commodity server virtualisation market is largely a new twist in the evolution of the Windows server market. Although today it might be widely perceived as the virtualisation market ‘also ran’, ultimately it should surprise no one if Microsoft were to inherit yet another market invented by someone else.

Unfortunately, organisations that allow themselves to think this way may be heading for a rude awakening: The era of server sprawl may really be over, but the era of ‘virtual machine sprawl’ seems to be ready to rise in its wake.

Virtual machine sprawl may still be a new and largely unknown concept to most IT professionals, but in the view of experts such as Gordon Jackson, virtualisation evangelist of the grid applications deployment management software vendor, DataSynapse, “it may not be long before everyone is talking about it.” And, if they do, he adds, “it will be because we [the industry] hasn’t done enough to warn them about what is coming, and how to prepare for it.”

What is coming is an explosive growth in the deployment of virtualisation technology and virtual machines. Although IDC expects that only 25,000 (around 9%) of the 7.4 million commodity servers that ship this year will be virtualised, by 2011, this figure is set to grow to 1.1 million virtualised servers hosting an estimated 5.5 million virtual machines, just in Western Europe. And even this could yet prove to be a highly conservative figure.

IDC has based its numbers on the assumption of a 5:1 physical-to-logical server ratio – current best practice guidance from virtualisation vendors. In the real world, organisations such as Sheffield Hallam University (see box) are already mounting 30 virtual machines or more on a single quad-processor box, and truly major users of Intel-based machines, such as the Qualcomm, are already grappling with managing virtual machine populations of more than 10,000.

Companies like Qualcomm are still exceptional today but they may not be so tomorrow. Analysts such as IDC’s consulting and research director, Chris Ingle, point out that so far most virtualisation deployments have been focused on consolidating physical services to save money – a process that does little to increase the burden of managing logical assets. However, this situation is likely to change dramatically as organisations begin to look beyond consolidation and explore its potential to accelerate new systems deployment.

Raghu Raghuram, VMware’s VP of products and solutions believes this change in customers’ expectations of virtualisation is already underway. Already, he says, “44% of our customers have virtualisation as a standard policy, which means that when a new application is to be brought into the data centre it will be rolled in as a virtual machine.”

As companies adopt this policy they find that many of the traditional barriers to new systems deployment – hardware costs, space constraints, the need for protracted and often disruptive systems testing or simply lack of time and resources – have mysteriously disappeared.

“Virtual machines are simpler than real machines,” says Dave Thornley, manager of support services at Sheffield Hallam University, which means there are far fewer reasons to say no to users’ requests for new services. This is clearly a good thing, but at Sheffield Hallam, it has also meant that, as Thornley puts it: “We have a lot of virtual machines, because we can have a lot of virtual machines.”

Thornley’s observation neatly sums up why experts such as Jackson and Ingle are worried about the future. Previously, in the commodity server world, if a server fails, all that is lost are the services of a single application. In these circumstances, says Ingle, “people have been reluctant to spend money on things like systems management, and have made do with using a spreadsheet and a clipboard to track what they have.”

However, in the virtual world of the future, a single failed server could mean losing a dozen or more business applications. Now, manual methods will not be enough. “You’re hardly going to be able to go around putting labels on virtual machines, you will have to have some level of systems management and systems automation,” said Ingle.

Investing in virtual systems management tools shouldn’t be viewed as an unwelcome if necessary expense. Products like VMware’s Virtual Infrastructure 3 and Citrix’s XenSource Enterprise Server are already transforming their customers’ expectations of what can and can’t be done with commodity server systems – including making it economically feasible for some commodity server users to enjoy the security of high-availability and disaster recovery for the first time.

However, even virtualisation’s staunchest supporters admit that the technology is still in its infancy, and the working practices necessary to harness it effectively are even less mature.

Some vendors have recognised this problem and put considerable investment into the creation of educational resources. Microsoft, for instance, offers extensive guidance on virtualisation practice via its website; VMware has recently launched a Wiki that encourages customers to share their experience of what Raghuram calls the “operationalisation of virtualisation”.

All of these experiences, of course, will differ according to the individual circumstances of different customers and their differing goals. It seems likely though that those with the most successful experiences will have one thing in common – an appreciation of the fact that if organisations truly wish to see the end of today’s chaotic server sprawl, they must make the investment in management tools and practices that will prevent the creation of its virtual machine equivalent.

Flexible benefits of virtual management

Four years ago Sheffield Hallam University faced the same challenge that is now prompting so many other organisations to virtualise their x86-based services today.

“We had two data centres on two sites about four miles apart, and there was this whole server sprawl thing going on,” says Dave Thornley, the University’s Service Support Manager. “We were approaching the power distribution capacity on one site, and we were running out of space in the other, but the University’s demand for IT services was not slowing down. We were going to have to start saying no to people, or we needed to do something different.”

The “different” options open to Thornley left little room for manoeuvre. The network bandwidth costs made using a utility service provider prohibitively expensive, and although Microsoft was then already promoting the use of operating system-level virtualisation in Windows 2003, Thornley was uncomfortable with risking multiple applications running against a single operating system image.

Instead, the University’s experience of using VMware’s Workstation and GSX in test and development work persuaded Thornley to go a step further, and pilot VMware’s ESX Server for business purposes. For 12 months, any business or academic user that asked the IT department to support a new service was given a choice: pay for a new service themselves, or install their new applications as virtual machines free of charge. By the end of the year, there were 30 virtual machines running on the University’s two pilot servers.

Today, Sheffield Hallam runs almost 200 virtual machines – more than half of all its applications – across a physical server estate that has been reduced by a third. And yet now, in this streamlined server environment, capacity has ceased to be an issue.  Thornley estimates that virtualisation has so far saved the University around £350,000 in operational costs – including a massive £43,000 in its electricity bill.

These benefits more than justify the decision to virtualise the University’s server estate, but Thornley has since discovered other advantages to virtualisation that may deliver more benefits in future – such as an infinitely more flexible and robust IT services infrastructure.

Like a lot of other large Intel server users, prior to virtualisation Sheffield Hallam’s systems management was not the most sophisticated part of its infrastructure. “We used something called Big Brother. It was just a Web page with a bunch of alarms on it. It was very simple and very straightforward,” says Thornley, but it was hardly a sophisticated systems management tool.

Virtualisation has obliged the University to increase its systems management investment, but it has been well worth the expense. Using VMware’s Virtual Centre management console, and the VMotion live migration tool has completely transformed the University’s IT infrastructure – and its ability to effectively manage it, says Thornley.

“In terms of VMotion, the flexibility it grants you is just phenomenal,” says Thornley. Routine maintenance that once was highly time-consuming and disruptive is now simply accomplished by migrating workloads from server to server as required. “By pressing a button I can move an application to another server four miles away. When I’ve finished, I press it again and it comes back,” he adds.

Not surprisingly, having made the leap to a virtual world, Thornley can see no reason for ever turning the clock back. “VI3 gives you great systems management, and great business continuity. It wasn’t the reason we did this, but now we absolutely couldn’t do without it,” says Thornley. As virtual machines proliferate, this could be a valuable lesson.