Two years ago a survey of European companies discovered that 56% of their CIOs believed they did not have enough storage. Curiously, at about the same time, the consensus among storage industry analysts was that most organisations typically only utilised between 20% and 30% of the storage capacity they already owned. Were the analysts wrong, or were the CIOs just paranoid?

In fact, the analysts were right, and so were the CIOs – to be paranoid that is.

Today, although it is still true that many organisations already own more disk capacity than they routinely use, demand for storage has never been higher. Gartner Dataquest,

for instance, predicts that shipments (in terabytes) of RAID storage systems will increase at a compound rate of 71.4% between now and 2006. Whilst according to the latest Hitachi Data Systems (HDS) Storage Index report, 80% of European CIOs say that their demand for storage will increase this year, with almost 50% of them expecting to increase their own capacity by between 16% and 30%, and another 30% set to buy between 31% and 50% more than last year.

This is the kind of growth that manufacturers of servers – always the principal engines of IT – can now only dream about, and it says something significant about the state of modern information technology practice. For the first time, companies are placing less emphasis on their ability to process data, and more emphasis on their ability to store, manage and retrieve information.

Certainly, the ability to ensure the integrity of and access to information is now considered to be a central tenet of any effective business continuity strategy, and business continuity is now at the top of many CIOs’ list of strategic priorities. According to the HDS Index, business continuity is considered the biggest storage investment driver by 80% of European CIOs, followed closely by data availability, which was highlighted by 78% of CIOs.

The prominence of these two demand drivers is significant because they do more than drive demand for simple storage capacity; they also drive demand for storage area management (SAM), software resource management (SRM) and virtualisation software. These are the technologies that, between them, hold the real keys to ensuring that the right data is available to applications at the right time, whilst at the same time ensuring that data that is no longer immediately important is safely and economically archived.

The latter requirement, in particular, is a matter of growing importance to CIOs. Record keeping and archiving, for so long background tasks that warranted little attention or investment from senior management, have been promoted to strategic issues in many sectors by the tightening net of government and industry regulation. Bills such as the UK’s Data Protection Act or the Financial Services Authority’s regulations, for instance, place a universal obligation on all electronic record keepers to ensure that those records are accurate and secure. More than that, they also demand that they can be recovered when required for inspection by relevant authorities or individuals.

In the case of financial services companies, this can mean holding thousands of hours of recordings of brokers’ telephone conversations, on top of the normal electronic transaction audit trail, and then being able to produce them for scrutiny by the Financial Services Authority, more or less on demand. Other sectors are less rigorously policed, but even then there is heightened need for record keeping which, if it cannot be done cost-effectively, could potentially require whole business processes to be closed down for want of being able to meet regulatory requirements.

The good news for CIOs is that the technologies needed to meet these business demands – such as SRM and virtualisation software – are slowly becoming available. The bad news is that there are a number of technical and managerial challenges to be met before they can welded into an effective and economic storage infrastructure.

On the technical side, the most pressing challenge facing CIOs is to make sense of the competing technologies and product offerings that populate what is still an immature and developing market. The storage industry, under the auspices of the Storage Network Industry Association (SNIA), has made much of its efforts to promote interoperability standards such as the Storage Management Initiative Specification (SMI-S). In practice, however, such standards have yet to be even widely – let alone universally – supported by vendors, and have also failed to inspire much end-user enthusiasm.

At Storage Networking World Expo in April 2004, for instance, end users such as Intel’s CIO, Doug Busch, criticised the storage industry for concentrating on highly technical standards, when what is really required are tools that allow modern storage systems to be more easily aligned with business policies.

Such criticisms are not unique to the storage industry of course. The pressure is on IT vendors of all kinds to make their products more accessible to the business executives who keep increasingly tight hold of the IT department’s purse strings.

At the same time, it is also increasingly important for IT professionals – including those charged with managing storage infrastructure – to align their own thinking with that of their business counterparts.

The latter is perhaps the most important, but possibly also the least widely recognised storage-driven challenge facing CIOs today: defining a new role for ‘storage managers’ both in the context of their own IT departments and the business at large.