The information explosion of the past two decades has touched all walks of life, but it is businesses and public sector organisations that have borne the brunt of it. According to a May 2009 report from analysts at market research company IDC, it is private individuals who create more than 70% of the known ‘digital universe’ of information, but it is organisations that shoulder the responsibility for managing more than 85% of that data.

For CIOs, the most pressing challenge is figuring out how to turn that burden into an advantage, converting structured and unstructured data back into meaningful information and analysis that allow business users to make smarter decisions.

Structured information – data held in fielded form within an organisation’s database systems – has been the traditional focus of most information management efforts. The database itself was the focus of much innovation in 2009. Besides the accelerated adoption of open source databases, a number of technologies began to emerge that promise to greatly improve the performance of databases, and therefore the sophistication of analysis that they are able to support.

These include massively parallel processing, a technique developed by search giant Google, among others, that allows analytical tasks that are too complex for traditional database software to be split up into smaller, more manageable jobs.

Beyond the database, 2009 was a good year for suppliers of business intelligence tools – used to analyse data in such a way as to answer pressing business questions – many of whom saw revenue continue to grow throughout the recession (although just 6.4% of respondents to the Effective IT Survey reported that they had deployed corporate-wide BI projects in the past year, suggesting that departmental projects were the norm). Vendors themselves argue that their revenue growth was due to the fact that companies needed rapid responses to business questions to navigate the economic downturn. There is probably some truth to that, but there is also evidence that widespread industry consolidation also played a large part – in particular, the 2007 acquisitions of Business Objects and Cognos, by SAP and IBM respectively.

These purchases, according to Gartner analyst Dan Sommer, triggered a slew of accelerated migrations and upgrades among jumpy CIOs concerned that they were using BI technologies for which support could be discontinued by new owners. Either way, that left plenty of organisations worldwide with budget to spend on BI, but neither the resources nor the inclination to invest in complex, enterprise-wide BI platforms.

In 2009, some of them started to explore the concept of BI tools delivered using the software-as-a-service (SaaS) model, with many ‘on-demand BI’ companies announcing a steady stream of customer wins. Meanwhile, open source BI from companies such as Jaspersoft and Pentaho also saw a significant uptick in interest, with adoption doubling year-on-year, according to Gartner analyst Andreas Bitterer.  

Certainly, the coming year looks set to be a strong one for the BI industry – 21.4% of respondents to the Effective IT Survey revealed that they intend to do deploy corporate-wide BI in the coming 12 months.

Some analysts believe that 2010 may be the year in which BI vendors start to deliver on their promises of ‘real-time BI’ and ‘predictive analytics’, technologies that aim to draw faster and more instructive analyses from data. The announcement by IBM in July 2009 that it was to spend $1.2 billion on acquiring statistical analysis software company SPSS, which claims its technology supports predictive analytics, signalled that this is an area of potential interest beyond the technology’s current niche of financial services companies. 
 
But despite these advancements, there is little sign that Microsoft Excel spreadsheets will be entirely displaced as the primary tool for interrogating business data any time soon. With Excel 2010, Microsoft will introduce better charting capabilities, closer integration with the SharePoint collaboration platform and the launch of PowerPivot, an in-memory analytic engine that the company claims will enable Excel analysis on millions of rows of data at sub-second speeds.  

Unstructured analysis

When it comes to unstructured data, meanwhile, the challenge for CIOs is arguably even greater. Such data, stored in myriad formats including emails, forms and multimedia files, is estimated to make up around 85% of a company’s total information, but the fact that it is typically spread around in a range of different ‘point’ systems, from a wide variety of vendors, makes it difficult to manage.

To some extent, industry consolidation has made the picture clearer for customers in recent years, as larger vendors such as Oracle and IBM have used tactical acquisitions to build portfolios that encompass document, records, web content and digital asset management tools under the banner of ‘enterprise content management’, or ECM.

That trend continued in 2009, with the purchases of Vignette by ECM company Open Text and Interwoven by search vendor Autonomy. But at the same time, new entrants have emerged, spurred on by interest in open-source and SaaS options for ECM, such as Alfresco and SpringCM respectively.  

In November 2009, Forrester analysts named EMC, IBM, Microsoft, Open Text and Oracle as ‘ECM leaders’ in their Magic Quadrant for the sector, while Autonomy, Day Software, Alfresco and SpringCM were seen as visionaries.

The coming year promises to be an interesting one in ECM. While debate continues to rage over whether Microsoft SharePoint can truly be considered an enterprise-class content management tool – it doesn’t have full capabilities, for example, for records management or eDiscovery – the system is widely used by organisations worldwide, albeit simply as a file-sharing platform.

The SharePoint 2010 beta, released in November 2009, clearly demonstrates that Microsoft aims to strengthen its foothold. And at Oracle’s OpenWorld conference in San Francisco a month earlier, the company announced plans to integrate its ECM document services with its enterprise applications during 2010, although details remain sketchy so far.

Another field of information management expected to gain traction is eDiscovery. Defined as the identification, collection and production of electronically stored information associated with legal and government proceedings, this technology has been widely touted as ‘the next high-growth market’ by companies including Autonomy, Recommind, EMC and ZyLAB for several years, thanks to its ability to reduce the cost of legal proceedings.

So far, their claims have proved to be premature, but Gartner analysts estimate that the market will be worth $1.2 billion in 2010, up from $1 billion in 2009.