Faced with ever increasing volumes of data and demand for storage, it’s simple for an organisation to spin up yet another virtual machine, and add or expand a disk to store even more data. Ultimately, however with data growth outstripping IT budget growth, a reactive approach isn’t sustainable and at some point financial considerations will force a more sophisticated response. It’s therefore essential that IT teams take a holistic view, considering solutions that will enable them to gain greater value from their information, and in parallel, avoid risks arising from regulatory compliance and legal claims.

Data management – a growing problem for IT?

Organisations now operate in a climate of vastly expanding, unstructured data sources often generated by individuals – rather than formalised systems. With organisations struggling to maintain and retain the already huge volumes of structured and unstructured data, as time passes, this burden will only become more challenging. Frighteningly, by 2020 the volume of data globally is expected to have grown to 40 zetabytes – an incredible 40 fold increase from 2010.

Less of a problem is 20-30% structured data, held in databases, or systems of record – which by nature is easier to understand, use and interrogate. The real headache is the remaining 70-80% of unstructured data – including documents, email, social media, images, IM and voice. These types of data have a significantly higher noise-to-value ratio, meaning the information involved is harder to exploit and harder still to manage.

> See also: IT’s focus shifts to data management

Critically, IT teams need to be in a position to control this unstructured data – whether it’s for legal hold, recovery or search and access. For teams lacking a well-managed approach, locating specific pieces of information, that date back many years, or across many different people can be nigh-on impossible.

Key considerations

Data policy

Organisations need an IT policy which provides a complete understanding of all their data sources, its relevance and how it will be captured, stored and accessed. Data mapping can be used to catalogue an organisation’s records by business unit, and done properly it provides an in-depth understanding of the nature, type and location of records. These retention practices enable users to centrally identify relevant data sources, in a consistent and seamless way.

Data in the cloud options

Where data is stored and archived is another organisational consideration. Today, an increasing volume of data lives in the cloud and with low cost storage per gigabyte, this option is for many, very attractive. Although organisations have a number of cloud choices, it’s important they select the best option for their specific needs.

Items to consider are features, functionality and degree of control, as typically these are critical to business applications. Data residency is also an important issue due to risk of screening that came to light last year with the NSA and Edward Snowden case. Other considerations are storage cost and management overhead.

Today, our customers are continually evaluating the public cloud model in growing proportions. Concerns over data security, once perceived to be an issue for the public cloud, are diminishing with compliance to industry security standards increasingly the norm. However, the appetite for risk remains low when it comes to proprietary data, and critical line of business applications. Increasingly, we see the demand for a hybrid model – utilising both public and private platforms that together enable businesses to achieve the benefits of both.

> See also: The dangers of data management

Over time, as CIO’s and CTO’s gain more confidence and providers insulate and innovate against external and internal cyber threats, the public cloud platform will dominate, but for today in 2014, the private cloud still has a key role to play.

Data migration

Data migration activity is not for the faint hearted! Although thought to be a job organisations’ can perform in-house, using off the shelf software, data migrations are a veritable minefield, often complex and time consuming, involving extracting, converting and moving huge data volumes. Also, using the right tools and processes are key to ensure migration efforts are defensible and chain of custody is assured.

The whole process of moving data should help organisations transform its data management policy. It’s therefore critical that it’s done correctly by a specialist provider, with a track record of being a safe pair of hands, so company information remains secure, intact and compliant.

Data search and eDiscovery

Organisations need appropriate applications that provide the level of search and eDiscovery capabilities their business demands. This covers simple data search, to more complex legal hold requirements. Unfortunately some organisations are failing to address these crucial areas.

An Osterman report surveying large UK organisations archiving, eDiscovery and content management activities, revealed better storage capacity and faster search are cited as key features that will motivate many organisations to seek new archiving solutions over the next 18-24 months.

Another threat exposed by the survey is the risk many organisations are running as they don’t possess effective processes and systems to swiftly respond to legal claims being made against them. More than 70% of UK-based organisations consider their eDiscovery capabilities to be incomplete, despite the fact that 73% believe that it is important to retain electronic business records for compliance and legal purposes.

> See also: Analyse this: big data

With the growth in new content types, 94% of UK-based organisations surveyed don’t currently archive social media content. This includes employee posts to Twitter, corporate posts to Facebook or content from enterprise social media systems. The research also indicated that only one in five UK organisations surveyed could impose a reliable litigation hold on various types of social media content.

Conclusion

Organisations face a multitude of challenges resulting from the exponential growth in data and it is essential that IT teams consider their current approaches to data management – ensuring that they are scalable, cost effective and compliant.

We urge all organisations to examine the changing information needs of their business and assess how they will adjust the platforms and applications for data storage – to provide anytime, anyplace, anywhere access that the business community will increasingly demand.