Remember all the hype around Information Lifecycle Management (ILM) back in the early 2000s? Once all the rage, it’s gone the way of all IT industry buzzwords and has slipped into obscurity.
It’s rare to hear the term thrown about amongst the IT press anymore, and with automation technology, the cloud and software-defined storage taking the limelight, it would be easy to assume that it’s an old issue.
But the concepts of classifying and managing information and storage over time according to cost and business value have become more relevant than ever as data volumes have exploded – the principles of ILM should still be front and centre for enterprises, it’s just that the focus has shifted.
While we might now call it e-discovery, or archiving, or file analysis, there’s still a strong desire for enterprises to know what they have, where it’s located, how quickly they can retrieve it, and whether it’s in the right place, which can often be the cloud.
As Gartner analyst Garth Landers explains, when we think back to the origins of ILM, when the term first appeared it was really about storage management, and making certain that data was on allocated to the correct tier based on cost and value.
‘Today there is less of an impetus or focus on managing just the storage,’ says Landers- ‘it’s really about the data. So we get quite a bit of interest on a number of data management techniques, processes and technologies that are a result of those ILM discussions many years ago. For example one of the biggest discussions we have with our clients are around retention policies.’
Now 25 years into the digital explosion, enterprises are often keeping data for longer periods of time. As a result, tiering and archiving policy has become increasingly important, as does the correct device and media.
‘Enterprises today have this conflict going on,’ says Landers. ‘They would like retention periods to be shorter, and around areas like email that is starting to happen. Organisations are taking a top-down approach to control content types like email, and are figuring out how long they’re going to keep it for, and are trying to adhere to that.'
> See also: Key considerations in Information Lifecycle Management
But on the other end of the equation, when we look at retention periods and ILM, oftentimes they’re actually becoming longer because IT is hesitant to delete it, because they aren’t clear on its business value.’
Compliance is king
The other thing really driving the renewed need for ILM, or ‘retention management’ as it’s referred to today, is not only the growth of content and the increasing duration that enterprises retain it, but an increasingly complex and demanding regulatory environment.
As we look at the EU and other parts of the globe, there are increasing regulations in areas like financial services, with many different issues emerging around issues like data sovereignty.
‘ILM today is really driven by governance concerns,’ says Landers. ‘Are companies regulatory compliant, and how do they enforce that? Are cloud service providers going to be enforcing your policy, or are you going to be doing that yourself when you have an enterprise hybrid environment? How is that going to occur? That would be one aspect.’
The other, says Landers, would be when an event takes place like an audit or investigation: ‘The major questions have become, how quickly can you respond? At Amazon Glacier speeds, or in minutes, seconds, hours or days? How quickly do you know what you have and where it’s located? How quickly you can retrieve it before responding to those events? The ease of doing that becomes a criteria as people look at retaining data in the cloud for the long term.’
As people prepare for the new regulations in Europe, those issues are likely to become even more relevant to ILM. But Landers sees this as a positive.
‘I think it’s an opportunity for it to add and provide value to the business and the enterprise in terms of becoming an information governance expert and someone that can provide domain expertise and skillsets in those areas,’ he says.
From Gartner’s perspective, this approach is bringing groups of people together who often don’t have much contact unless it’s a calamitous event like an audit or investigation and the organisation is in reactive mode.
‘We see our clients becoming very proactive,’ says Landers, ‘and that means no longer just IT calling Gartner up – it’s IT with an attorney and a barrister or a compliance person on the phone to uncover what the requirements are, the correct technical assessments to make, tools to purchase, and how to move forward, not only working with compliance and legal but also business stakeholders. Because oftentimes how to retain information is a subject activity and crafting these policies can’t be done in a vacuum.’
> See also: Out of service: when yesterday's IT service management meets today's world
Intelligent thinking
Often events like migrations to the cloud, data centre consolidations, mergers and acquisitions provide a catalyst for enterprises to look again not only at their data retention policies and other process questions, but how to clean up their technology environments.
‘The next question becomes how do we automate that policy, how do we move data and how do we do it in a way that is automated and can scale,’ says Landers. ‘So with that in mind we see technology like file analysis software start to really pick up. This technology really provides intelligence to the matter – it can essentially index, classify and report and the reporting provides a focal point for decision making around understanding what I have, where it’s located, knowing how much of it is duplicated, how much is outdated or trivial as well as its business value in some use cases.’
Gartner notes in a recent marketplace report that new storage environments like DataGravity and Qumulo are emerging that can provide intelligent storage management where the analysis is part of the embedded storage, bringing a new level of data-awareness to ILM.
‘This is an emerging, nascent area and one that really combines the file analysis type capabilities and the storage itself at the same time, so it’s sort of a natural progression,’ Landers explains.
Back to the future
As the big data trend has hit organisations, the value of archived data has changed, and questions about the cost of storing all this data have begun to drive ILM discussions again.
‘We see our clients really struggle with data warehouse growth,’ says Landers. ‘They’re keeping data in expensive data warehouses that were never built to handle this kind of growth. We see our clients moving that information and structured data into environments like Hadoop which are very low cost and scalable.’
Hadoop is a nascent area that fulfills many of the economic requirements of archiving, while the boundaries between productive and archive data continue to blur. And by ‘archive data’ we’re not talking about long-buried data from years of file sharing accumulation anymore. Much of it is relatively fresh and still needs to be analysed and used and queried against. This represents a big opportunity, says Landers.
> See also: A virtuous lifecycle: how to do ILM in a cloud world
‘We do see archiving vendors moving into this space. Vendor solutions are not only managing the storage but interfacing with the query and analytics tools, not only providing SQL capabilities but interfacing with tools like Spark. This, we believe, is the next stage in the evolution of archiving.
It might be in its early stages, says Landers, but Gartner clients are looking more and more into environments that combine structured and unstructured data, with Hadoop increasingly on their radars.
‘The key is,’ he says, ‘you can’t simply just throw it into Hadoop, you need indexes, classifications, and the ability to maintain context from that warehouse or database environment to be able to analyse it.’
But the biggest challenge for enterprises in ILM going forward isn’t going to be about technology.
‘There is no shortage of tools out there in the form of data centre management technologies, file analysis, archiving tools, structured data archiving tools – sorting through all that is the big challenge,’ says Landers. ‘The other aspect of it I think is it’s really a collaborative effort, with multiple stakeholders. IT cannot do this in a vacuum.’
