Using data to tackle cyber threats and fraud is far from new – basic intrusion detection systems for detecting suspicious patterns in networks have been around for at least three decades. But as data volumes creep up to monstrous volumes, from gigabytes to terabytes, exabytes and beyond, protecting it is proving an increasingly daunting task.

Gartner estimates that the amount of data analysed by enterprise security organisations will double every year through to 2016. As a result of this explosion, businesses are failing to detect security breaches early enough.

But by sitting on their wealth of security data for longer, big data – and the new capabilities that are driving it – could actually become companies’ biggest security asset. According to a recent report by security vendor McAfee, 58% of firms are storing their invaluable security data for less than three months.

‘Organisations must start retaining their security data for longer, and apply analytics to reveal patterns, trends and correlations that will inevitably help them spot and deal quickly with advanced and persistent threats,’ says Raj Samani, EMEA CTO of McAfee. ‘By using analytics, businesses can spot and block trends in real time, and long-term analysis of the vast amounts of security information could also ensure that even dormant threats are found quickly, before they have a chance to do any damage.’

Placed in context

The evolution of big data tools is enabling security analytics to effectively add a level of context and awareness to security incidents that was previously impossible to achieve.

‘By having additional context around threats, security professionals can detect issues that traditional security analytics tools may have missed,’ explains Neil King, VP at big data analytics provider Guavus.

‘For example, by detecting a piece of code that has never been seen before on the network, it stands to reason that there is something suspicious happening. What’s more, integrating big data into security analytics can help prioritise multiple concurrent threats, helping to root out false positives and ensure that IT teams focus their attentions on the most serious threats.’

Big data analytics can also put businesses in a better position to predict attacks in advance by comparing network states before and after attacks. Businesses can identify when, where and how an attacker is most likely to strike and take preventative measures accordingly. These tools can even help identify weaknesses in the network, and alert security teams appropriately.

Out with the old

The reality is that traditional security information and event management (SIEM) tools are just not able to capture unstructured data from all over an organisation that is becoming relevant to enterprise security. This data, which stems from modern security priorities such as web and email priorities, is hard to analyse and make sense of using SIEMs alone.

And as network boundaries are dissolving, companies are opening up their perimeters to partners, suppliers and others, and to staff working from home. Perimeters that were once easy to define and fortify are now vague and transient.

‘The whole Internet of Things scenario that’s emerging is definitely going to multiply those problems,’ says Martin Baldock, London managing director at cybercrime response specialist Stroz Friedberg. ‘We are trying to help our clients think about a risk-based approach in real time. The only way you can do that is to use these complex data-mining and big-data-type techniques. You can’t have a static defence any more because the attack vector changes very quickly now. You need to spot correlations of events, and the only way you can do it is by bringing these sources together.’

Many organisations have traditional ‘point products’ such as previously isolated intrusion detection systems and access control systems, but not a huge number are bringing those together, ‘and that’s really what you need to think about in terms of big data’, says Baldock.

‘It gives you a much richer analysis of what’s going on in real time. Rather than just, “There’s an alert over there on that application, what does that mean?” Instead, they can instantly say, “Is there some connecting event that’s happened?” That’s where people that use big data techniques are beginning to get ahead of the game – in real-time threat analysis.’

McAfee’s survey found that only 35% of firms can detect data breaches within minutes of them happening. On average, it takes ten hours for a security breach to be recognised.

> See also: The 2014 cyber security roadmap

Security guru Paul Nguyen, board member of the Internet Security Alliance, and president of global security solutions at CSG Invotas, stresses the critical importance of bringing detection up to speed. From his experience, it currently takes most organisations days or even weeks from the detection of a breach to deploying actions to challenge it.

‘Of the two dimensions that security teams focus on today, the one where people generally invest the most time and effort is minimising the time to identify an attack,’ says Nguyen. ‘In some cases these attacks go undetected for months, because they’re targeted campaigns that are not as noisy or as prevalent as most attacks are.

‘In the early 2000s, attackers were carrying out big, broad attacks. Now, campaigns are far more surgical, and time to detection is a lot longer than most folks want it to be. A lot of companies can only retroactively investigate how data was lost.’

Building the business case

With the wealth of big data tools available for real-time threat analysis, what is holding most companies back from using them? Many of the issues around big data come back to the new varieties of data and bringing it together from disparate sources across an enterprise, making correlation easier said than done.

Traditional access control systems dealt with a straightforward format: the person trying to gain access, and whether or not they had access. The landscape has changed, however – today’s network traffic is of a very different format and moves around a network at a dizzying velocity.

The value in all analytics is where data is transformed into information, but taming this information can be like coming up against a brick wall, as TK Keanini, CTO of network intelligence specialist Lancope, explains: ‘When you look at security data, you quickly realise that the standards to present this data and the common identifiers across security solutions are all over the map – it’s a mess.’

The irony of this is that those coming up against this problem are mostly large enterprises, which have the most data to gain from, and the most to lose. Although security is very much on everyone’s agenda, it often takes a back seat to things like marketing, business development and sales.

> See also: The risks of ignoring big data

‘It is changing, but it’s a very slow process to get investment into something that isn’t necessarily profit making,’ says Baldock. ‘But ultimately the benefit is the protection of your reputation and brand. Companies need a catalyst, like a data breach that really catches the board’s attention, and then things start to move.’

Human-machine synergy

More organisations are finding that because big data security is a maturing domain, they don’t have the workforce to tackle it, and the complexity of the attacks is increasing much faster than they can train and hire the talent and resources to respond to them.

Nguyen and CSG Invotas are involved in removing the ‘human bottleneck’ that comes with big data analytic security solutions – looking to automated solutions to bridge that gap.

‘We have gone to market with an automated solution that allows you to trigger off analytics to tell you when an attack is occurring, then reconfiguring network defences in real time from a machine-to-machine standpoint, so that you’re containing and closing that gap as quickly as possible,’ he explains.

‘Normally, today, a machine identifies, let’s say, the analytics platform. A human has to then analyse it and reconfigure the machine elements on the other side that are representative of their defences to contain the attack. So the bottleneck becomes a human. We are trying to identify how can we have a human on the loop observing the machine-machine reaction versus being in the middle.’

> See also: Big data, mobile and security pros see highest salary hikes

To achieve risk-based security intelligence, address advanced persistent threats and improve security monitoring, McAfee’s report says businesses need to store and analyse the right information in a way that goes far beyond log management. This has to include automation and proper use of SIEM systems.

‘Without an automated approach and high-performance systems, this is a real challenge,’ advises Samani. ‘By deploying technologies that provide intelligent detection and automated collection, this will give organisations greater external threat and internet user context. Storing and investigating long-term trends should be a priority alongside providing real-time analysis of data, which is essential to derive security value from SIEM.’

However, Lancope’s Keanini argues that the human element in asking analytical questions of SIEM cannot be overlooked. Security analytics to date has been mostly about custom reports and dashboards with minimal interactivity, with simple analytics you just render to screen.

‘With big data,’ says Keanini, ‘there are multiple data sets being combined synthetically, and now there are several ways in which you can perform the synthesis and analytics, forcing a lot more interactivity for the user when done well.’

For the time being, it’s impractical and inadvisable for companies to eliminate SIEM in favour of big data tools – SIEM offers some invaluable capabilities for capturing one class of data and monitoring network traffic, combined with the richer sources of data from across an enterprise for both real-time and long-term analysis. But as Keanini puts it, ‘It’s exciting to watch vendors be forced to innovate or die.’

The value for enterprises lies in not just being able to efficiently collect and store large data sets, but being able to make sense of the data.

Ultimately, adds Keanini, ‘SIEM systems must be big data systems or they will be made irrelevant.’