Traditional security solutions have relied on rules, signatures and blacklists to prevent security attacks. These solutions are effective at finding the 'known bad' but are increasingly ineffective in helping organisations detect the 'unknown bad,' which include most modern, advanced attacks.

These types of attacks are now the norm as hackers are more sophisticated and users are more mobile, increasing the enterprise attack surface. But machine learning allows security analysts to use math and algorithms to detect attacks and risky behaviours that have bypassed traditional detection systems without relying on rules, signatures or blacklists.

Additionally, machine learning is well suited to take advantage of the large volumes of data produced by cybersecurity systems, enabling companies to better defend themselves in an increasingly complex threat environment.

Does machine learning live up to the hype as a panacea for enterprise security?

Machine learning helps organisations with the problems they face when detecting and investigating compromised users and hosts who may be under attack, as well as negligent and malicious insiders.

> See also: Why machine learning will impact, but not take, your job

While machine learning can accurately detect anomalies, despite weak signals and intelligent attackers, it also makes it easier to know if attackers are lurking in an organisation’s network by sifting through the vast amount of data that organisations have, annotating and enriching it, even if suspicious activity is not raised to the level of an alert. For a sector that is severely understaffed, machine learning can also bolster the skills of even the lowest-level security analysts and make them more efficient.

Today, the average time it takes to identify attacks inside a network is more than six months and a vast majority of attackers are bypassing existing detection and prevention systems.

During this time, attackers are using a variety of methods to exploit their presence inside a network, but in doing so are leaving a trail via huge volumes of log, packet and network flow data.

Machine learning automatically analyses these vast amounts of data, shortening the time to detect and investigate attacks before more damage is done, a huge win for customers.

Enterprises can purchase user and entity behavior analytics (UEBA) solutions, which can consume the diverse data that organisations are already collecting such as logs and network traffic, and heavily leverage machine learning to produce user- or host-oriented security insights to help detect and investigate advanced threats and risky behaviors.

Who's doing it

Several enterprises are deploying security analytics solutions to get insights into what is truly happening inside their organisations and many of these solutions are using machine learning to generate results. Google’s machine learning algorithms take in reams of data to enable self-driving cars.

Additionally, IBM’s Watson uses natural language processing and machine learning to reveal insights from large amounts of unstructured data. It’s able to analyse batches of medical data to make recommendations for diagnosis, and most recently is training to learn cybersecurity practices. In each of these instances, machine intelligence empowers a human, who closes the loop.

The market is still early for these solutions and companies should evaluate products in their networks before making an investment. Security analytics is only as effective as the data it can use for analysis.

My suggestion is to explore vendors that have an expansive approach to accommodating diverse data sources for analytics. Organisations should also look for vendors that have a more comprehensive approach to help detect and investigate modern threats that not only solve their immediate use cases, but also can be extended as an organisation’s needs evolve.

> See also: Machine learning set to unlock the power of big data

Ultimately, organisations need to spend time shaping the machine learning output with business context, which will ensure that the results are more meaningful and insightful. This requires analysts to spend time on the system and infuse it with their context and insights.

IT managers should:

Invest in detection and prevention technologies such as next-generation firewalls and log management systems to ensure that a basic security posture is in place.

Identify security challenges that go beyond what detection and prevention technologies can provide. By pinpointing the problems they’re trying to solve such as exfiltration, privilege escalation, abnormal resource access, password sharing or credential theft, organisations can help ensure better outcomes.

Determine what data can be brought into the analytics – it’s about having access to the right data sources (e.g., packets, network flows, logs, alerts, threat feeds, endpoint, etc.), rather than more data sources.

And lastly, be mindful that there are no perfect solutions, and a healthy security posture relies not only on technology, but also on people. Staffing and maintaining a robust security analyst team is just as important as the technology you ultimately deploy.