“Prevention is the daughter of intelligence,” said the famous English poet and navigator Walter Raleigh four centuries ago. But the explorer couldn’t predict that the intelligence he was talking about would one day be artificial intelligence. Indeed, AI has become a reliable ally in preventing unwanted outcomes, thanks to the anomaly detection and forecasting capabilities of its sub-branch known as machine learning.

But what are these powers based on, and how can they be leveraged in various scenarios and AI use cases? Let’s find out.

Cyber detectives searching for clues

When detecting anomalies, the typical way to go in many business areas was traditionally based on predetermined rules. For example, a fraud detection system could spot suspicious card payments which greatly exceeded a spending threshold. The main problem with this approach is its lack of flexibility, given that the set of rules must be continuously updated to cope with ever-evolving scenarios, such as anomalous activity due to a new type of malware. 

Here lies machine learning’s full potential. Any system fuelled with this technology can digest enormous datasets, autonomously identify recurring patterns and cause/effect relationships among the data analysed, and create models portraying these connections. In addition, when properly trained, such models will be capable of processing additional data to make predictions, further refining their skills through experience as they consume more and more information.

Norm vs. anomalies

The recurring patterns that ML models discover by rummaging through data are the “norm”. But what if the system runs into data that does not fit any existing pattern among those previously identified? Well, it’s likely to be an anomaly or an outlier.

Anomalies are typically classified into three archetypes:

• Point anomalies: An individual data instance is anomalous with respect to the rest of the data, such as a suspiciously high-sum transaction.
• Contextual anomalies: The anomaly is context-specific. For example, an increase in network traffic during the night.
• Collective anomalies: A set of data instances that may not be anomalous by themselves, but look suspicious as they occur together, such as a sharp spike in login attempts or a series of unusually expensive purchases.

We can train a machine-learning system to identify the aforementioned anomalies as well as patterns and relations among data in different ways. The most common are:

• Supervised learning: We provide the machine learning system with already labelled data, which is data that has been previously prepared and labeled as “nominal” or “anomaly”.

• Unsupervised learning: This is the approach to take when we don’t know exactly what we are looking for because we are faced with an unknown scenario. Therefore, we do not provide labeled data and let the system define eventual categories and relationships.

Whether trained via supervised or unsupervised learning, the advantage of deploying these solutions for anomaly detection is that they don’t require pre-compiled sets of rules and are very adaptive, as machine learning systems can learn over time and fine-tune their models with new data. Let’s have a look at three real scenarios in which we can harness these capabilities.

1. Fighting fraudsters and cybercriminals

The “dark side” of the massive shift towards a fully digitised economy was the exacerbation of pre-existing fraud trends combined with a steady rise in new forms of fraud and cybercrime.

According to PwC‘s 2022 Global Economic Crime and Fraud Survey, 51 per cent of surveyed companies experienced fraud or economic crime in the previous two years, which is the highest level in the last 20 years. 

Considering the growing importance for companies and public institutions to protect their sensitive data, networks, and financial assets, it should come as no surprise that machine learning’s anomaly detection capabilities have been widely leveraged to prevent and counter cybercrime.

Implementing machine learning in fraud detection aims to scan data regarding money transactions or interactions with applications, platforms, and corporate networks to probe behaviours that look out of the ordinary. These anomalous actions may be signs of potential criminal attempts and can be flagged by the system to require an additional human inspection.

For example, hackers may try to violate corporate systems or networks to compromise or steal assets and data. A machine learning-powered intrusion detection system (IDS) using network behaviour anomaly detection (NBAD) can deal with similar attempts by tracing any atypical event, such as coordinated access via multiple accounts provoking a spike in traffic volume and bandwidth, and flag it as a potential cyber attack.

>See also: Fraud rings scaling attacks around the clock — Onfido research

2. Improving medical diagnostics

Deploying machine learning solutions is not just a matter of business but, literally, of life and death. Medicine is undoubtedly a sector in which the predictive capabilities of machine learning and automated visual inspection can shine. In fact, these forecasting powers allow physicians to quickly identify patient condition anomalies that could be clues to upcoming health complications (including brain aneurysms and tumors) and set up preventive measures before it’s too late.

This can be achieved by training machine learning systems with physiological data from previous clinical cases. The algorithms will process such information, spot recurring patterns related to standard or non-standard health conditions, and build a model capable of recognising any sign of deviation from the norm.

For example, a 2021 report showed how a deep learning-based system powered by convolutional neural networks (CNN) detected 95 per cent of skin cancers, compared to 89 per cent identified by human dermatologists.

As for the speed of diagnoses, it’s worth mentioning a project carried out by the InnerEye team at Microsoft in collaboration with the UK’s NHS and focusing on anomaly detection for diagnostic radiology. The project developers created a machine learning-based system to automate the tracing of tumors and other anomalies in 3D radiological images in order to speed up radiotherapy and surgery planning. As a result, this solution accelerated the overall radiotherapy planning process by 13 times.

3. Enhancing maintenance operations

The forecasting powers of machine learning based on anomaly detection, combined with the growing adoption of computer vision in manufacturing, have significantly boosted condition monitoring and predictive maintenance. Such capabilities can be applied in real estate to keep an eye on a property’s electrical system and in manufacturing for industrial asset management.

These techniques involve collecting via sensors, and processing through machine learning systems, a massive amount of data regarding the typical functioning of machinery, power grids, and other industrial components. Once the systems are trained, they can easily spot any shift from the ideal performance of this equipment and send an alert, as such anomalies may involve an impending failure.

A successful example of machine learning-based anomaly detection for predictive maintenance comes from San Diego Gas & Electric. This public utility company faced a widespread energy leakage problem. The utility’s system failures became increasingly common due to the age of the transmission and distribution lines. As a result, an ML solution preventively detected high-risk T-splices, which were the main cause of malfunctions.

>See also: Shadow factories: unlocking manufacturing capacity

A proactive approach to risk management

A real strength of machine learning is that it enables humans to predict and proactively address potential dangers instead of dealing with them when the damage has occurred. As we’ve seen, machine learning-based anomaly detection has proven to be a valuable tool for protecting physical and digital assets and, most importantly, helping save lives.

Andrey Koptelov is innovation analyst at software engineering company Itransition.

Related:

Tech leaders need a new approach to maximise AI value — Discussing the need for tech leaders to adopt a new approach when looking to maximise the value of AI.

Tech Leader Profile: the keys to strong AI leadership — Shaun McGirr, EMEA Director of AI Strategy at Dataiku, spoke to Information Age about how he goes about leading a strong AI strategy and empowering customers.