Big data’s been kicking around for a lot longer than most people would give it credit for. But big data in the early noughties wasn’t quite the same beast as it is today, and indeed how it is viewed by the business community.

When businesses talk about big data today, the emphasis has shifted away from simply understanding the technologies, to practical examples about how organisations are putting the technology into practice.

>See also: 6 steps to successfully apply big data for business intelligence

It took a while to move past the hype, but it finally feels like we made it, as the focus for businesses increasingly shifts towards maximising the value of the data they hold.

AI not sci-fi

The ideas circulating around artificial intelligence (AI) have been around since the 60’s. Fast forward to today, and after decades relegated to the world of science fiction, AI has entered back into mainstream discussions about cognitive computing, machine learning and machine intelligence. There are now any number of real world examples – particularly in the financial and retail sectors – of it in action and we have seemingly reached a tipping point.

So why the renewed interest in AI and what’s changed? It comes down to the “three V’s” – velocity, variety and volume and our ability to maximise the outputs of modern processing models, which can provide 10-20x cost efficiencies over more traditional platforms.

It’s also about the availability of commodity hardware that enables the user to do supercomputing for a fraction of what it cost a few years ago. Simply, machine learning and the deployment of AI has become more affordable and therefore more practical.

>See also: How can a business extract value from big data?

You only have to look to Google to see how simple algorithms executed frequently against large datasets often achieve the best results, and the real value of AI is now in its application to high volume repetitive tasks, where consistency is more effective than human intuition.

Data as the life-blood of a business

The usage of data lakes to store all an organisation’s data to maximise its possible uses has gained a lot of attention in recent years, but simply having one isn’t enough to gain competitive advantage. The decision to deploy a data lake should be a business-driven, data lead initiative.

The whole rationale behind a data lake is that storing or categorising data in silos means you potentially miss out on leveraging all of its possible uses but at the same time, identifying all those uses can be challenging.

There needs to be a convergence of historical analytics with immediate operational capabilities to address customer needs, process claims and transmit information to communications devices in real time, and at an individual level.

This merging of analytics and operational applications can be explained through several use cases. Looking at business to consumer e-commerce platforms, many now provide personal recommendations and price checks in real time.

>See also: How to manage big data in the age of digital

Meanwhile, insurance firms need to determine whether claims are legitimate or fraudulent, by combining historical analytics with current information from fast transactional processing systems.

The final example of where this is increasingly prevalent is in media companies, who need to personalise content through set top boxes and OTT (over-the-top) content platforms.

Agility in the data lake

In order to deliver these services, organisations need access to an agile platform that will provide both operational and analytical processing. The value of data isn’t just the provision of answers to any number of historical or immediate questions but in ensuing data processing systems are architected to drive long term value for the business.

As businesses begin to recognise the importance of understanding data in context and taking appropriate actions, processing and analytic models are beginning to provide similar levels of agility.

>See also: Big data is helping reinvent companies through digital transformation

The emergence of converged data platforms enables the same instance of data to be used to support batch analytics, interactive analytics, global messaging, database and file-based models. Not only that, a more agile approach to analytics arises when a single instance of data supports a broader set of tools.

Micro-services delivering real-time results

Historically, the deployment of lightweight micro-services that incorporated machine learning were limited to so-called ‘fast data’ integrations, applied to narrow bands of streaming data.

But we are beginning to see a shift in the usage of micro-services and machine learning towards stageful applications that record information about changes in the state of data caused by events during a session, including those caused by user interaction. This is being made possible where converged data platforms are being deployed, and machine learning capabilities are running on top of the data storage.

Where there is data generated by multiple domains with advanced analytic applications running simultaneously, micro-services become even more critical to the process.

>See also: Using data analytics to improve business processes and reduce waste

These applications can now incorporate key aspects of machine learning to look back at historical data, generate a clearer picture of the context of incoming streamed data, and in turn provide more accurate insights to business to make more coherent decisions.

Data governance vs competitive advantage

It goes without saying that businesses have access to huge swathes of information about their customers and partners, which means the vast majority are walking a regulatory tightrope when it comes to data value and governance. This makes it more important than ever for businesses to manage their data between regulated and unregulated use cases.

While it is mandatory for regulatory bodies to be able to report and track regulated data throughout any transformation process to its original source. While this is a mandatory step for regulatory use cases, this can be limiting for non-regulatory use cases like attaining a 360 degree view of the customer.

>See also: What’s the key to big data and AI being successful?

Here, more effective results come from higher data cardinality, real-time analysis, and a mix of structured and unstructured data sources yields more effective results.

Meaningful business outcomes

Big data has come a long way in the last few years in its ability to deliver meaningful business outcomes across a whole spectrum of industries. Strategic data-based initiatives that have the power to transform the way any organisation operates on a day-to-day basis have become easier to execute and more importantly for the CIO, easier to articulate.

As the value of such deployments becomes increasingly tangible, we’ll see the time to market come down, speeding up the return on initial investments made in big data technology.

Sourced from Martin Darling, vice president, UK, Ireland and Middle East at MapR Technologies