Imagine having a car sat nav that only gave you directions and not traffic information. Worse still, imagine there is a 30 second delay in pinpointing your location. And now imagine, we didn’t have any other information or access to a database to give us points of interest. We take for granted the connectedness; real-time; multiple data types scenarios in our everyday life.

Yet when it comes to business life, many are often still trapped in a world of data silos.

However, an information technology revolution is quietly underway, and it revolves around the data. Unlike the previous great tech shifts like the personal computer, which brought software to the masses, client servers that introduced distributed applications or even virtualisation, with its greater hardware efficiency and flexibility; the data revolution is more subtle.

>See also: Harnessing the data revolution: Future leaders, future skill sets

At the epicentre of this shift is the need to feed the next major technology leap forward of which machine learning is the starting point towards true artificial intelligence. To get there, we need data, and lots of it, however, we are still living in a world where data is still closely coupled to applications.

However, what we are seeing now from pioneers is the idea that data needs to be more fluid. Take Outbrain, a discovery platform that serves up 250 billion recommendations a month or Edwards, a leader in vacuum and abatement that is using sensor data to make manufacturing plants more efficient; both are making data truly independent of the silos that it currently lives within if it is to be exploited to its full potential.

It’s all about the data

At present data tends to fall into three broad types. The most common is the zettabytes of archive data stored served up from databases that include everything from purchase histories to map data. Alongside this are the trillions of packets of transactional data that zip around private and intranet works every second with process requests and responses for tasks such as online banking and supply chain logistics. Added to this trove are real-time data streams that can range from telemetry from cars to the status of heart monitors.

>See also: Data revolution: the gold rush of the 21st century 

Much of the data is highly structured in neat tables and rows but it is joined by increasingly large volumes of unstructured data, like pictures and video, that can often only be interpreted subjectively by a human. In concert, there is a global store of data that is now measured in the hundreds in the Yottabytes.

In this new age, applications are increasingly designed to ingest, transform and analyse vast amounts of data all with different characteristics to create new insights or solve seemingly intractable problems. Yet the issue of how to get data out of one app or device and into another app, device or workflow while working within certain parameters persists.

To this end, a whole ecosystem of technologies has sprung up that aim to make data independent of the current siloed world. Platforms such as Hadoop, Apache Kafka, and NoSQL databases are all, in their own way, abstracting data from underlying dependencies and helping innovators to develop new ways of working.

There are still challenges to overcome, especially as much of the new ML and ultimately AI work revolves around trial and error and in some cases fundamentally changing how we work with data that moves away from the old input, process, output paradigm.

>See also: The top five data trends coming in 2018

Many of the most innovative projects that utilise data in new and exciting ways are still hidden behind closed doors. For example, the major petrochemical company that is examining oil production, supply, demand and even weather patterns to find better ways of globally routing huge super tankers; or the logistics company running predictive order scheduling to more effectively stock and organise its warehouses to speed up delivery times.

Both are examples of MapR customers that are delivering competitive edge but not solely because of our technology – but because they have embraced the new data revolution. Just don’t tell anybody.

Sourced by Pinakin Patel, head of Solutions Engineering, MapR

Nominations are now open for the Women in IT Awards Ireland and Women in IT Awards Silicon Valley. Nominate yourself, a colleague or someone in your network now! The Women in IT Awards Series – organised by Information Age – aims to tackle this issue and redress the gender imbalance, by showcasing the achievements of women in the sector and identifying new role models