To compete effectively, large organisations need to extract actionable insights from all their data faster and more accurately than ever.

However, they are usually hampered by complex IT estates which make the untangling of this data spaghetti expensive and difficult, typically the data is taken from 20 to 100 different platforms.

The result? Datasets fail to reconcile, new data feeds take months to establish and different departments duplicate one another’s work (inconsistently).

So confidence in the quality of data is low, hindering business action from insights.

Analysts entering a data warehouse may find as many as 5 or 6 different versions of a metric they are seeking, since the analyst does not know which version to trust or is most suitable, he creates yet another metric, so typically 80% of his time is eaten up wrangling data – making it usable.

>See also: Top 8 trends for big data in 2016

A massive source of duplication and a waste of valuable time that should have been spent producing actionable insights.

An organisation operating in this environment has no single source of the truth, leading to mistrust as departments build their own data marts in order to get on with their own work.

At the same time, projects frequently over-run both in terms of cost and time, stacking up overheads.

Metadata is the key

Now, a new approach is enabling businesses to save millions of pounds by integrating data faster, better and more cheaply than ever, irrespective of technology.

This is the metadata driven estate (MDE), generating and authenticating insights through the use of metadata in a managed hub that spans all platforms.

In simple terms this means that MDE unpicks the complexity of the “spaghetti” to produce data lineage showing exactly where a metric has come from and how it is calculated, whether it is in any sense polluted and who is using it.

MDE gives the analyst an easy to use search box to find all available metrics appropriate to his subject, the lineage and usage of each so that he easily decides which to choose.

>See also: The 3 pillars of big data analytics potential

MDEs automation ensures all the data is quality-managed tracking DQ issues on each data refresh, this improves confidence among analysts and end-users, across all data management platforms including Hadoop.

Slashing costs, boosting speed

MDE is implemented as standard fixed fee service packages, typically 4 weeks in duration, providing a step-change in performance and as much as 75% reduction in cost.

The process commences with a discovery phase of 4 weeks to automatically ingest metadata from the existing estate, generating data lineage to show how data is used and identifying duplication and areas of poor quality.

Data integration using this approach is faster than traditional hard-coded data feeds, typically MDE new data feeds are prototyped during the 4 week discovery, then a 4 week design/build/test phase.

This new speed, flexibility and reliability will transform the provision of analytical insights which in the digital era must power any type of business-as-usual reporting and underpin exploratory insights for “what-ifs” and strategic decision-making.

As well as producing actionable insights in a fraction of the normal time and cost, the metadata-driven approach can also liberate premium storage for high-value use.

>See also: What are the numbers, facts and figures behind big data?

High-performance access is provided only for those needing it, based on the patterns of usage that have established, with lower cost platforms used (including Hadoop) for lower value requirements, all within a single integrated hub.

The result is a further substantial cost-saving.

Freedom to improve

Yet the greatest benefit of all for a large organisation implementing this approach is that it can undertake data tasks today that would previously have been unthinkable because of budget constraints.

By rationalising the use of data, the metadata-driven approach reduces operating expenditure from the tens of millions of pounds to hundreds of thousands.

Instead of spending 80% of their time finding the data and working it into a useable form, analysts can now devote almost the entirety of their time on a project to developing deep insights that are required by the business.

>See also: Machine learning set to unlock the power of big data

Using a metadata driven approach to populate databases instead of writing software, cost savings of 75% have been proven.

Once the data is in its target database, the analysts no longer have to toil away, searching for the data they need.

Instead they can focus time and attention on driving business improvements and delivering real value faster than ever.

In short, the business can finally deliver on their vision of a single source of truth, and this does not require the data to sit on the same platform.

Sourced by Elaine Fletcher, professional services partner, Teradata