Now is the most exciting time for data. In 2015 Gartner finally retired ‘big data’ from several of its ‘Hype Cycles for Emerging Technologies’ and dropped the ‘Hype Cycle for Big Data’ altogether – not because it’s obsolete, but because we have finally moved beyond ‘big data mania’ and into practise.

As the Gartner researchers put it, ‘the characteristics that defined big data, those pesky 3 Vs – volume, variety and velocity – are no longer exotic,’ they are now part and parcel of doing business in the information age.

But just because storing and handling large amonts of data is now commonplace, it doesn’t mean all organisations are necessarily making the best of data as a competitive resource – and that doesn’t make everyone a ‘data leader.’

> See also: Do you have what it takes to be a data leader?

In 2013 and 2014, Information Age’s Data Leadership conference was praised for its ‘360 view’ of data in enterprise by showcasing the innovators and visionaries ahead of the data tide as they unpacked the big data concept in detail, exploring its impact on infrastructure, governance and wider business strategy and transformation.

Having been gathering momentum over the past three years as one of the UK data community’s premiere free to attend symposiums, this November’s event saw industry thought leaders and prominent end-users take the discussion to the next level and delve even deeper into the nuts and bolts of how to design their business strategies, infrastructures and teams to get true value from data.

Those end-users that were pioneering the use of data in their businesses two or three years ago are now able to speak about what they have learnt, and over 200 IT leaders in attendance at Data Leadership 15 were privileged to hear about their journeys firsthand.

Iconic UK retailer Marks and Spencer, European travel operator Eurostar and charity Age UK all brought their data transformation stories to life, as did Charles Ewan, CIO of meteorological organisation the Met Office.

Future-proof your infrastructure

Producing about four million data sets a day from observational platforms around the world, the Met Office is a fascinating example of data processing on a vast scale, as it moves from petabytes to unknown quantities of variable and complex data in an attempt to increase mankind’s understanding of the working of global weather systems.

Ewan talked about how his organisation is facing the struggle against ‘data gravity’: the tendency of data to accumulate together in one place the more it attracts services and applications, making it increasingly difficult to move and creating an exponential ballooning of information in places that were not architecturally designed to cope with such volumes.

‘There gets to a point where data gets so big that it won’t be solved by pushing it into the cloud,’ said Ewan. ‘We are already at that point where we’re running into problems. We’ve carried on doing things the same way, with the same thought process, just doing things at a larger and larger scale. But this is no longer tenable. We’ve had to really sit back and think differently in order to maximise the information payload.’

‘We are redesigning our whole data architecture, contain and decouple strategies to take what is a highly heterogeneous complex architecture (code for a mess) and better segment that in hard information based layers that are decoupled and connected together by loose services, available in their own right as intelligent services.’

The Met Office’s hugely ambitious data strategy relies on the free exchange of observational data across geopolitical boundaries over a long space of time in what is a massive collaborative operation.

‘We are feverishly open,’ said Evans, ‘because we have much more data than we could ever exploit as an organisation.’

Be an open book

As the day’s opening speaker Sir Nigel Shadbolt, co-founder of the Open Data Institute explained, you don’t have to look just to public sector and non profit organisations to see great examples of where opening up data for collaboration is making a real difference – in the commercial world, Elon Musk, owner of Tesla Motors, has made his many patents in electric car technology available for other people to support, suggesting that an open-source platform would be of great benefit to manufacturers and drivers alike by driving the advancement of electric vehicle development.

‘Open data is a fundamental part of the innovation food chain that more and more organisations are exploiting,’ said Shadbolt. ‘Musk wants the ecosystem around technology to be vibrant one, and realises he will make more money that way. There’s more in data that will pull a community together. Toyota is doing a similar thing with its fuel cell technology.’

But this kind of data innovation is ‘not yet a done deal’ added Shadbolt- in the vast majority of cases it is not yet routinely embedded in the workflow of organisations, and there are many fundamental challenges remain unresolved. One of those challenges is locating the data within a broader strategy.

Take ownership

Peter Williams, head of enterprise analytics at Marks and Spencer explained that getting value from data isn’t just a problem of IT and infrastructure. How does a traditional British retailer with more than 130 years of history make use of such an incredible amount of data amassed over time?

‘We know there are problems with data quality, but that doesn’t mean it should be a barrier,’ Williams said. ‘We’ve been happy to run a business for 130 years on this information, so we’re just cracking on, being pragmatic, and using the data we’ve got and are comfortable with as an impetus for data we’re not so comfortable with.’

> See also: The UK’s top 50 data influencers and leaders

Even without perfect data quality, Williams explained how the retailer has been able to put that data at the heart of decision-making by generating a community of analytical people throughout every team, training everyone from IT managers to senior commercial leaders to ask questions, enforcing a sense of ownership throughout the entire organisation.

Through this inclusive approach, M&S has been able to make good on the promise of big data.

Lead the pack

But Richard Lee, managing partner of consultancy IMECS LCC and regular Information Age columnist, argued that the pervasive use of data throughout an organisation cannot happen when the vast majority of CEOs and boards are still ‘analytics illiterate’.

‘What we need is strong, accountable top-down leadership, not more creation of ‘chief whatever officers,’ said Lee. ‘We should not annoint data leaders by proxy, but those on the board should set the tone and direction for a fully-aligned organisational culture. This is the lynchpin of your strategic plan and becoming a ‘predictive enterprise’ able to optimise decision-making, mitigate risk and exploit insights for competitive advantage.’