Toby Balfre, vice-president, field engineering EMEA at Databricks, discusses how organisations can get ahead of the National Data Strategy to drive business value
The pandemic years taught the UK Government the importance of harnessing data. Throughout this period, data insights were used to inform public policy and to take swift action – whether this was determining changes to the “traffic light” system on a regional level, or managing vaccine deployment. This influenced the National Data Strategy, laid out in 2020, which seeks to unlock the value of data across the economy. The strategy’s prime focus is to encourage all organisations to modernise the way they use, access and share data.
However, organisations cannot simply click their fingers and be data-driven. Research from the DCMS highlighted several challenges to data sharing across the economy. These include a lack of knowledge around the use and value of data, a general lack of incentive to share data, and a deterrence stemming from the cost of accessing and sharing data. As such, it’s likely some organisations may view the National Data Strategy with some apprehension. It may seem like an unnecessary, expensive pressure after a challenging couple of years, however the future lies in drawing meaningful insights from data. Organisations that don’t recognise this will eventually find themselves on the backfoot. It’s critical, therefore, for tech leaders to carve a path towards embracing the value of data and data sharing.
It all starts with the foundations
Within the policy framework for the National Data Strategy, the first focus is to “establish foundations”. And that is entirely correct, no matter how extensive or simplistic an organisation wants their data strategy to be: whether you’re building a skyscraper or a bungalow, you start with the foundations. The problem is, many organisations currently operate on legacy data architectures, such as data warehouses. The complexity of these legacy structures can make delivering a data strategy challenging – they can cause information silos to form, and prevent data from being easily distributed. Furthermore, inaccurate datasets that contain duplicated or outdated information may be shared, causing larger problems down the line. This may describe the exact nightmare scenario some might fear the National Data Strategy will force them into.
The answer lies in building a strong, modern data foundation from the outset – such as a data lakehouse. A key aspect of the lakehouse is that it reduces the number of different platforms needed, taking away much of the complexity of legacy architectures and making rolling out a data strategy far easier. Lakehouses, built on the data lake and with a single security and governance model for all data, ensure the timely flow of accurate data, easily storing data for analysis, as well as artificial intelligence (AI) and machine learning (ML) use cases. Adopting the likes of a lakehouse can simplify data storage and access, giving teams greater flexibility and governance. The easier the data is to work with, the more open an organisation becomes to expanding its use cases.
Be open to being open
Today, the main barrier to organisations succeeding with their data is challenges around sharing it – which just so happens to be at the heart of the National Data Strategy. There is a growing need to share data with external entities, such as partners and third party organisations. However, there are many frustrations around how data can be shared and, once it has been shared, there are real challenges with maintaining value. Waiting for data to be shared can be slow and, in the time that passes, conditions may change and momentum may stall – rendering the data useless. Data sharing shouldn’t be a barrier to innovation in this way.
Fortunately, we’re seeing a worldwide shift towards open source tools and technology. Open source makes accessing, distributing, reusing and modifying datasets quick and simple. This takes the waiting out of the game, driving up the pace of innovation and allowing data teams to integrate with well-established scientific packages, such as RStudio. As such, being willing to embrace open source is key for any organisation seeking to harness the value of their data. Some may be reluctant, with concerns around security and whether opening up their data is really necessary. Adopting a single platform – such as a data lakehouse – that is able to securely share data in real-time, whilst meeting privacy and compliance requirements, will be key. To assist, tech leaders will again need to focus on encouraging an organisational culture shift to being “data positive”. Embracing open-source will equip data teams with greater flexibility and collaboration, whilst benefiting from the security and scalability that cloud offers.
Take stock of your skills
Finally, having the right skills on hand is crucial when building out a data strategy.
A key benefit of open source architecture is that it makes having access to the right skills much easier, as organisations are effectively tapping into a wider skills pool. It’s also important that organisations look to develop the right skills from within. Many organisations will have a heritage of analytics and statistical skills, which they can look to bridge into modern analytical and data science roles. Here, the role of learning and development will be key, and leaders must engage in a concerted effort to ensure individuals are equipped with the right training to truly harness their organisations’ data. Finally, the power of community to promote peer-to-peer skilling must not be overlooked – indeed, this is what has made open source such a driving force in technology.
Having the right skills in place will also empower tech leaders to drive a strong, business-wide data culture – creating a collective understanding of the value that data can bring and of how to use it to get that value. Throughout the organisation, individuals will be united on the importance of a strong, robust data strategy.
A central aim of the Government’s National Data Strategy is for companies to “lead, cooperate and collaborate” when it comes to data. By embracing these steps, organisations will do just that, and will be primed to better share and access data across their own businesses and with external parties. They will be the trailblazers of the data-driven future, leveraging the National Data Strategy to their advantage.
Written by Toby Balfre, vice-president, field engineering EMEA at Databricks
