Data is at the heart of everything that we do, and that trend isn’t slowing down, especially with the pandemic accelerating digital transformation and refreshing the ways that companies think about their data. According to McKinsey, the digital offerings for companies around the world has leapfrogged seven years of progress during the pandemic, as businesses quickly moved their operations online, and millions of people were forced to work remotely. This surge in digital assets and processes means that more data is being amassed than ever before and data analytics is playing a key role in helping to shape businesses.
The value of data in today’s world
The variety of data sources available today makes it possible for organisations to surface trends and provide predictions about their business through data analytics that were otherwise considered not possible. It’s these data-driven organisations that are seeing success in today’s digital world. Investments in the data analytics market are set to reach $103 billion by 2023 as organisations demand more valuable and actionable insights from their data. However, while organisations seek to find the right tools and solutions to their data challenges, there are also human complexities involved in data management that create barriers for success.
Hot topics and emerging trends in data science
Introducing the ‘Data Mesh’
Data mesh is a groundbreaking new approach for overcoming these barriers. Similar to many software engineering teams that have transitioned from monolithic applications to microservice architectures, the data mesh is, in many ways, the data platform version of microservices, except that it is as much about removing human bottlenecks as it is technical ones. Envisioned by Zhamak Dehghani, a principal technology consultant at ThoughtWorks, data mesh recognises that data is naturally decentralised within an organisation and that, contrary to all previous data warehouse thinking, this decentralisation is in fact a good thing!
Rather than having a centralised monolithic platform for all of your data, the data mesh considers each group of human experts that manages a particular dataset as a “domain”, who are responsible for producing “data products” that are then consumed by anyone in the organisation in a self-serve manner. At the core of this philosophy is a distributed architecture where each domain has its own data product owners, ultimately allowing the company to achieve greater analytical velocity and scale.
The future of data management
Data mesh is the new path to data management. With data volumes set to increase at an annual growth rate of 19.2%, it is critical that organisations of every size reconsider their strategy for building a data architecture that can stand the test of time. The data mesh offers a framework for companies to democratise both data access and data management by treating data as a product, curated and governed by the domain experts themselves. If you are concerned about the scalability of the data warehouse model, a data mesh approach is worth serious consideration.
