Dietmar Rietsch, CEO of Pimcore, identifies best practices for organisations to consider when managing modern enterprise data architecture
Time and again, data has been touted as the lifeline that businesses need to grow and, more importantly, differentiate and lead. Data powers decisions about their business operations and helps solve problems, understand customers, evaluate performance, improve processes, measure improvement, and much more. However, having data is just a good start. Businesses need to manage this data effectively to put it into the right context and figure out the “what, when, who, where, why and how” of a given situation to achieve a specific set of goals. Evidently, a global, on-demand enterprise survives and thrives on an efficient enterprise data architecture that serves as a source of product and service information to address specific business needs.
A highly functional product and master data architecture is vital to accelerate the time-to-market, improve customer satisfaction, reduce costs, and acquire greater market share. It goes without saying that data architecture modernisation is the true endgame to meet today’s need for speed, flexibility, and innovation. Now living in a data swamp, enterprises must determine whether their legacy data architecture can handle the vast amount of data accumulated and address the current data processing needs. Upgrading their data architecture to improve agility, enhance customer experience, and scale fast is the best way forward. In doing so, they must follow best practices that are critical to maximising the benefits of data architecture modernisation.
Below are the seven best practices that must be followed for enterprise data architecture modernisation.
1. Build flexible, extensible data schemas
Enterprises gain a potent competitive edge by enhancing their ability to explore data and leverage advanced analytics. To achieve this, they are shifting toward denormalised, mutable data schemas with lesser physical tables for data organisation to maximise performance. Using flexible and extensible data models instead of rigid ones allows for more rapid exploration of structured and unstructured data. It also reduces complexity as data managers do not need to insert abstraction layers, such as additional joins between highly normalised tables, to query relational data.
Data models can become extensible with the help of the data vault 2.0 technique, a prescriptive, industry-standard method of transforming raw data into intelligent, actionable insights. Also, graph databases of NoSQL tap into unstructured data and enable applications requiring massive scalability, real-time capabilities, and access to data layers in AI systems. Besides, analytics can help access stored data while standard interfaces are running. Enterprises can store data using JavaScript Object Notation (JSON), permitting database structural change without affecting the business information model.
2. Focus on domain-based architecture aligned with business needs
Data architects are moving away from clusters of centralised enterprise data lakes to domain-based architectures. Herein, data virtualisation techniques are used throughout enterprises to organise and integrate distributed data assets. The domain-driven approach has been instrumental in meeting specific business requirements to speed up the time to market for new data products and services. For each domain, the product owner and product team can maintain a searchable data catalog, along with providing consumers with documentation (definition, API endpoints, schema, and more) and other metadata. As a bounded context, the domain also empowers users with a data roadmap that covers data, integration, storage, and architectural changes.
This approach significantly reduces the time spent on building new data models in the lake, usually from months to days. Instead of creating a centralised data platform, organisations can deploy logical platforms that are managed within various departments across the organisation. For domain-centric architecture, a data infrastructure as a platform approach leverages standardised tools for the maintenance of data assets to speed up implementation.
3. Eliminate data silos across the organisations
Implications of data silos for the data-driven enterprise are diverse. Due to data silos, business operations and data analytics initiatives are hindered since it is not possible to interpret unstructured, disorganised data. Organisational silos make it difficult for businesses to manage processes and make decisions with accurate information. Removing silos allows businesses to make more informed decisions and use data more effectively. Evidently, a solid enterprise architecture must eliminate silos by conducting an audit of internal systems, culture, and goals.
A crucial part of modernising data architecture involves making internal data accessible to the people who need it when they need it. When disparate repositories hold the same data, data duplicates created make it nearly impossible to determine which data is relevant. In a modern data architecture, silos are broken down, and information is cleansed and validated to ensure that it is accurate and complete. In essence, enterprises must adopt a complete and centralised MDM and PIM to automate the management of all information across diverse channels in a single place and enable the long-term dismantling of data silos.
4. Execute real-time data processing
With the advent of real-time product recommendations, personalised offers, and multiple customer communication channels, the business world is moving away from legacy systems. For real-time data processing, modernising data architecture is a necessary component of the much-needed digital transformation. With a real-time architecture, enterprises can process and analyse data with zero or near-zero latency. As such, they can perform product analytics to track behaviour in digital products and obtain insights into feature use, UX changes, usage, and abandonment.
The deployment of such an architecture starts with the shift from a traditional model to one that is data-driven. To build a resilient and nimble data architecture model that is both future-proof and agile, data architects must integrate newer and better data technologies. Besides, streaming models, or a combination of batch and stream processing, can be deployed to solve multiple business requirements and witness availability and low latency.
5. Decouple data access points
Data today is no longer limited to structured data that can be analysed with traditional tools. As a result of big data and cloud computing, the sheer amount of structured and unstructured data holding vital information for businesses is often difficult to access for various reasons. It implies that the data architecture should be able to handle data from both structured and unstructured sources, both in a structured and an unstructured format. Unless enterprises do so, they miss out on essential information needed to make informed business decisions.
Data can be exposed through APIs so that direct access to view and modify data can be limited and protected, while enabling faster and more current access to standard data sets. Data can be reused among teams easily, accelerating access to and enabling seamless collaboration among analytics teams. By doing this, AI use cases can be developed more efficiently.
6. Consider cloud-based data platforms
Cloud computing is probably the most significant driving force behind a revolutionary new data architecture approach for scaling AI capabilities and tools quickly. The declining costs of cloud computing and the rise of in-memory data tools are allowing enterprises to leverage the most sophisticated advanced analytics. Cloud providers are revolutionising how companies of all sizes source, deploy and run data infrastructure, platforms, and applications at scale. With a cloud-based PIM or MDM, enterprises can take advantage of ready-use and configured solutions, wherein they can seamlessly upload their product data, automate catalog creation, and enrich it diverse marketing campaigns.
With a cloud PIM or MDM, enterprises can eliminate the need for hardware maintenance, application hosting, version updates, and security patches. From the cost perspective, the low cost of subscription of cloud platforms is beneficial for small businesses that can scale their customer base cost-effectively. Besides, cloud-based data platforms also bring a higher level of control over product data and security.
7. Integrate modular, best-of-breed platforms
Businesses often have to move beyond legacy data ecosystems offered by prominent solution vendors to scale applications. Many organisations are moving toward modular data architectures that use the best-of-breed and, frequently, open source components that can be swapped for new technologies as needed without affecting the other parts of the architecture. An enterprise using this method can rapidly deliver new, data-heavy digital services to millions of customers and connect to cloud-based applications at scale. Organisations can also set up an independent data layer that includes commercial databases and open source components.
Data is synchronised with the back-end systems through an enterprise service bus, and business logic is handled by microservices that reside in containers. Aside from simplifying integration between disparate tools and platforms, API-based interfaces decrease the risk of introducing new problems into existing applications and speed time to market. They also make the replacement of individual components easier.
Data architecture modernisation = increased business value
Modernising data architecture allows businesses to realise the full value of their unique data assets, create insights faster through AI-based data engineering, and even unlock the value of legacy data. A modern data architecture permits an organisation’s data to become scalable, accessible, manageable, and analysable with the help of cloud-based services. Furthermore, it ensures compliance with data security and privacy guidelines while enabling data access across the enterprise. Using a modern data approach, organisations can deliver better customer experiences, drive top-line growth, reduce costs, and gain a competitive advantage.
Related:
How to get ahead of the National Data Strategy to drive business value — Toby Balfre, vice-president, field engineering EMEA at Databricks, discusses how organisations can get ahead of the National Data Strategy to drive business value.
A guide to IT governance, risk and compliance — Information Age presents your complete business guide to IT governance, risk and compliance.
