Data warehouse automation is one of those inventions that is so early on and so innovative that only early adopters and visionaries have taken the leap of faith. The rest of the business world has either not heard of automation or mistrusts something that looks too good to be true. But just because it looks too good to be true doesn't make it false.

A monk writing out copies of the bible with his quill might have thought a typewriter was too good to be true, who knows what trickery he would have imagined was contained in the subsequent development of word processors, computers and iPads?

The printing press modernised and revolutionised the world of print. Now data warehouse automation is revolutionising data warehousing with similar results; financial savings, gains in speed, efficiency and accuracy. But before we look at the future of data warehousing, let's go back to the eighties when data warehouses were created, to see why it's time to move on.

The problems of traditional data warehouses

The traditional way of building data warehouses is snail slow. ETL coding is written by hand so it takes months to build the data warehouse which is often out of sync with requirements by the time of deployment. It’s like a man taking years to restore his two-seater convertible to have it ready just as his wife announces she’s pregnant with twins.

> See also: Five reasons why automation promotes collaboration

The sad fact is that the true value and capabilities of the data are rarely understood until the data warehouse is built, but by then it is too late. The warehouse, if it isn’t abandoned before completion, is an expensive, inflexible disappointment.

Perhaps this is why many technologists and thought leaders are ready to declare the data warehouse dead – no longer relevant in the age of big data. But these prognosticators are mistaken. Big data can extend and enrich a data warehouse, but cannot replace it. It is not data warehouses that are dead, but the traditional way of designing and building them.

Getting your data warehouse right

At its core, the data warehouse integrates critical and valuable enterprise data that is not found in big data sources and that continues to be the primary data resource for descriptive, prescriptive and decision analytics. It serves as corporate memory, collecting the body of history that makes time-series and trend analysis possible.

The data warehouse also organises and structures data to make it understandable and useful for consumption by many different business stakeholders. This business intelligence gives organisations the edge, making them more competitive, more customer focused, more profitable.

Data warehouse automation, the future of the data warehouse

Data warehouses will be needed for the foreseeable future, but they need to quicker to build and at reasonable cost, readily adapt to changing requirements and be responsive to business and technical change. And all of this has to occur without compromising solution quality.

Enter data warehouse automation, the future of the data warehouse. Data warehouse automation delivers quality and effectiveness through the ability to build better solutions; solutions which meet real business requirements.

With data warehouse automation the business can make changes much later in the development process and change can occur more frequently with less disruption, waste and rework. This efficiency is not only a joy, it saves time, resources and money. In a traditional data warehouse build, it is especially difficult to get complete and correct requirements due to the linear development process. Automation also brings quality benefits through standards enforcement and standardising the development processes.

> See also: Automation software: the 'universal remote' for enterprise IT

The agility of the automated data warehouse is not limited to its ability to change in the warehouse development process, it can also handle changes in business requirements. Responding to change in real time and without the delay of lengthy projects is the essence of business agility.

Build better and faster

Speed is the critical factor that enables agility. The ability to generate quickly and to regenerate equally fast when change occurs are fundamental automation capabilities. The ability to fail fast is also important.

Sometimes warehousing teams can’t deliver what the business needs due to data unavailability, data quality issues, or elusive and difficult to define business rules. Discovering these issues as early as possible reduces waste in time and resources.

Ultimately, building better, building faster, and changing quickly when needed bring substantial cost savings to data warehouse development, operation, maintenance and evolution.

Beyond developing and changing a data warehouse, automation offers many technical benefits that contribute to extended lifespan and ease of operations for the warehouse. Consistency of components in a data warehouse is improved through the ability to build in standards and conventions. Automated documentation capabilities ensure comprehensive documentation that stays in sync with the implementation.

Impact analysis for planned changes is supported with extensive metadata capabilities and testing is simplified with test automation during development and as a validation capability of operations processes. Maintenance becomes easier with improved consistency, better documentation, simplified testing, version control, automated implementation of changes and standardised deployment methodology.

> See also: CIO as maestro: how automation is turning the CIO into a business visionary

So, how does it all work? How is the process automated? Well, it’s all about patterns.

Design patterns are fundamental to data warehouse automation. Identifying and reusing patterns is central to the ability to achieve consistency, quality, speed, agility and cost savings simultaneously.

Design patterns encapsulate architectural standards as well as best practices for data design, data management, data integration and data usage. Applied patterns in a data warehouse automation tool support the goals of accelerated design and development, but equally importantly they drive compliance with standards and consistency of data warehousing results.

The Business Intelligence that data warehouses enable business to mine is invaluable. It keeps business competitive, profitable and ahead of the crowd. And now, the slow, painful, out-dated machines that attempted to deliver these reports have been revamped, modernised and automated.

Sourced from Rob Mellor, general manager, WhereScape Mainland Europe