The utopian vision for a business intelligence system has traditionally been a single, megalithic repository of data that supports operational and analytical functions for the entire organisation. This is extremely ambitious, and may explain the notoriously high rate of failure among BI projects.
Today, the idea that a more variegated BI infrastructure might better serve the needs of the various constituents within an organisation is gaining traction. “BI has traditionally applied a one-size-fits- all approach to information delivery,” says Avi Marco, head of corporate business intelligence at UK broadcaster BSkyB. “But we are beginning to realise that in fact there are multiple levels of demand and multiple levels of acceptable latency.”
In the past, Marco says, it has not been possible to build BI infrastructure that reflects the varying requirements within an organisation because the precise nature of those requirements has been difficult to ascertain, especially when scoping out a large BI project. “It is very rare that the business articulates its requirements at the start of a project,” he says.
At BSkyB, this is beginning to change, as Marco’s team has started to apply elements of the Agile software development methodology in BI projects. Marco describes a traditional, waterfall BI project as “very much focused on heavy- duty analysis – you write your integration layer, then you write your database layer, then you integrate the data, and then you build your reporting layer,” a process that may well take months.
In BSkyB’s agile BI projects, by contrast, an operational system is delivered after the first two-week sprint. That system is then refined and enhanced in subsequent two- week iterations.
One advantage of this approach is that it quickly creates an operational system that the business user can play with. They can therefore give immediate feedback on whether or not it meets their requirements. “When they’ve got something in front of them, it’s not an abstract conversation any more,” Marco explains.
This rapid pace of BI development is made possible in large part by recent developments in data warehouse appliance technology. The speed and ease of integration of these tools (BSkyB uses appliances from Netezza, a company that was recently acquired by IBM) allows developers to spin out new data warehouses “on the fly”, Marco says.
The combination of this technology with the agile approach means that potential pitfalls in BI projects emerge much earlier in the development cycle. “What we’d traditionally do is take a tiny subset of data from the source system, write all our code, go into production and then we’d get hit by some data quality issue with the source data,” he explains. “What we’re able to do now is profile the data in week one, so we’ll see some of the quirks in the data and where the quality isn’t good enough very early on.”
So far, the evolution towards agile BI which began in 2009 has made BSkyB’s BI projects more cost effective, and they now deliver business value sooner, Marco says.
He is keen to point out, however, that his team is not using the full “Agile with a capital A” methodology, which is better suited to traditional application development. One point of distinction is that, while in application development individual items of functionality can easily be defined as separate ‘stories’ (the unit of work in the Agile methodology), in BI it is not so simple, Marco says.
“When somebody says, ‘I’d like a report that will show me customer churn by product over the past seven months”, that’s difficult to define as a single story. It has to be broken down into a large number of elements, but we don’t want to talk about those elements with the business. I don’t go out to the user and tell them we’ve integrated a feed.”
Database innovations and agile development techniques are giving rise to a new approach to business intelligence, says BSkyB’s head of BI
But Marco believes that the adoption of an agile approach, in the looser sense, is helping his organisation to work towards a new, more sophisticated model of BI in which the right tools are used in the right circumstances. “This approach allows the requirements to evolve,” he explains.
