The market for web analytics software is exploding. As Information Age reported last month (‘Customers in the tank’, August 2006), in the past 18 months businesses have begun investing heavily in software to get a clear, detailed and near real-time picture of how their websites are being used and how they can be adapted to increase sales.

But the rapid corporate adoption of web analytics, and the elevation of the technology into a widely-used corporate platform, is beginning to raise a big technical and organisational challenge for most of those companies that have so enthusiastically embraced it: Would it not be useful to analyse online visitor behaviour alongside the information that other offline channels are generating? And would it not be a good idea to apply the process discipline and analytical power of customer relationship management (CRM) and business intelligence (BI) systems to the all the data?

Take, for example, a financial services company that has been advertising on TV to promote a new investment. The advert gives a phone number and website address. The phone calls are all recorded by the in-bound call operators, and tracked in the CRM system and analysed. But the web traffic data is recorded separately by the online analytics package – and unless it is extracted, merged and analysed, that is where it stays.

And what if there is a hidden premium? What if 15% of the visitors to the website also linger on the pages inviting visitors to open an online account? That kind of information is gold-dust to the marketing director planning his campaigns – and in most cases, it is missed.

Some companies have used integrated CRM and web analytics data to measure the efficiency of their call centres and websites. By merging web activity data with call centre activity logs, for example, it is possible to measure if self-service systems on the web are reducing the need for operators.

As online business matures, it is clear how intertwined online and offline operations are becoming. This is particularly evident in the retail sector: a 2003 study of cross-channel retailing by Neilsen NetRatings, for example, found that customers were 27 times more likely to visit a store if they had a good experience at the online site. And several studies have shown that, while many shoppers visit a store, look at a product and then buy it on the Internet, there is also a huge flow the other way – a product is researched on the Internet, but then bought at a local store.

“Painting an actionable picture of customer behaviour often requires following a surfer from the web to an [toll free] number, a physical store or a call centre,” says IDC in a recent report on web analytics. “Predicting the combination of behaviours that leads to a sale, a cross-sale, or the termination of a relationship can involve multiple data sources.”

Another reason why companies are calling for web analytics to be more integrated: some big investment banks, retailers and telecoms companies are planning to build integrated ‘event-driven’ systems that can react more or less instantaneously to events. One US mobile phone company is looking at exactly this: when it introduces a new low-cost package, it doesn’t know immediately how popular it will be. But if visits to stores, enquiries from resellers and a high level of hits to the promotional area on the website are all recorded as events, then an action – perhaps an alert to increase production – can be triggered as a result.

The rub

Almost everyone involved in web analytics believes there are huge benefits to be achieved through online and offline integration. “Closing the loop between on- and offline, and being able to pass data between the two, is something of a holy grail for a lot of organisations,” says Conrad Bennett, European technical services director of Webtrends, an analytics supplier.

Now here’s the rub: integration between web analytics and BI systems, is difficult, immature and rarely practised. And there are at least several reasons why. The first of these is that that web analytics has a dirty little secret: accuracy. Although analytics systems, by analysing the click rates and web logs, produce reams of very precise data, there are big debates about what should and shouldn’t be measured, and what technologies should be used. Different systems produce different results, and different applications may require different methods.

Before businesses start to integrate, they need to get clean data on both sides. “Organisations are mostly trying to get the web data right – to get decent data – before they starting trying to integrate,” says Neil Mason, principal consultant with analytics consultancy, Applied Insights.

The next problem: most CRM systems, and a lot of web-based ecommerce systems, use personal data – they know who the customer is because he or she told them. That makes it easy to track and model behaviour against, for example, demographic profile and previous history – and it can make personalisation accurate and powerful. But most web analytics systems simply aggregate clicks – they deal in anonymous data. Analysts trying to put the two together are therefore faced with the much greater challenge of trying to produce insights from two very different types of data sets.

But there are other technical problems, too. Most web analytics systems, and certainly those that are offered as a “black box” solution or as an online service, tend to generate proprietary, non-standard and preset reports that are difficult to integrate with other systems. Only in the past two years have suppliers such as Omniture, WebTrends and WebSideStory begun to introduce XML or ODBC interfaces (although some systems, from SAS and Site Intelligence, for example, do use open SQL databases). But all the suppliers agree that much more work is to be done, especially in terms of creating more easily manipulated, event-driven or real-time data output.

In fact, most integration between web analytics systems and, for example, BI systems, uses a very basic but largely manual method – the data is downloaded into a spreadsheet. This is sometimes then reformatted and displayed into marketing portals.

Once technical integration does begin and, for example, web services interfaces between the systems are created, a new architectural issue then arises: Where should the data be held? Should the web analytics system feed the BI systems, or the BI systems feed the web analytics system? Analysts say they have seen both approaches tried, and suppliers of both types of software will encourage the maximum use of their solution, but, to date, no definitive best practices have yet emerged.

Mason of Applied Analytics believes that, while all these technical challenges are impediments to the better use of web analytics, there is perhaps a bigger reason why many organisations have yet to integrate their on and offline analytics: “It’s more about process than technology. Who is going to do it? Whose job is it? And why? Someone somewhere in the business has got to want it done.”

If no one volunteers, then the likelihood is that the need for a tactical real-time site optimisation system will outweigh the need to view and analyse the multi-channel data – and the opportunity to produce some important insights will be lost.

The pressure to integrate, however, will not go away. “My sense is that integration has not been on agenda,” says Neil Mason, citing the technical issues and the problems with data quality. “But it’s starting to become an issue as the web becomes a bigger part of everyone’s business.”

“Why is web analytics in a domain of its own when it’s so important?” asks Opperman of SAS, which advocates pulling web data into a more comprehensive analytics tool. Slowly, the web analytics tools vendors are starting to agree, and are now positioning their tools as interoperable with other systems. That raises the question: Will BI or CRM vendors eventually buy up the  smaller and faster growing analytics companies?

Josh James, the CEO of Omniture, one of the leading pack of analytics companies, believes there is a fundamental difference between aggregating real-time data and carrying out detailed retrospective analysis. But others think some BI and CRM vendors will eventually assimilate the analytic package suppliers, if not the big service providers (whose business is partially about running huge data centres effectively).

Salesforce.com and Omniture? SAP and Webtrends? Business Objects and WebSideStory? Watch this space.