The desire to track the actions and reactions of visitors to web sites has led to a burgeoning market in web analytics software. Analysts at Aberdeen Group say the market is tripling in size each year and will be worth €4.5 billion by 2004.

The software is, in fact, relatively simple. Web analytics analyses web server logs to determine which pages visitors look at, how long they look at them, how often they come back, where they go next and what they request from the site – a technique known as 'clickstream analysis'.

That information can signal to companies when and where they need to modify the construction and/or content of their site. The goal, of course, is to hold onto visitors for longer and, more importantly, make them buy more. But while web analytics software can pinpoint when a visitor leaves, the current technology can't offer any guidance on why or suggest what may have prevented it.

A number of vendors are, therefore, working on a new offshoot of web analytics that promises to offer a better view of visitor psychology – the why behind the what – by employing psychological profiling techniques similar to those used in criminal investigations. By studying how visitors interact with a site over time, they argue, web site operators can build up a much more meaningful profile of their visitors.

Building a profile
One such vendor is UK-based Applied Psychology Research (APR). At the heart of the company's software is technology devised by the company's founder and chief technology officer, Daniel Brown, while he was a psychology researcher.

What is different about APR's approach, says Brown, is that its software uses profiling techniques to make generalisations about people that apply over time and with different levels of confidence. Not only do you need to capture some relatively straightforward and specific information about a person, such as what they say they want in a typical search enquiry, he says, but it is also important to build a profile of a person in terms of knowing the sort of thing that they want, rather than a specific, one-time example. "The aim is to dig deeper into customer behaviour," he says.

Site Intelligence and its UK-based competitor, NCorp, are touting a similar solution, using both clickstream and search engine analysis to observe/predict the interactions of web site visitors. For example, UK car magazine Auto Trader uses NCorp's Ijen software on its web site to understand what is important to individual customers by analysing the searches that customers make and the results that appear to interest them. "We can find out if people are price-sensitive or if they prefer prestige German cars, for instance," suggests NCorp CEO Nick Bidmead.

Davisor, a company based in Espoo, Finland, meanwhile, offers a Java-based profiling system for inclusion in other developers' software. It uses natural language processing, probability techniques and database classifications to create profiles of people viewing specific documents. But it also takes account of how much 'interest' the visitor seems to show in the document.

But does it work?
The profiles created by these programs can create good predictive models of future visitor behaviour. They also have the advantage of being anonymous – something that will appeal to privacy advocates.

Profiling – which does not require any knowledge of personal data other than the most general kinds, such as gender or age – is a far less worrying idea for the average consumer and far less likely to clash with European privacy laws. As APR's Brown muses: "Do I have a right to see not just what data a business has about me, but what it thinks of me? It's an interesting philosophical question."

All that said, profiles still cannot define exactly why a visitor looked at a particular document or piece of content, nor can they gauge their mood at the time. For this reason, visitors can appear to give out mixed signals. "At a trivial level, you might be searching for information about your work at one time and searching for information about your home life at another," says APR's Brown. "At those different times, you are more different from yourself [sic] than other work colleagues."

There is little indication that this can ever be overcome. The key limitation to understanding a visitors' behaviour at a specific instant is the lack of good information available to the profiling/analytical software beyond textual interactions and web logs. Current research, according to London-based speech therapist Sarah Barry, suggests that British and US natives use non-verbal communication such as tone of voice and body language to convey respectively 70% and 80% of information in conversations. That leaves only 20% of information conveyed by the words used. In southern European countries, where body language is even more prevalent, this figure is closer to 10%.

Teaching computers to recognise particular emotions and even to mimic them – so-called 'affective computing' – acknowledges, almost without exception, that correctly determining someone's emotions without being able to see, hear or touch that person is almost impossible. This generally requires the input of additional hardware, such as IBM's temperature-sensitive 'Emotion' mouse.

Rosalind Picard, the founder and director of the Affective Computing Research Group at the Massachusetts Institute of Technology Media Lab, acknowledges that most of the emotions that people feel are not going to be picked up by technology. The Internet essentially remains an emotion-free medium. "Autistics, who typically have impairment in recognition of emotion, have commented that they love being on the web because it levels the playing field for them," she says. "In a sense, everyone is autistic online."

She adds: "With the exception of gifted poets and others who work hard to lessen the ambiguity of the emotions expressed online, most of the emotions we show to our keyboards, monitors and mice are not transmitted. Sometimes, this is good; but, often it's a source of misunderstanding and miscommunication, resulting in lost time, damaged relationships, and reduced productivity."

If humans have difficulty reading emotions online, what hope for computers? The danger is that users will be even more likely to leave a site if it misreads their desire for help or information. Attempts by Microsoft to create interfaces for PCs that analyse user behaviour and try to understand it led to 'Bob' in 1995 and his equally dislikeable successor Clippie, the Microsoft Office paperclip. Microsoft found that many users actually preferred the standard interface to the new 'friendly' interface.

Companies experimenting with psychological profiling admit that current technologies are far from perfect. "We're aware that in everything we do, there is error," says APR's Brown. "Error in eliciting information, error in building the profile, error in classification, error in matching. So each process is error prone. One of our goals is to minimise error and minimise the consequences of error."

The development of software that is able to read users, minds undoubtedly remains the stuff of science fiction. But instead of trying to understand a person from moment to moment, which, Brown contends, is hard even for humans to do, longer term profiling – predicting general behaviour and interests – is a more achievable goal. " In my view, we're creating a piece of technology that's helpful and understands. How far have we got? About 5%. But it's a terrific goal," he says.