“I would like to see search engines become like the computers in Star Trek. You talk to them and they understand what you’re asking.”

As the very first employee of the biggest search company of them all, Google, Craig Silverstein should know he is not talking science fiction.

His comments, reported in the recently published book on Google, The Search by John Battelle, underscore the extent to which search technology has recently become the focus of attention. Computers have historically understood narrowly structured information while people can process masses of free-flowing information. The question is can machines based on on-off binary switching ever make sense of a largely unstructured world?

Solving this is a major preoccupation of computer scientists – and a problem one Amazon executive describes as “only 5% solved”.

But there is another reason for the current fascination: effective search, whether over the Internet using a search engine or within an organisation seeking to solve information management problems can be extremely rewarding. The billions of dollars in Google revenues, or the rapid growth and success of enterprise search companies such as FAST and Autonomy, are testament to this – and so are some of the success stories associated with more accurate enterprise searching.

Some of these examples are simple but effective: the rapid retrieval of news stories and film clips at the BBC; the near real-time access to customer data by call centre operatives at BT; or ability of financial regulators to search the email archives of investment banks. Others are more sophisticated: mobile phone company Vodafone searches voice calls to test ‘sentiment’, while Honda managers are alerted to potential quality problems through automated analysis of the content of customer emails.

Content in context

In spite of the power of these examples, many experts warn about the unrestricted use of search engines in business, recommending careful attention is paid to taxonomies and categorisation.

“I think search is over hyped and overrated. There is a ‘Google effect’ sweeping the enterprise,” says Clive Holtham, professor of information management at the CASS Business School in London. “I think search is a way of avoiding the difficult decisions about how to have some structure in unstructured or semi-structured information. “

Tammy Alairys, global lead of information management services at IT consultancy Accenture, also recommends a considered approach. “It is critical from a usability standpoint that companies get the right filtering, personalisation and post-processing search results from enterprise search,” she says.

The argument of these experts is that searching unstructured information will usually yield unstructured – and therefore unpredictable – results. Google often returns irrelevant or skewed results – in spite of the enormity of the investment and the patented ranking algorithms. And while some knowledge workers may need to carry out ad hoc searches, many workers are more likely to be repeatedly looking for something specific.

A dedicated team of consultants at Accenture’s Information Management Services (AIMS) division work in conjunction with business partners and customers to build taxonomies at the outset of each enterprise search project. In some cases, the company may already have its own predefined taxonomy. In February 2006, for example, FAST signed a contract with legal and professional publishing company LexisNexis which will see FAST reselling and integrating its Enterprise Search Platform (ESP) with LexisNexis’s existing human-built taxonomies. The plug-in application will be integrated with 24 industry-specific taxonomies, as well as general business and news that are updated daily.

“Taxonomy building is a very structured process, but depends on how much time and resources the company can afford,” says Alairys. “It ranges from a quick and dirty half-day session – where companies are shown what they should be doing to get most value from their ESP – to a week-long process where we do a much deeper dive on the actual business areas and what content is relevant.”

Not everyone, agrees: Google recently moved into the enterprise market, and has allied with consultancy Bearing Point; this combination is unlikely to advocate deep categorisation. Meanwhile Dr. Mike Lynch, CEO of enterprise search company Autonomy, argues that advanced Bayesian techniques and background tracking can help deliver relevant search results without requiring the user to do anything explicitly. This works because Bayesian technologies take account of the context – people search for whole ideas, while automated systems are limited to searching for words.

Search returns

Whatever these differences in approach, it is clear that enterprise search is catching on rapidly – and, producing significant savings in some areas.

BAE Systems embarked on a virtual university project with Autonomy after discovering that over 60% of its networked employees spent over an hour a day duplicating others’ work. Here, ‘implicit query’ technology generates real-time user profiles automatically, based on the pages its 35,000 engineers visit and the documents they read, and the average search query takes just 200 milliseconds.

 “We discovered engineers working in different areas on precisely the same problem – such as a wing construction issue,” says Richard West, head of organisational and e-learning at BAE. “One group established best practice, which was then transferred to plants in other geographical locations with multi-million dollar savings.”

This last example illustrates one of the future trends in search technology – it is increasingly becoming user- and process-aware, producing information about its users, their behaviour, searches and processes.

Increasingly, search results will be integrated with other applications – for example, they will trigger alerts to individuals or applications. And instead of just returning a list of documents from a search query, search engines can link to other business intelligence applications, refining the analysis to show key trends.

Not everyone will go as far as Salim Mitha, director of Yahoo! Search in the UK and Ireland, when he proclaimed that “search has become the central navigation point for information discovery”. But the business benefits of having a powerful enterprise search function in most enterprise applications will soon be beyond dispute.