There is a spectre haunting the world’s information and technology industries. Google, the search engine company, might only be eight years old, and its corporate actions still seem childlike, but it has executives across the world scrambling to the drawing board.

In the media, book and newspaper publishing companies fret about loss of advertising and breach of copyright, and TV stations worry that Google could syndicate programmes on-demand without their involvement. In ecommerce, those same portals that disrupted the old world fear that Google could now displace them. In telecoms, operators suspect Google’s foray into voice over IP could mean a loss of customers and revenues. In software, even the biggest suppliers see how Google is delivering, or preparing to deliver, services such as email and word processing for next to nothing over the web. Even governments are concerned about a loss of control over information resources and distribution.

“This is what is driving Steve Ballmer [CEO of Microsoft] to write memos to ‘kill Google’ and Ray Ozzie [CTO of Microsoft] to say ‘maybe software should be paid for by advertising’,” said John Gantz, research director of IDC at this year’s annual Regent Associates conference. “It’s the Google effect – fear of Google.”

Such fears may or may not prove to be justified over time, but in one area, at least, Google has already kicked over the chairs: search – its first and main market. In Internet search, its technological innovations (see article: The evolution of search) have revolutionised ecommerce and given it a near 50% share in just seven years.

Exploring the enterprise

Now it is the turn of enterprise search – until now a distinctly different business. Staff throughout industry are looking at what Google can do and asking: why can’t our systems – even our PCs – find files in the same way that Google can? “There is a Google effect sweeping the enterprise,” says Clive Holtham, professor of information management at Cass Business School in London.

The effect has energised a backwater of the software industry that, a few niche pockets aside, has underperformed for nearly five decades. Enterprise search technology is hot, strategic, and attracting capital, innovation and CIO attention.

“Google sets a very high standard for the web. We would like to set the next standard, so that people will find it so easy to do things at work, that they’ll wonder why they can’t do them on the Internet,” Arthur Ciccolo, search strategist for IBM’s unstructured information management architecture told a recent press tour.

This is not just about the Google effect. Executives are becoming aware that vast amounts of mission-critical data is accumulating in unstructured data repositories such as Word files, emails and instant messaging servers, says Royce Bell, who heads Accenture’s recently formed Accenture Information Management Services unit.

One response to those concerns can be seen in the rapid growth of enterprise search companies, such as Autonomy of the UK and Fast of Norway. “People think that consumer search is very different from enterprise search. But it’s not necessarily. Enterprise search is not a commodity, it’s mission critical,” says John Lervik, CEO of Fast.

Top four

Four big issues now dominate the race to build effective Enterprise Search 2.0. The first is innovation: will new capabilities be developed that enable more precise, more useful searching? Many information management experts think that the technology still has a very long way to go before it can be truly useful, and for this reason, its role is secondary in the enterprise. The second big issue is: if search technology is to be embedded into applications and used, as Yahoo’s Salim Mitha recently put it, “as the critical navigation point for information discovery”, how will it be delivered in the enterprise?

There are at least three distinct approaches to this. The first, and still the most commonly adopted, is advocated by Barry Litwin, CEO of Hummingbird, a leading enterprise content management provider. He believes that search technology is best offered as part of an information management suite, and that, unless there are specialist requirements, a commodity-like “good enough” approach offers businesses the best value for money.

A second approach is taken by Google, amongst others. This involves indexing everything and storing it in a separate database, which delivers the results as a web service. Systems integrators such as Bearing Point will integrate the service.

The final approach in the enterprise is the most expensive – but is proving increasingly popular. This involves integrating search into an overall information manage-ment platform and architecture. As with Google, search services will still be integrated with web services, but the platform will also handle security and access, resilience and recovery, interaction with structured data, and analytics. It is in this area that consultants such as Accenture and IBM see enormous potential.

The third issue is tagging and the relationship to unstructured information (see article: The evolution of search). Traditional enterprise information management relies heavily on tagging, metadata and taxonomies to organise data – methods that tend to produce reliable results. But the Google approach is very light on tagging – as is the Bayesian approach taken by, for example, Autonomy, and some of the emerging semantic search specialists.

With the launch of its enterprise search division, Google has threatened to disrupt the enterprise search business by offering low cost “search appliances” into which all documents are deposited, and from where they can quickly and easily searched. Apple and Microsoft have responded with equivalent solutions built in software. The Google Search Appliance is not just a tactical solution, says David Chalmers, European partner manager for Google. Channel 4 and the NHS are among the 4,000 plus users of Google’s low cost devices.

Although, so far, the appliances have been applied as minority, tactical experiments in most cases, which have not always produced the required results in an enterprise setting, improving technology and easy, web services style attachment will almost certainly make them increasingly popular.

The fourth issue is perhaps the biggest of all: will the results of unstructured searches ever become so precise, and so trusted, that they can be incorporated and automated in critical business processes?

“We’re all used to the idea that the role of IT is automation. If I go to a bank machine, it gives you your notes automatically. I think you’ll see more and more of that: the computer finds the information and does the business task,” says Mike Lynch, the CEO of Autonomy.

In fact, says Lynch, the best search engines will go one step beyond this, answering “implicit queries” by reading the screen, producing the answers and taking action without any prompting.

This is biggest opportunity – and threat – that improving enterprise search will bring. If 80% of all data is unstructured, as is often reported, then a very large proportion of decision making and action requires human involvement, if only to establish the meaning. A good portion of that 80% could soon be ripe for automation.

Further reading in Information Age

•  Enterprise search tools

Further reading on the web

• The Anatomy of a Large-Scale Hypertextual Web Search Engine , by Larry Page and Sergey Brin