Recent events, however, suggest that this is changing rapidly, and that the deeply-technical pioneers of the search community world are starting to break into the mainstream. Helped by a massive and urgent demand to extract information and meaning from vast databanks of structured and unstructured information, the search technology business is, belatedly, starting to look like the enterprise software market of the mid-1990s. It is attracting investment, spawning start-ups, enjoying a lot of hype and, at long last, achieving substantial revenue growth. (See Search as a platform, Information Age March 2007).

 “There is an arms race of search underway. This is the dawn of search in the enterprise,” said Chris Weitz, US managing director of consultancy BearingPoint at a recent search conference. “Everyone who writes code for a living is now claiming that search is their differentiator”. Mike Lynch, the CEO of search engine leader Autonomy, echoes this view:  “There is a desperation [among vendors] to have a way of handling unstructured information,” he says.

The sudden take off of enterprise search is clearly demonstrated in the financial figures of the three vendors that Gartner, the research company, rates as the leaders in the sector. Fuelled by both acquisitions and organic growth, Autonomy (UK), FAST (Norway) and Endeca (US) all grew at over 60% (see box) in 2006, turning over more than $500 million between them.

Just one topical example gives a measure of that demand. In early May, the UK’s Ministry of Defence announced it would use Autonomy’s IDOL enterprise search system as part of its new single information infrastructure. The project will help make relevant information available to more than 400,000 staff.

Growth on this scale has not gone unrecognised by the giants of the global software industry. IBM has at least three major search projects underway, including iPhrase, which it acquired in late 2005. Other major announcements are expected during 2007. 

Oracle’s Secure Enterprise Search is just beginning to win customers, and Microsoft has opened a search technology centre in China and embedded deep search capabilities into both Sharepoint Server and the Vista operating system.

Consumer search behemoth Google, meanwhile, has sold its enterprise search appliances to thousands of businesses – although its lack of management functions and reliance on secret link-ranking algorithms have restricted its wider use as an enterprise platform.

So far, none of these big companies have managed to challenge the leadership positions of the specialist search providers, who are enjoying a frenzied spell of activity.

In April, FAST further underlined its position by acquiring the Retrievalware business of Convera, bringing with it over 200 customers from the US federal government. Meanwhile Autonomy, having capitalised on its acquisition rival Verity, now plans to float its Blinkx subsidiary, an Internet image search company, on the London stock exchange later in 2007.

Privately held Endeca is proving an unexpectedly strong competitor. In March, it announced that BT had chosen it, over Autonomy, to provide the main search engine for a range of activities, including embedded searching in call centres. Paul Sonderegger, Endeca chief strategist, attributes the success to the company’s iterative approach to search, which involves creating document summaries on the fly and making helpful and new suggestions to enable the user to find the right answers quickly.  The BT deal is “a turning point for Endeca in Europe,” says Sonderegger.

These three leaders, however, are not alone. In its ‘Magic Quadrant’ for the enterprise search sector, Gartner lists over 30 companies. Some consolidation is inevitable as the market matures. Although many of the smaller companies will undoubtedly be snapped up for their specialist technology or vertical customer bases, the ultimate position of the leaders is less clear. Such is the promise of search technology that big vendors such as Oracle, Microsoft, SAP and IBM will surely be tempted to buy market share.

Both Mike Lynch of Autonomy and John Lervik, the CEO of FAST, stress that their advantage lies not just in the underlying technology, but in the expertise, wide range of adaptors and the partnerships that they have built up.

It would not be easy for any vendor to replicate this. Autonomy, points out Lynch, has a high valuation (over $1.4 billion as of late April), so any buyer would really have to believe it needed to own the company to succeed.

If there are no big acquisitions, there may be some interesting flotations. Endeca’s venture capital backers will ultimately need to exit; and FAST, having tidied up various corporate governance issues, has long been linked to a possible listing on either London or Nasdaq.  Either way, the search sector is quickly moving from a market with million dollar plus companies to one where billion dollar companies hold sway.

Leaders in enterprise search