There was a time when analysing giant data sets was the preserve of large corporations and universities – the only organisations who could afford either very powerful computers of their own or access to supercomputers.

That time has long gone. The combination of cloud computing, which means companies can rent just the IT infrastructure they need, and “big data” technologies that split large analytical workloads across many, cheap servers, means that organisations of any size can conduct complex and large-scale analyses  – as long as they have the brains to do it.

A handful of start-ups are exploiting this trend, and using big data analytics to provide new information services that would have unimaginable for companies of their size not long ago.

Information Age recently caught up with two London-based examples – Growth Intelligence and Digital Shadows – who are crunching the ocean of data about organisations that is available on the web, in two very different ways.

Growth Intelligence

Thomas Gatten is a former BBC journalist who left the media to join a global market research company. A few years ago, that company was approached by Google, who wanted to know how fast online discount company Groupon was growing in UK.

Unfortunately, the market researcher had no information beyond what businesses are obliged to declare with Companies House, which is, on average, 16 months old.

"So Google went off and found out for themselves using data they collected from the web," Gatten recalls. "I thought that was very interesting, because they were somehow able to use web data to achieve what the paper trail couldn't."

The experience inspired Gatten to strike out on his own. Having seen Google's demand for information about Groupon, he founded Startup Intelligence.

Using freelance researchers all around the world, the company would compile reports on topics such as "the top 25 machine-to-machine communications start-ups in Brazil", and it was very lucrative. "Our clients would pay £20,000 for a report on 20 companies," he says.

At first, he thought Startup Intelligence's customers, which included the likes of IBM and Google, were interested in finding investment opportunities. In fact, it later became clear, they were looking for sales leads.

"We realised that if we could analyse data about businesses on the web, we could offer our clients real-time intelligence"
Thomas Gatten
Growth Intelligence

"They wanted to find companies that were in a specific stage of growth so they could sell to them," Gatten explains. "For example, if IBM knows the fastest growing M2M start-ups in Turkey, they can get on the phone and sell them some database software."

Gatten realised that this was a bigger opportunity than intelligence for investors. “There are maybe 10 to 15 venture capital firms in London, and 10 to 15 private equity firms,” he says. “But there are thousands of businesses that sell to other businesses.”

He therefore renamed the company Growth Intelligence, and set out to pursue the B2B sales information market. But Gatten knew that Startup Intelligence’s modus operandi, using paid researchers to conduct interviews, could not scale to meet the size of the opportunity.

Early last year, Gatten met Prashant Majmudar, a computer scientist who was at the time working for BAE Systems Detica. “He was working on a project where they would drag an array of microphones under water, analyse the soundwaves, and work out the location of a nearby submarine,” Gatten explains.

“We realised that if we could do this with data about businesses on the web, we could offer our clients real-time intelligence.”

The system that Majmudar built pulls in data from a variety of online sources. This includes social media data, news coverage, official records such as import and export certificates, and technical data such as the number of PCs that a company has connected to the web. Gatten calls this information the company’s “digital footprint”.

That footprint is analysed to provide a growth profile of every company in the UK, with each one ranked as either “very fast growth”, “growth”, “neutral” or “negative growth”. This information is presented through a software-as-a-service (SaaS) application that allows sales people to search for relevant companies with a particular growth profile.

Growth Intelligence, Gatten claims, falls into the “very fast growth” category. “A year ago, we were two people, now we’re seven,” he says. "We've gone from nothing to very significant revenues in just over a year.”

The company recently completed Accenture’s FinTech Innovation Lab programme in London, an experience that has prompted Gatten to pursue the financial services market. Today, he claims, 75% of the UK's retail banks are doing pilots with product or are about to start.

Digital Shadows

Digital Shadows is another company turning sophisticated analysis of the web into a service for business. But unlike Growth Intelligence, which alerts companies to opportunities, Digital Shadows is looking for risks.

Co-founders Alastair Paterson and James Chappell are also former BAE Systems Detica employees. They had both been looking for an opportunity to set up on their own, and two years ago alighted on an idea.

“I had been thinking about how people leave digital footprints on the Internet, and how those footprints are increasing,” recalls Paterson, now Digital Shadows’ CEO. “It really hit me when friends started uploading their ultrascan photos that children today have a digital footprint before they even are born.”

“James and I started thinking about how this affects organisations,” Paterson explains. “As we thought about social media, cloud computing and BYOD, we realised that there is more and more information pouring out of enterprises and on to the Internet.”

“It has got to the point where businesses need to start worrying about the impact of that information on their reputation and their security.”

"Information is getting out indirectly through suppliers, partners, clients and employees, and you can't protect against that at the boundary"
Alastair Paterson
Digital Shadows

Digital Shadows’ aim is to help businesses mitigate the risks by scanning the web for confidential information or data that could be used to stage an information security attack. It does this using natural language processing algorithms that it has developed itself.

Information leaks onto the web in a number of ways, Paterson explains. Most large organisations themselves have data loss prevention software installed at their perimeters that prevent files being uploaded to the web. But those companies also have thousands or employees, partners and customers who may well accidentally upload the data onto an insecure cloud file sharing system, for example, or even a social network.

“We’ve seen one example where the son of supplier got hold of their parents’ laptop and managed to share the whole C: drive on a public file sharing service,” says Paterson.

Another way in which sensitive data finds its way online is through technical discussion sites and forums. “You quite often find that the IT staff in a company might post information about its infrastructure when they asking for help on one of these sites,” Paterson explains. “In one case, we saw an IT admin post the configuration details of their employer’s firewall software in an online forum.”

There is even a site called RateMyNetworkDiagram.com, where systems administrators are encouraged to post pictures of their companies’ IT infrastructure.

Digital Shadows is also aiming at the financial services market. Paterson believes that while the major banks are armed to the teeth with security systems that protect the perimeter, that alone is no longer enough.

“A lot of information is getting out indirectly through suppliers, partners, clients and employees, and you can't protect against that at the boundary, because it's going out from other exit points.”

The argument seems to working – Paterson says the company is working on commercial contracts with a number of the largest banks in the world, and ealier this year it was named as a “Cool Vendor” in the security service sector by analyst company Gartner.

Growth Intelligence and Digital Shadows have more than cursory link: Paterson plays in a band with Growth Inteligence’s CTO Prashant Majmudar, and in fact introduced him to Gatten. They are both also based in the Level 39 start-up incubator in Canary Wharf.

Both are keen to stress that the information they each seek on the web is very different. “We are working with similar technologies, but the information we are looking for is different, and the sources we use quite different,” says Paterson.

They agree, though, that neither company would be possible without the vast amount of information that is now available on the web, and the relative ease with which the technology infrastructure required to analyse it can be accessed.