In a few short years, Hadoop has become one of the most talked about technology innovations of the current decade. Backed by Yahoo and boasting Tesco, Google and Amazon as customers, it is gaining traction in the market.
Hadoop has undoubtedly been grasped by a group of global firms with enormous volumes of complex data (and resources) ready to be gobbled up. It is very attractive to those large organisations with petabytes and billions or hundreds of billions of rows of data.
But, sadly, for smaller organisations – which let’s face it, make up the bulk of the UK economy – Hadoop is just too big for them. They simply don’t have the data to warrant its deployment.
Plus, Hadoop needs new infrastructure, such as commodity servers and local disk. By contrast, most firms nowadays have centralised infrastructure. This means to deploy Hadoop, additional hardware is needed and not too many small to medium firms could get those costs through the FD.
>See also: To NoSQL or not to NoSQL?
All is not lost though. Hadoop might not be quite right for the smaller business, but it has proven to be an amazing ‘trickle down’ technology. It has inspired many, many businesses of all sizes to look at the big data they own – its volume, variety and velocity – and agree ways to manage and analyse it. If it weren’t for Hadoop, these companies wouldn’t be looking at analytics at all.
Lots of firms are already now using data warehouses and data lakes with BI software to analyse data, for example. Some are investigating keeping their data in situ, but using analytics software to seek it out and analyse it.
Plus, one of the biggest areas of adoption will be NoSQL databases. These databases can store different types of data as well – i.e. variety – just the same as Hadoop, but they work better with lower volumes (which is more suitable to UK firms).
There are a couple of open-source NoSQL databases that are proving popular: MongoDB and PostgresS. Designed for smaller data sets, these databases are both much easier for small businesses to deploy than Hadoop.
The next years are going to be interesting as large firms demonstrate their successful Hadoop deployments and ROI, inspiring smaller firms to follow.
Smaller businesses will be looking at other technologies to eat up their data and spit out some answers. There are lots of options on the table, but NoSQL is sure to take some market share.
Chris Brown, big data lead, OCF
