In Gartner’s hype cycle, the term ‘big data’ was once a staple of the yearly report. It moved swiftly into the peak of inflated expectations, weathered its way through the trough of disillusionment, and is now prevalent – somewhere between the slope of enlightenment and the plateau of productivity.
Expectations of the technology are high, a Gartner survey in September 2015 showed more that 75% of companies are investing or planning to invest in big data in the next two years, and 37% of those projects are being driven from board level.
But as a term, ‘big data’ still has no clear definition. For some, a dataset over a terabyte is big data – for others, it might be a million rows, and others still may have smaller datasets that is changing many times a second.
>See also: How big is big data – and what can I do with it?
In the era of Google, Facebook, Amazon and web-scale data, no dataset should be too difficult to analyse. It is all about having the right tool for the job.
So instead of defining big data as a number or a size, it is more interesting and relevant to define it in terms of history, growth, compute and value.
History
29 million
That’s the size, in number of records, of the world’s first big data project in 1937. At that time, the administration in America were looking to keep track of social security contributions from some 26 million Americans, and 3 million employers and partners were sought.
IBM, with its giant punch-card machines of the time, got the contract – simultaneously setting the foundations for the Big Blue known now and setting in motion the start of automatic record keeping and data analysis on a massive scale.
Growth
1 Zettabyte
For the first time since its conception, global internet traffic will surpass 1 zettabyte (1 billion terabytes) in 2016, according to a Cisco research paper, having risen fivefold in the past five years.
A separate study estimates that 90% of the world’s data was generated in the past two years. Not only are we clicking, emailing, chatting and taking photos or videos more than ever, companies have cottoned onto the fact that data is valuable so are storing more and more data. Datasets such as website access logs and click data are no longer being thrown away – they are being archived and mined to generate valuable insights.
Compute
24 months
That’s the period that Gordon Moore, co-founder of Intel, observed was the amount of time it took the number of transistors in a dense integrated circuit to double. Similar observations have been found in other areas – it has been estimated that the amount of data transmittable through an optical fibre doubles every nine months and storage density doubles roughly every 13 months.
Organisations can process, transmit and store more data than ever, and all three are exponentially increasing commodities. Companies are better placed than ever before to deal with data.
Value
$59 billion
That is the estimated value of the big data market in 2015 and it is expected to roughly double to $102 billion by 2019, according to IDC. Big data is big bucks, a 21st century gold rush – and to extend the metaphor, big data analytics is the modern-day equivalent of panning for gold.
>See also: Top 8 trends for big data
For data-first companies, monetisation comes in the form of advertising – and big data analytics helps them to show an appropriate advert as well as analyse the results. For other companies it is often about increasing sales (the ‘I see you bought this, what about this?’ offers), automating decisions (big data gives the proof that an option is the correct one to take) and decreasing costs (driving efficiencies in the supply chain).
It’s clear big data is growing – it’s here to stay and, right up to board level, companies have woken up to its potential.
Companies see that with big data analytics they can find insights that enable them to outperform their competitors and reduce costs. With the right tools, these insights are easier to obtain than ever.
Sourced from Sean Jackson, CMO, EXASOL
