Data science has come a long way since it was heralded as the sexiest job of the century by the Harvard Business Review.

That was in October 2012, when only a few existed in the world and there were no degrees in it. Two years on and times have changed: there are plenty more of them (although their rarity still demands a high salary) and universities are increasingly offering courses to produce a whole wave of them.

Their relevance is, of course, related to the rise of big data. As businesses clamber to draw value from the explosive growth in and variety of data, they need the talent to do this in a cost-effective, secure, governable and, most of all, lucrative way. 

Two years is indeed a long time in technology these days, but data science still remains in its infancy when compared to its ultimate potential in the enterprise. Here are four predictions for where it will go in 2015.

1. Data science sophistication will increase

2015 is the year data scientists will become a mainstream career choice. Most universities are offering courseware in data science in preparation of the dramatic rise of this profession – already high on recruiters’ lists, demand for data scientists will snowball in the coming year.

Nearly every enterprise will have data scientists that will do more than just studying customer behaviour; they will expand into new areas such as data forensics to combat rising cyber threats, and create new types of businesses based on data services.

>See also: UK’s data scientists face burnout due to work-related stress

Data science adoption will be driven by the business and its requirements, which will lead to an explosion of new tools and services focused on specific industries and departmental needs. 

In addition, the gap between expectations and reality will increase in 2015. Because many technologies are still very immature, newly hired data scientists are often very inexperienced regarding business questions and face the challenge of bridging the gap between business departments’ operational needs and the capabilities of the technical infrastructure.

2. The gap between expectations and fulfilment of data scientists will get bigger

Data science as the science of optimising a business by exploring data is becoming a mega trend, but there is a lot of confusion in the market. 

Regarding the job of data scientists, many big companies have already hired complete data-science labs without any clear strategy. 

The role of a data scientist has to evolve next year by becoming a kind of human interface between “data business incubators” and the actual business departments that need to gain competitive advantages over their competitors. 

In 2015, the need for additional human resources in that area will accelerate, but the number of skilled people to fulfil these requirements cannot be satisfied. Therefore the gap between expectations and fulfilment will open even more.

3. Hadoop will improve data mining efficiency

A huge part of Hadoop projects are not successful, nor lead to any real business applications. But Hadoop systems are very agile, cost-effective, self-service systems for data storage – allowing new types of data to be quickly stored and processed.  

Today, this is the appeal of Hadoop for data scientists. However, Hadoop has deficiencies when it comes to highly productionalised analytics processing for real-time at massive scale.

Hadoop systems will be forced to evolve into heterogeneous systems of open source and traditional enterprise technology in order to achieve the ROI demanded of the business in order to justify the costs of these system.

In-memory, applications and data-management technologies will hybridise and integrate with Hadoop to create a redefined Hadoop, open source and even operating systems that will challenge the dominance of the mega software vendors – but also bruise the rising cadre of cloud systems providers in its wake.

4. In-memory will reshape the IT landscape

In-memory has turned into an accepted technology, and CIOs more and more often see an in-memory layer as crucial within their complex data management ecosystems. 

The need for high-performance, in-memory layers on top of data silos will accelerate significantly in 2015 – providing a critical role in extending the life of these older-siloed systems, before their eventual decommissioning and redeployment in the cloud. 

The trend of storing more data volumes from a growing range of sources, in addition to the availability of more business applications to leverage insights from that data, will accelerate the necessity for specialised, dedicated appliances to provide performance, scalability and enterprise-ready capabilities. 

>See also: What makes a great data scientist

Many businesses have held back from diving into the in-memory pool because there has been a lot of confusion around the terminology and offerings.

However, the business pressure to stay competitive will ratchet up the adoption of in-memory technologies and companies will find themselves establishing partnerships a new guard of technology companies in order to achieve this step change in their competitive prowess.

Sourced from Aaron Auld, CEO, and Mathias Golombek, CTO, EXASOL