It is very difficult for organisations to discern which tools will bring them the most benefit in the year ahead, and which issues they need to plan for, such is the volume of technological choices available, data trends is no exception to this.
New technological developments provide the platform for the next generation of innovation, as we’ve seen with the evolution of ‘Big Data’ into advanced analytics, machine learning and artificial intelligence. How can businesses navigate this increasingly-complex data landscape to make the wisest investments?
Here is our guide to the top seven data trends that should be on every organisation’s radar for the year ahead.
Data trends in the air: developing cloud data infrastructure
We recently conducted research which found that 85% of respondents said they already had some of their data infrastructure in the cloud, and other surveys of IT executives reveal that many are planning to increase their investments in Software as a Service (SaaS) and cloud tools.
Next year, cloud migration will be fueled by the option of new cloud technologies, such as serverless and containerisation. Meanwhile, the choice of partners will also grow as vendors and popular open source data projects make their offerings easier to run in the cloud.
A data trend in creation: Building the foundations for machine learning
For most companies, the road toward machine learning (ML) involves simpler analytic applications. This is good news because ML demands data, and many of the simpler analytic tools that precede ML already require data infrastructure to be in place. The growing interest in ML will spur companies to continue to invest in the foundational data technologies that are required to scale ML initiatives. This includes items like data ingestion and integration, storage and data processing, and data preparation and cleaning.
Data trends for minding your own business, privacy becomes a priority
Privacy was one of the defining issues of 2019, from the advent of the GDPR to the Cambridge Analytica scandal. We can therefore expect to see an increased focus on tools for privacy-preserving analytics. Organisations will begin to identify and manage risks that accompany the use of machine learning in products and services, such as security and privacy, bias, safety, and lack of transparency.
Top five trends for harnessing data in 2019
Making machine learning sustainable
Early indications are that many organisations are – wisely – focusing their first forays into machine learning projects on use applications designed to improve their most mission-critical projects. For example, financial service companies are investing ML in risk analysis, telecom companies are applying AI to service operations, and automotive companies are focusing their initial ML implementations in manufacturing. This is also reflected by the emergence of tools that are specific to machine learning, including data science platforms, data lineage, metadata management and analysis, data governance, and model lifecycle management.
Data trends that need culture: Building a data culture
In a recent O’Reilly survey, we found that the skills gap remains one of the key challenges holding back the adoption of machine learning.
With the average shelf life of a skill being less than five years and the cost to replace an employee estimated at between six and nine months of the position’s salary, there is increasing pressure on tech leaders to retain and upskill rather than replace their employees in order to keep data-led projects on track. We are also seeing more training programmes aimed at executives and decision makers, who need to understand how these new ML technologies can impact their current operations and products.
Businesses are realising that investment in technology is not enough – they need to build a data culture throughout the organisation, and we’re already seeing businesses creating centres of excellence as they strive to foster a data-first mindset throughout the organisation.
The top five big data trends coming in 2018
Data trends we have been waiting for: IoT leaves the drawing board
A few years ago, most Internet of Things (IoT) examples were still at the theoretical stage. Thanks in part to the rise of cloud platforms, cheap sensors, and machine learning the IoT is now poised to leap off the drawing board and into our lives. Beyond the smart cities and connected consumer devices that have dominated most IoT discussions so far, we’ll still start to see other interesting use cases involving closed systems (factories, buildings, homes) and enterprise and consumer applications.
Automation
The increased pace of machine learning and analytics adoption will only be possible with scalable tools that enable data scientists to tackle many more problems and maintain more systems. This will lead to more process automation, including data preparation, feature engineering, model selection, and hyperparameter tuning, as well as data engineering and data operations. There are already some early applications of machine learning aimed at the partial automation of tasks in data science, software development, and IT operations.
Quantum Cryptography: The next-generation of secure data transmission
Overall
If there is a unifying theme to these seven trends, it is that culture, skills and processes are now as crucial to strategic success as any single technology. Businesses that invest in building these foundations will be those with the best chance of success in the years to come.
Ben Lorica, is the Chief Data Scientist at O’Reilly Media, Inc. and Program Director of both the Strata Data Conference and the Artificial Intelligence Conference
