In the last decade, as the number and sophistication of technological devices has increased, there has been a massive growth in both the data generated and retained by organisations.

Traditionally, it had been the role of business analysts to sort through a small amount of this information to dictate business processes and financial strategy. However, the sheer volume of data now available makes identifying the data to analyse impossible.

Hidden in all this newly generated data is the information needed for everything from improving quality of life and solving societal issues, to guiding military defence and responding to humanitarian crises. It is no surprise that being able to understand this data, and sift through the parts which matter most, is becoming a highly desirable skill.

>See also: The future of tech in healthcare: wearables?

Tackling this challenge head-on are data scientists, the talented individuals tasked with understanding big data and identifying its insights for organisations. They achieve this by utilising a range of specialised skills, including: statistics, coding, mathematics, computer science, data visualisation and data mining.

They also develop AI and machine learning algorithms to help filter data. A single person having all of these talents is rare, and as a result the demand for data scientists will have outpaced supply by 50% by 2018.

Thanks to their work, we are already able to interact with big data through our devices and use it to make our lives more comfortable. We know where to eat, which films to watch, and the fastest travel routes at the touch of a button. But more interesting is the work being done to leverage big data across the defence and security sector.

From data to defence and humanitarian assistance

The use of big data in responding to crises situations has become increasingly important. Disasters – both natural and conflict driven – for example, are chaotic situations and big data is a crucial tool in helping aid organisations respond effectively. Through computer algorithms and analytics, data scientists have the potential to provide better understanding of emergencies and find patterns from connected devices and private data sources.

>See also: How data science is driving digital transformation

During the 2015 refugee crisis, for example, Sweden’s Migration Board saw an increase from 2,500 asylum seekers in a month, to 10,000 per week. Despite this dramatic spike in refugees, the agency handled the intake – hiring extra staff, starting the process of procuring housing early, preparing supplies – because it had utilised data analytics to predict the increase. The Swedish Migration Board had already been using big data and analytics for several years, and this helped it prepare for the crisis in a way other organisations may have had difficulty with.

There are obstacles to overcome in order to achieve these results. Data scientists may have difficulty gaining access to the relevant information to analyse. However, this has improved in recent years with the launch of open data initiatives, aimed at removing the barriers to useful data for organisations.

They could also have difficulty acquiring the required resources, whether that involves identifying the best technologies for their needs, or finding the money to pay for the technology and talent. Not only this, but historically most data scientists have only applied their talents to businesses, and may not have experience in other sectors.

Challenging data scientists

To help tackle these difficulties, and challenge data scientists to hone their skills, the Defence Science and Technology Laboratory (Dstl) and other government partners has launched the Data Science Challenge. Part of a wider programme set out in the Defence Innovation Initiative, it aims to test the talents of data scientists to develop new approaches to real and complex problems. Understanding that the best minds are rarely the ones that already work for you, Dstl’s challenge is open to entrants of all backgrounds and specialisations.

>See also: Big data is eating the world – but it’s not eating the data scientist

The challenge involves two tests, the first of which evaluates a data scientist’s ability to analyse data in documents such as media reports, which can provide a better understanding of a degrading political situation.

The second test requires creating a means of detecting and classifying vehicles such as buses, cars and motorbikes, from a set of aerial images. This solution would be utilised for identifying safe passage of vehicles through conflict zones.

While organisations such as Google and Facebook have been rushing to snap up artificial intelligence and machine learning talent, it’s clear that the expertise of data scientists go far beyond the business and tech sectors.

As the quality, availability and amount of data continues to increase, data scientists will find themselves becoming even more valued and desired than they already are. From defence and security to consumer convenience, data scientists are the key to unlocking the future.

Sourced by Leo Borrett, capability adviser at DSTL

Nominations are now open for the Tech Leaders Awards 2017, the UK’s flagship celebration of the business, IT and digital leaders driving disruptive innovation and demonstrating value from the application of technology in businesses and organisations. Nominating is free and simply: just click here to enter. Good luck!