2017 was a fascinating year in the data storage industry: key events were the breakthrough of software-defined storage (SDS) and the fast adoption of non-volatile memory express (NVMe).

1. SDS will replace traditional storage arrays thanks to Server SAN

The concept of SDS meets the requirements of many modern data centres: instead of bulky, expensive proprietary hardware-based solutions, standard servers can be leveraged with intelligent software to build storage infrastructures that scale granularly, limitlessly and offer extreme flexibility and efficiency. SDS initially became popular with object storage. However, object storage is struggling to make real headway in the market as its primary use case is typically active archives – and many companies find it easier to simply archive in the cloud.

At the same time, the storage industry is going through a transition as NVMe grows in popularity. NVMe-based flash, with its extreme performance benefits, lends itself perfectly for low latency and high-performance use cases such as big data analytics, simulation and high-performance computing.

>See also: The storage industry forecast: Predictions for 2018

Server SAN combines the concept of SDS with the benefits of NVMe and new technologies that enable the sharing of NVMe-based flash across standard networks, and it is gaining momentum with significant year over year growth. Server SANs are based on standard servers and offer considerably more operational flexibility than traditional SANs at a fraction of the cost.

Customers are switching from enterprise storage arrays to agile, software-defined architectures such as Server SAN and some predict it is this, rather than object storage, that will facilitate the breakthrough of SDS in 2018.

2. NVMe-based flash will become commodity while Storage Class Memory (SCM) innovations will raise the performance bar

NVMe-based flash has slowly but steadily been entering the market. 2017 was undeniably a breakthrough year: customers across all verticals started to deploy NVMe-based flash for a variety of use cases, from video analytics to supercomputing, from retail to simulation.

Technologies that share NVMe over networking fabrics (NVMe-oF), or full scale-out storage solutions that enable the use of NVMe at large scale are facilitating this trend by enabling full utilisation of NVMe-based flash.

It is not surprising that NVMe-based flash is becoming the norm for enterprises that rely heavily on fast access to their storage. NVMe-based flash is simply flash storage with an optimised controller and protocol that supersedes legacy technologies designed for slower Hard Disk Drives (HDDs), and as a result, delivers significantly improved performance and reduced latency. End users therefore require fewer drives to achieve the concurrent performance levels for their workloads.

>See also: Enterprise storage in 2017: trends and challenges

Meanwhile, flash manufacturers have been working on even more significant innovations, namely SCM. SCM aims to fill the gap between extremely high speed, low capacity and expensive memory (e.g. DRAM or Dynamic Random Access Memory) and storage such as flash, solid state drives and hard disk drives, which are lower in cost and performance, and higher in capacity. Technologies such as 3D Xpoint are leading the way and raising the bar of what is considered high performance when it comes to data storage.

These are the early days of SCM, but it is already clear that customers will require software to deploy and manage this. Now more than ever there is a need for software-based storage solutions that enable customers to deploy existing and future high-performance storage, including traditional solid state drives (SSD) and SSDs based on NVMe or SCM.

3. The feud between storage and network teams will come to an end

Let’s face it, storage and network teams are like fighting siblings: they need each other, can’t live without each other but they certainly do like to pin the blame on each other for all of their problems.

For decades, storage architectures have been designed according to the limitations of either storage or networking. One or the other caused the bottleneck: for instance, switching from HDD to SSD improved storage performance but the network wasn’t able to keep up, then developments in network speed (Ethernet – Fibre Channel – Infiniband) required faster SSDs and new protocols such as NVMe-based SSDs.

>See also: Flash storage: transforming the storage industry

The industry has now reached a point where this to and fro is no longer necessary because the network will become the storage. Next generation SSDs, as mentioned above, are evolving rapidly and provide unseen levels of performance. And at the same time, next generation networking enables new distributed storage architectures.

This will mean that, through leveraging protocols like NVMe over fabrics, high-speed storage devices can essentially sit anywhere on the network and be treated as one large pool of high performance storage without compromises or bottlenecks: storage and network resources working together to provide an optimal solution.

4. Cloud for secondary and on-premises SDS for primary storage

The public cloud will win the battle for secondary data. Over the last few years, the public cloud has been contending with on-premises object storage to be the secondary storage solution of choice but as public cloud offerings attract an increasing number of customers, object storage vendors have had to shift their messaging and sell hybrid solutions that enable customers to keep some data on-premises and move other data to the cloud.

As for primary data, there is a market for placing this in the cloud – especially in the case of those customers who solely use the public cloud and don’t have an infrastructure of their own. But it is not expected that customers who have their own data centre to migrate their primary data to the cloud any time soon. There are many reasons for that: latency is one, value of the data and the associated security considerations another.

>See also: An industry perspective of the evolution of storage – hyperconvergence

In addition, a revolution is underway with block storage – for primary data storage needs. Several vendors are pushing software-defined block storage, leveraging standard servers and intelligent software.

To use the 80-20 rule, it would be safe to estimate that 80% of secondary data will end up in public clouds, and 80% of primary data will remain on-premises, but stored on more agile, efficient and flexible SDS infrastructures.

5. 2018 will see the emergence of exactly 12 1/2 new storage startups

Well, not 12.5 exactly, but it is expected that there will be a healthy breed of new startups in 2018.

2017 was an exciting year for the storage industry with many acquisitions, several new players entering the arena and a number of technologies disappearing. For example, Dell terminated DSSD, a clear sign that the market is asking for storage architectures that are not dependent on expensive proprietary hardware.

Innovation is driven by innovation. New application workloads demand new storage architectures.

>See also: Storage in an unstructured world – the role of object storage

In 2018, new vendors will continue to respond to requirements of emerging applications across industry verticals. This is not the end of IoT and businesses/society are barely at the beginning of AI applications, both of which are completely redefining standards of performance and scalability.

The trends discussed earlier, such as the rise of NVMe, SDS, and Server SAN will continue to drive innovation for primary storage solutions. It is expected that existing players and newcomers for block storage will move towards software-based architectures and NVMe or SCM, and away from proprietary hardware and older protocols.

Sourced by Tom Leyden, VP corporate marketing, Excelero