Spotify is planning to move its 690-node Hadoop cluster, reportedly the largest in Europe, from Cloudera's distribution of the big data platform over to Hortonworks's Hadoop Data Platform (HDP).
The Swedish music streaming service first adopted Hadoop five years ago. It uses the big data platform to process customers' music consumption data, in order to personalise their user experience and to support supplementary services such as a demand tracking for bands and record labels.
Wouter de Bie, Spotify's data infrastructure team leader, said that the company was looking for a "true parter" in its Hadoop support services provider. “The cultural fit was an important factor in our selection and we have appreciated Hortonworks’ relaxed, helpful and open approach,” he said.
- See also: The challenges of adopting Hadoop
"Their true open source approach and the work they have done to improve the Apache Hive data warehouse system also aligns well with our needs, as we use Hive extensively for ad-hoc queries and for the analysis of large data sets," de Bie added.
Spotify will run HDP on top of the Debian open source operating system. The work Hortonworks will do with Spotify to make this possible will allow it to offer HDP to other customers running either Debian or Ubuntu, the company said.
Winning Spotify's business is a coup for Hortonworks. According to a survey by data processing software vendor Syncsort, conducted at two major European big data conferences earlier this year, Cloudera leads the European Hadoop market with 41% of respondents who had already adopted the platform.
Hortonworks was in third place with 18%, behind the core Apache code at 30%.
However, among those attendees that were planning to adopt a Hadoop distribution, Hortonworks was in second place with 23%, compared to Cloudera with 29%.
Hortonworks was spun out of web giant Yahoo!, where Hadoop was first developed (by Cloudera's chief architect Doug Cutting) in 2011. The company has a strategic partnership with Microsoft – HDP forms the basis of Microsoft's two big data offerings, HDP for Windows and HDInsight on the Windows Azure cloud.
Last month, CEO Rob Bearden said the company is planning an IPO in the next 15 to 24 months.
