After several years of non-stop hype by virtually everyone associated with the notion of big data, I believe that it is time to issue a scorecard regarding all of the promises that have been made but remain unfulfilled.

How would you measure the success (or otherwise) of big data at this point? I would offer the following categories for consideration, along with the grades that they merit.

>See also: Is data mesh the next big data architectural shift?

Adoption

Enterprises have traditionally been slow in the up-take of new paradigms and technologies without a compelling business requirement. Big data has clearly had such a requirement in my opinion, as most business leaders have been clamouring for a means to fully exploit their available (and third-party) information assets for analytical purposes in order to create competitive advantage. Once again, however, the discussion – and apparent focus – has been on technology and its assimilation into an enterprise landscape. This has led directly to a low adoption rate, other than tinkering by IT development teams.

Grade: D (for disappointing)

Benefits and outcomes

A corollary of adoption, this criterion is a measure of the achievement of tangibles from big data and the impact that they have had on the business. There have been many compelling use cases proposed by advocates of big data, but few to date have been fulfilled. There is clearly a lot of promise in respect to this area, with considerable effort being applied.

Grade: C (good effort, but little success so far)

Maturity

I continue to see overreach by the entire vendor and analyst community on this grading factor. Already the hype around ‘Big Data 2.0’ has reached din levels without any validation that 1.0 has achieved much of anything. What began as open source remains primarily that. The vendor community is struggling to productise this technology in a fashion that will meet current enterprise requirements, and many start-ups have leapfrogged to cloud-only solutions. Both have serious shortcomings in all dimensions, especially in relation to privacy requirements.

Grade: F (for failing to focus on the true priorities and fostering hype in spite of this)

Leadership

If hype and promises were leadership measures (or a proxy for them) then the grade would be A+.  However, they are not in any respect. What I continue to observe is a free-for-all in relation to who is leading vs. who is following. There is a battle for leadership between the established vendors and the start-ups, with neither winning, and more importantly creating cognitive dissonance within the customer community. This will delay market adoption and push out tangible business outcomes more significantly than any other factor being graded.

Grade: D (for extremely disappointing, as well as very frustrating to observe)

Growth (trajectory and opportunity)

Other than in the number of words said, written and tweeted, I have seen very little growth in respect to big data altogether. What I have seen grow is the arrogance and hubris of the entire big data hype machine. Without any real success to herald, they are collectively advocating new organisational designs and hierarchies, privacy and security paradigms, and adoption practices (à la ‘trust me’). I find it all very annoying and extremely disruptive to what had been good growth progression in both information management and analytics, as well as business accountability.

Grade: F (complete failure)

There will be howls and push-back by many in the hype stream when they read my assessment, but most on the end-user side will no doubt agree. Big data is a big disappointment so far, in spite of having such high intrinsic potential. If the vendor community wants to succeed, it needs to change its approach dramatically and quickly.

See also: The risks of ignoring big data

What can big data actually do for me today?

Not all news is disappointing in the big data world. There are many benefits that can be derived today if you take a practical approach and set your expectations accordingly. These would include unstructured data integration into your existing structured world, unstructured search and text analytics, and machine data operational analytics. I would focus on these to gain competencies and to deliver outcomes in advance of the bigger picture coming to fruition (whenever that might actually be).

What is missing from the big data conversation?

When the conversation is a non-stop stream of hype, what is needed is a strong dose of reality to counter it. When I say reality, I do not mean the weak and self-serving customer case studies and testimonials that we continue to hear, but a strong dose of honest reality where all of the factors that I have reviewed are addressed. To be successful, this community must move forward together. There will be no clear market leader in terms of technology or market share, but leaders will emerge who have worked to solve customer challenges and create tangible business outcomes using big data capabilities alongside existing enterprise investments and solutions. There are few ‘big data-only’ opportunities in the market, and we should minimise our myopic focus on them alone.