People have always talked about the businesses they patronise, the public services they use and the politicians that govern them, whether it is to make recommendations or to air grievances.

In the past decade, however, it has become possible for organisations to eavesdrop on those conversations. That is because, more than ever before, they are taking place in public, through social networks, personal blogs, video-sharing sites and all other forms of social media.

The nature of the web means that the content of these sites can be analysed systematically, providing marketers with an unprecedented window into public opinion and customer sentiment. The value of this is self-evident, and it comes as no surprise that the nascent field of social media analytics has already attracted a gaggle of software vendors, among them IBM, the SAS Institute and Autonomy, all of whom have announced their entrance to the market this year.

“Social media analytics is an important way to track your brand’s health and the issues that are associated with certain products and your competition,” explains Gartner analyst Andrew Frank. “It is a necessary piece of the puzzle for how social media can be leveraged by marketers and how investments can be tracked and verified.”

However, it is still an emerging field, and there are issues to be resolved. What, for example, is the best way to interpret social media content? How accurate are the available tools? And does it place consumer privacy under threat?

Buzz and sentiment

There are two common styles of social media analytics tool – those that assess the ‘buzz’ around a given topic and those that assess ‘sentiment’.

Put simply, the former is purely a measurement of how frequently a brand is mentioned. The surprising power of this simple metric was seen when a political social media analysis website used it to predict the outcome of the 2010 UK general election. The final prediction – made two days before the polls opened – was strikingly accurate, despite being based on little more than word counting (see box ‘How Twitter predicted the election result’).

Measuring sentiment requires more complex analysis. The most common approach is to judge each word or phrase in a blog post, Twitter message or Facebook status update as positive, negative or neutral in tone. These judgements are then collated into an overall assessment of the sentiment of the content.

Most systems work by analysing the language algorithmically. These algorithms are developed by ‘training’ them against human subjects – i.e. employing people to assess the accuracy of judgements until the algorithm ‘learns’ the correct interpretation of words or phrases.

Analytics software giant the SAS Institute this year launched a sentiment analysis tool that allows marketing professionals to train the algorithm themselves. According to the company’s head of analytics, Laurie Miles, this allows the tool to be tailored to users’ particular industry. “When you set these things up, you need a certain amount of domain expertise,” he says. “For example, if you’re in the supermarket industry, you know that ‘long queues’ is a bad thing, whereas if you’re in the entertainment industry, it’s more likely to be a good thing.”

Using human subjects to train algorithms also allows analytics systems to decode the idiosyncratic and constantly morphing language used on social networks. Without this capability, says Jeff Catlin, CEO of text analytics software provider Lexalytics, analytics tools can be misdirected by the slang, sarcasm and irony that peppers the language of the web. “You train the system to recognise sarcastic commentary by giving it a lot of examples,” he explains, “from which it then figures out how the language is used and how people talk when they’re saying something sarcastic.”

Accuracy and influence

At present, the social media analytics space is dominated by a handful of small, independent providers, including Alterian, Lexalytics and Radian6, although larger names are moving in. These providers often claim that their sentiment analysis tools are 75% to 85% accurate.

Philip Sheldrake, a partner at UK marketing and communications firm Influence Crowd, questions these figures. “At the moment, the finest of algorithms out there can achieve an accuracy of around 65%,” he says. According to Sheldrake, accuracy statistics are generally calculated by compiling a test database of phrases, which are manually judged for sentiment by linguistics analysts before being run through an algorithm. Discrepancies in the algorithm are then corrected to better correlate with the findings of the analysts. “The tweaking of your algorithms then depends very much on the content of that test database.”

Despite his scepticism of some of the claims being made today, Sheldrake believes that social media analytics will play a major role in marketing strategies in future. It will not only help organisations observe the market, he says, but it will also help guide them as they participate in social media.

In time, Sheldrake says, organisations will learn to use social media tools in a more sophisticated fashion, just as the use of customer relationship management (CRM) systems has matured since their introduction in the 1990s.

“When CRM was born about 15 years ago, it went through a phase where it tried to treat everyone equally, until somebody realised that some customers are just worth more,” he explains. “Their lifetime value is higher than others, and therefore CRM started to stratify customers into different levels of service level expectation.”

The same is true in social media, he argues. Certain individuals have greater influence than others, whether it is because of the number of ‘followers’ they have on micro-blogging service Twitter or because of the frequency of their contributions to user review websites.

According to Lexalytics’ Catlin, some companies are already putting this into practice by “reaching out to influential bloggers to start marketing campaigns via social media”. Not only can social analytics inform which influential ndividuals are best placed to launch these campaigns, he adds, it also helps organisations to “measure the effect as it ripples out from one contributor”.

Privacy protection

In a recent assessment of social media analytics software, IT analyst company Gartner predicted that a quarter of all organisations will be routinely using such tools by 2015.

However, the group also highlighted a potential obstacle. Social media analytics involves harvesting personal information and self-expression for financial ends. Is it morally appropriate, or even entirely legal, Gartner asked, for businesses to build detailed ‘profiles’ of their consumers based on their opinions, relationships and personal characteristics?

Sheldrake of Influence Crowd warns that there is a real danger of angering customers by invading their personal space through social media analytics technology. “You must manage any potential for the public to kick back against some kind of Orwellian overlord,” he says. “You’ve got to allow the public to quiz all the information that you hold about them.”

And while he predicts that privacy legislation may one day be introduced that forces businesses to disclose what personal data they hold within their databases – a Freedom of Information Act for private organisations – those companies that open up access to their customer data on their own initiative will “win so much goodwill that it’s worth doing before” any legislation comes in, Sheldrake says.

Despite these concerns, Gartner analyst Andrew Frank believes that it will become too difficult for organisations to ignore the discussions taking place in the social media sphere in the long term, and that subsequently, social web analytics will become a natural extension to the concept of what is today known as web analytics.

He also highlights the notable absence of Microsoft and Google from the social media analytics market. Frank believes it is only a matter of time until they make a move as, like all businesses, social media is something they cannot ignore.

“Being able to leverage social media is going to become a necessary component of any analytics product” he says. “It’s not a matter of being successful – it’s more a matter of avoiding failure due to not addressing something that everyone else is clearly focusing on.”

See also: Tableau 8.1 and the future of social media analytics