Research Methodologies

July 14, 2011

The Dilemma of Social Media Measurement

There appear to be three approaches towards social media measurement: Word Counters, Influence Enumerators, and The Linguists.

Michael Wolfe

by Michael Wolfe

0

Editor’s note: I am thrilled to welcome our newest guest author on the GreenBook Blog: Michael Wolfe of BBDO. Michael is a renowned expert on marketing mix modeling, social media metrics, and advanced analytics and is a sought after speaker and contributor on those topics. He is doing some ground breaking work on developing new, ROI-driven models for social media analysis and it is a great honor to have him share some of his insights with us.  Michael is getting to the meat of what drives investment in research in this post and poses a provocative challenge to market research at the end. He will also be sharing more details on his work with all attendees of the upcoming Market Research in the Mobile World conference next week in Atlanta. I think all of our readers are going to really enjoy this one!

 

Everyone with any sense of what is going on in the field of marketing right now knows that there is almost a mad frenzy about social media, the likes of which we have seldom seen.    In fact, this frenzy has become so strong that it has spawned almost a gazillion self-proclaimed experts and gurus.   It almost feels like the American Wild West, with the itinerant snake-oil salesman coming to town to sell us a bottle of his miracle cures.

As I have reviewed the literature on this subject, there appears to be three camps or approaches towards social media measurement.   These are as follows:

  1. The word counters.   These simply count the frequency of mentions of any given subject or brand name across social media such as Twitter, Facebook and various Blogs.    The theory behind this approach is that word frequency is sort of like market penetration, the more the better.   Some who follow this approach might go one step further and parse their word counts into positive and negative sentiment groups.  This sometimes evolves into more sophisticated sentiment analysis via sophisticated text mining tools.
  2. The influence enumerators.   This camp believes that it’s not the words that matter, but the influence of the person saying them.  Via some algorithms for scoring individuals based on number of friends, followers, re-tweets or klout scores, it is possible to differentially score the word counts based on the level of influence of the person posting the conversation.
  3. The linguists or engagement scorers.   This approach believes that language and semantics is everything.   Through the science and rules of linguistics, it is possible to “quantitatively score” social media conversations and determine how “engaged” these conversations are.   Influence and impact in social media is very much a function of language, word nuance and context.

Presently, there is a lot of confusion regarding which approach is most valid.   The only research I can cite is from Bernardo A. Huberman of Hewlett Packard.   Bernardo found that social media can predict such things as movie success.  He also found that the quality of a message can pierce through the noise and influence an audience.  He likewise found that the number of friends and followers on social media does not add much to the influence of the message.

The impetus for developing social media metrics is obviously targeted at developing a capability to measure the impact of social media conversations on people’s attitudes and behaviors.   Significant here is the conversations on brands and how these conversations affect brand performance and ROI.

My own journey here tends to favor the linguistic approach.   As a player in BBDO’s marketing-mix modeling consultancy, clients challenged me to come up with an approach to measure the impact of social media on their brands’ performance.   My approach to this is fairly simple.   That was to find a metric which correlates best with brand sales of any of our 8 blue-chip clients.    In this journey, I tested and evaluated many of the “word counting” and “sentiment” metrics from major social media data vendors.  Sorry to say, none of them were found to have greater net correlations to brand sales of about 0.15.   Some of the sentiment indicators, in addition, even had counter-intuitive relationships with client brand sales.

Our journey finally led us to a metric called the Semantic Engagement Index or SEI as developed by a company called Linguistics Insights.  This metric was developed by Dr. Peyton Mason and Dr. Boyd Davis.  Davis is Professor of Applied Linguistics at the University of North Carolina.  Together, Mason and Davis developed an approach which first parses large quantities of social media conversations into positive and negative sentiment groups.   From there, each group is quantitatively scored using Linguistics science and rules.   The scoring algorithm used defines consumer engagement by scoring conversations on two dimensions.  Those dimensions are the degree of personalization and emotional effect contained in the conversational language.’

With our first client, we generated a time-based SEI metric.   We plotted this metric against brand sales for one of our clients.   As shown here, we plotted the ratio of positive to negative SEI.  It’s correlation to brand sales was an amazingly high +87%.   We have never found any single social media metric which tracks so closely and has such a strong connection to brand sales and performance!

We have since found similar and strong correlations of the SEI metric with brand sales of other clients.  We obviously think we have made a very significant discovery here.   Certainly, our efforts have enabled us to include the SEI metric in our marketing-mix models and directly measure the impact and ROI of social media on our respective client brands.   Overall, the evidence so far seems to support the linguistics approach to social media measurement; but obviously we need more supportive evidence across many more brands in order to draw definitive conclusions.

In sum, our research has found at least three different approaches to developing social media metrics, all with the purpose of measuring its impact on brands and consumer attitudes.   While I believe there is some evidence supporting the linguistics approach, I must say that the evidence to-date of which approach is superior is significantly lacking.   The bottom-line here is that this really represents an opportunity for Marketing Research to step to the center and demonstrate leadership.   Certainly, many researchers have access both to the metrics and brand sales data to do so.  Whether they have sufficient moxie to do so remains to be seen.

———————————————————————————-

Get Inspired. Stay Informed.  Sign up for our free email newsletter.

0

linguistics

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.

More from Michael Wolfe

Super Bowl 2019 Advertising Effectiveness

Super Bowl 2019 Advertising Effectiveness

Super Bowl Ads more effective than ever.

Michael Wolfe

Michael Wolfe

Artificial Intelligence and the Future of Marketing Analytics

Research Technology (ResTech)

Artificial Intelligence and the Future of Marketing Analytics

New software can potentially facilitate campaign analytics and driver analysis of panel data, cutting time and effort dramatically.

Michael Wolfe

Michael Wolfe

The Myth of Millennial Ad Responsiveness

Insights Industry News

The Myth of Millennial Ad Responsiveness

Are Millennials as unresponsive to advertising as commonly believed? Michael Wolfe challenges this assumption.

Michael Wolfe

Michael Wolfe

Research Methodologies

Super Bowl 2017 Ad Effectiveness

While Super Bowl ads do well on “likeability” and generating “buzz”, they didn’t fare as well on critical measures like “purchase intent”.

Michael Wolfe

Michael Wolfe

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*