1. SIS International Research
  2. RN_ESOMAR_WP_GBook_480_60_11_14
  3. la2015_480x60
  4. usa-banner2015

Two Key Challenges To Measuring The ROI Of Social

The measurement of a social campaign is about much more than likes, shares, downloads, and plays. It needs to be in the context of the objectives, and those objectives need to link to things like sales.



By Ray Poynter

Two Guides

Want to know how you should be evaluating social media campaigns? Do you want to know how to balance short-term activation events with long-term effects? The answers are in the recently launched #IPASOCIALWORKS Guide to Measuring Not Counting.

As one of the authors of the Guide I have been involved in several events, including the launch at the IPA, workshops, and conference sessions. Whilst these events have been generally positive, two major challenges have been exposed by our interactions with attendees and people working inside advertisers and agencies.

These two challenges do not include the complexity of econometric modelling and experimental design. Although those topics are complex, there are people who can help. No, the two problems are:

  1. Being asked to measure social too late
  2. Not having access to sales data

Measurement needs ‘baking in’ to social

All too often the agency or social team are asked to evaluate campaigns that are about to start, or perhaps underway, and even sometimes that have finished. Occasionally this is possible, but usually it is just folly.

As the Guide points out, the measurement of a campaign is about much more than likes, shares, downloads, and plays. The measurement needs to be in the context of the objectives, and those objectives need to link to things like sales. To measure the complex interactions in social, and between social and other channels, it is necessary to create a plan for what will happen, when it will happen, and how it will be measured. For example, if a campaign is going to be assessed it is best if it does not all start at the same time, and it is best if different regions or groups are exposed to different elements of the campaign.

Asking for the evaluation too late is likely to lead to a report that talks about likes and shares, rather than sales and shifts in likelihood to recommend.

The Guide sets out a five step plan to ‘bake’ measurement into a social campaign.

  1. What is the campaign/activity designed to do? Defining macro objectives, like sales or likelihood to recommend, and micro objectives, such as downloads, plays, and engagement.
  2. Why social? What is the role of social? Including issues like whether social is being used on its own or in conjunction with other channels.
  3. What decisions will be made on the strengths of the evaluation? For example, are the measurements going to be used to manage the campaign in real-time?
  4. What are the appropriate datasets and metrics? With the objectives and uses defined we can select the right things to measure, and make sure that the campaign is deployed in a ways that facilitate measurement.
  5. Designing the evaluation process? The gold standard is market mix modelling, but that may not be affordable in terms of time or cost. The need for accuracy and the need to be pragmatic have to be traded off to select an appropriate design.

Many of the people we have spoken to say they only get approached at stage 4. In these cases it is essential that somebody takes ownership of 1, 2, & 3. For example that somebody lists the objective, both macro and micro, defines why social was chosen, and highlights the decisions that will be made using the measurements.

Measuring ROI and impact requires sales data

Brands guard their sales data very jealously, for fairly obvious reasons. However, without sales data it is not possible to effectively measure the ROI of most social campaigns. If sales data are not available the teams tend to fall back on measuring the available metrics, such as likes and shares, which in turn devalues the measurement of social.

To measure the impact of social typically requires granular data (e.g. weekly data), it requires data on what happened (e.g. impressions), it requires data on who was exposed to what, and it requires sales data.

If brands want to accurately evaluate social, to properly decide how much they should be spending on it, they need to make sales data available to the teams measuring the ROI – which can either be internal teams or specialist agencies.

Have you had problems accessing sales data when modelling? Do you have any tips to share on how to make the case for more data being made available?


Monitoring Marissa: How Market Research And Yahoo! Face The Same Uncertain Future And What To Do About It

Yahoo’s continued existence probably depends on the success of Marissa Mayer to make the internet giant relevant and engaging to consumers. I think it’s no exaggeration to suggest that ours may very well depend on these same factors.



By Jonathan Deitch, Ph.D.

This essay represents my personal point of view about the challenges we face as an industry that increasingly struggles to acquire its essential raw material: respondents. My belief is unambiguous: we are the problem, and we need to change to survive. We must re-imagine our offer, which will only happen when we understand what the online survey research experience looks like in today’s context from their point of view, from the moment they decide to spend time with us through survey and reward, from environmental context to screen size, from comprehension to duration. We need to re-engage them—and make them want to be engaged— in our journey.

The fast-paced multi-billion dollar world of internet search, advertising, and content is not easy to follow. Tools and practices change quickly. To continue to exist, companies must innovate routinely and quickly. If they don’t, someone else will. A stale experience means fewer eyeballs. When nobody returns, advertisers flee. When advertisers flee, there is less funding for innovation, and so on…

This is effectively what happened at Yahoo and what ultimately led to the appointment of Marissa Mayer. I’ve been following her appointment at Yahoo because, though our industries are very different, I believe we face essentially similar challenges.

The Rise and Decline of Yahoo

Not quite two decades ago, Yahoo started as a small site that aimed to “organize the Web”. This seemingly grandiose mission was possible in those days; the web was still small enough to manually curate. Websites were brochures and offered little to actually do. As a portal for discovery, Yahoo was the unequivocal leader in those ancient times.

A few years later, an upstart company began doing the same thing, except they had automated algorithms that served to weigh a site’s importance by the extent to which others thought it was valuable. They began to lure away Yahoo’s users as their results were seen as more relevant, complete, and even democratic. Yahoo’s importance dwindled, and Google rose into prominence.

History clearly shows who won this battle. Yahoo’s response to Google was insufficient. Google launched new products (like Gmail and Google maps, two products Marissa oversaw) while continuing to improve and capitalize on their core search product. They attracted more visitors, and with them more advertising dollars. Yahoo failed to innovate and steadily lost ground. Google’s market value went up. Yahoo’s went down. Over the last several years Yahoo has gone through one CEO after another, but none have been able to reverse the decline.





Enter Marissa Mayer. An early Google employee, Marissa’s strength was her ability to blend a real understanding of technology with strong vision about how to mold it into appealing products. Her success with Gmail and Maps made her one of the “superstars” of Internet business and Yahoo sought her out because of this. The fact the Yahoo Board chose her over more experienced general managers is all the proof we need to conclude that, to them, perspective mattered more than tenure.

While Marissa’s profile as a leader is thought-provoking, what interests me more are the parallels I see between her mission at Yahoo and ours in the MR industry. Let me be clear here: I don’t believe we have lost our way like Yahoo did. Nevertheless we face similar challenges, the biggest of which is our ability to captivate and retain our audiences.

Marissa’s Plans

So what is Marissa going to do to reverse Yahoo’s fortunes?

The trade press suggests she’s focusing on two things. One of them relates to key infrastructure; the other, to Yahoo’s users themselves.

Yahoo is losing out on advertising revenues in part because they lack top tools. For Yahoo to attract advertisers, they need to be able to deliver highly targeted ads that maximize ROI for all parties. With a new engine to run the business, they hope to revitalize the key driver of Yahoo’s revenue. Yahoo is also betting heavily on mobile. The conventional wisdom is that Yahoo has no mobile products that are on par with Google or Facebook. To think that this is simply an “app” problem misses the point.

For Yahoo, mobile means more than just shrinking content to a smartphone screen. It means understanding that the environmental context, needs, use cases, and methods of interaction exhibited by a person on the go are different from when s/he is sitting at a desk. To be successful at mobile for Yahoo means no less than reimagining their offer by taking their great content and delivering it to people in a way that’s useful, accessible, meaningful, and engaging. For Yahoo, mobile is neither the end nor the means. It’s a change of paradigm, a new way of thinking that is as fluid as the name implies. The endgame is to keep users coming back.

Parallels for MR

What Marissa is doing at Yahoo is exactly what we need to do ourselves.

We are in the middle of our own change in industry infrastructure. Sampling has evolved from the utilities of panels to being the heart of survey ecosystems and routers that are integrated with field and project management platforms. Technology is allowing us to not only leverage the latest in online research and sampling methodologies, but to also more efficiently manage the flow of respondents. These integrated platforms are becoming our new engine and a necessary but not sufficient component of our broad strategy.

What’s our analogy, then, to Yahoo’s quest to become more mobile?

Some would argue (I’m one of them) that we need to (quickly) become more mobile in the literal sense of the word.

Using the Expanded Definition by ESOMAR as the basis for calculation, online only approaches account for 55% of global spend at $33.5B. According to Comscore in 2014 60% of all internet traffic of any type originates on a mobile device. That means that $20.1B of our industry revenues are likely driven by mobile as well.

One simple thing “mobile” means for us is having a survey that displays properly on a small screen. The grid question format, one of our most widely-used (yet indisputably painful for respondents even on big screens), simply won’t work. Fortunately, everyone on the sample and data collection side of the business see this and are pushing to become mobile friendly. The work now lies with helping clients manage through this transition.

Mobile success requires more than new interfaces though. Marissa knows—and Yahoo hired her because—she understands that product success is first and foremost dependent upon understanding how users will engage with that product. With each revolutionary wave of how people engage with technology, Marissa’s challenge is to make sure Yahoo remains connected with its users and engages them with compelling and relevant offers that elegantly package content and context.

If Yahoo can’t make this work, it will never return to its former glory. Indeed Yahoo’s continued existence probably depends on her success. I think it’s no exaggeration to suggest that ours may very well depend on these same factors.

Unlocking Engagement

Our industry faces a very stark future unless we figure out how to reengage with our respondents.

Low response and high attrition rates are the symptoms. In an attempt to cure the disease, we developed new methodologies like routers and river sampling. While these give us greater reach and efficiency, they are mechanical solutions that still treat people as if they were an inexhaustible resource. To use an analogy, it’s like we’ve treated a water quality problem by simply opening the taps wider hoping more water will dilute the contaminants. We still have dirty water and we haven’t done anything to stop the pollution at its source.

To address the real cause we have to understand what our offer looks like from the perspective of our users. This means two things.

First, we must understand that our competition isn’t from other other research firm. It is Facebook, YouTube, Angry Birds, the television, the telephone, the kids, email, text messages, apps, the loud person next to us on the train, and so on. We ask for someone’s valuable time and their full attention when both of these things are in increasingly short supply. We need a better understanding of the constraints when we do this, and we need to change our expectations.

Second, we need to stop treating the respondent as a commodity and his actions as a one-off transaction. The answer here lies in trying to create relationships like those that exist in community panels. Community panels (we used to simply call them proprietary panels!) are making a big comeback in these days of extreme social connectedness. This approach leverages the natural affinity panelists have for the brand sponsoring the panel. Through new types of research-centered participation, people keep coming back. This is exactly what we as an industry need to do more of.

Imagine if our panelists were proud to be a part of the research process? Imagine that, through a satisfying survey experience and the feeling their time was well spent, the average panelist who might ordinarily only complete 5 surveys in her life gave us just one more? The benefits would be dramatic.

Our Actions

The implications of the above reasoning are manifold.

Our survey experience must change. It needs to be top drawer. It must be modern, interesting, and engaging. It shouldn’t be clunky or abusive or difficult to do. Rightly or wrongly, our panelists are comparing us to sites crafted by a corps of super-skilled designers, engineers and developers. They are the benchmark, and we need to put out a much better product, one that feels more welcoming than an application for a bank account.

We need to truly understand what our offer looks like through the eyes of our potential respondents. From the first ad they see enticing them to take our surveys, we then need to maximize the likelihood that they respond accurately and completely, whether they’re on their laptop, or on their smartphone on the bus, or on an iPad while watching television while instant messaging with a friend.

We need to think like brand managers. Retention isn’t just about incentives. Saatchi and Saatchi sell their clients on a vision that says if you give people an experience they love and you earn their respect, they will show you loyalty beyond reason. Is it ambitious to think respondents show us loyalty beyond reason? Maybe. But the fact is we have to aim for this peak. If we don’t, our fate is surely sealed.

We’ve taken the first step in this journey by hiring a senior leader to oversee our engagement efforts.


Marissa Mayer’s challenge is to change Yahoo: to transform it from being a “legacy” Internet company to a modern one, one where users can engage with the company however they want, wherever they want. This is an article of her faith. Yahoo’s existence depends on it.

Though our situation is not (yet) as dire as Yahoo’s, we face the same challenge. We need to transform our industry from being a “legacy” data provider to a modern one. We must reengage our respondents. We need to see the world from their eyes. We need to give them an experience that is more satisfying for them. We need to treat them and their time with respect. This is an article of my faith, and I believe our existence depends on it.


The APPification of Marketing

Brace for the flood of Marketing Apps. Developers will find many new contexts and use cases in which marketers, brands, and other constituencies can interact with consumers and users providing rich experiences.
The APPification of Marketing

Editor’s Note. In Peter Orban’s brilliant new post he focuses on the “appification” of marketing technology, but make no mistake every single point made here also applies to market research. The argument here is not solely about the idea of “mobile apps” (although that is a big part of the story), but rather how automation, data, and technology are working together to create new tools and models for insight generation. Zappistore, Instant.ly, Qualtrics, Toluna, Macromill, and a host of other companies are flooding the market with these new solutions, and that trend will continue.


It was almost exactly three years ago when Gartner Inc. predicted that the CMO would spend more on technology than the CIO. Today’s alphabet soup in marketing conversations well illustrates the radical transformation taking place: CDP/DMP, SSP, WEM/WCM, UX, DSP CRM, SEM, API, RTB, EMM, SEO – quick, which one doesn’t belong here?

In the beginning, there were discrete technical and functional silos: an era where various flavors of marketing and their support infrastructure were isolated and disconnected.

In that era it was the marketing channel itself that created a convenient organizing principle: the discipline and tools were centered on how the consumer was reached. Case in point: TV ads. The creation, production, dissemination, and measurement were all specific to this domain. Similarly “BTL,” “B2B” or “PR” were all served by their dedicated expertise and infrastructures, and the tools of these domains did not cross over to other disciplines.

Rapid and profound changes in technology – notably Mobile, Social, Programmatic and Big Data underpinning all – not only effectively demolished barriers meshing together these domains, but also created new ones: just think about anything pertaining to “real time”, “content marketing” or “attribution”.

How can marketers & marketing technology buyers make sense out of this? What organizing principle & framework will help in deciphering not just today’s landscape, but tomorrow’s?

The evolution of “Mobile” offers an effective and convenient framework. Perhaps not a coincidence considering that Mobile is the biggest catalyst in 21st century marketing.

The mobile ecosystem begins with a base layer: the “invisible” but quintessential telecommunication infrastructure: spectrum, cell towers, routers, etc. On top of this sits a middle layer: the glorious handsets and their slick mobile operating systems: But arguably, the most important top layer is what drastically expands utility: applications. Apps are focused, rich experiences individually or connected to other apps “sitting atop” the base and middle layers, the tech stack.

Similarly, today’s marketing and advertising technology space has three layers:

  1. Infrastructure: big consumer platforms (Google, Facebook, Twitter), the Cloud, databases and development tools.
  2. Middle layer: Data/content management and transaction platforms (DMPs/CDPs, SSPs DSPs WCMs/WEMs, ecommerce, personalization engines, etc.)
  3. Finally, drastically expanding the utility of these layers: Marketing Applications. These applications are simple and purposeful but the technology underneath is anything but. As typical with technology it starts out clunky and complicated but it gets simpler as we wrap unified user experience around it.

Marketing Apps fall into two main categories:

  • User facing apps are manifestations of the Brands/Marketers intent; interactive instances of engagement, e.g. mobile apps, displays and video ads, blogs, search ads, social media, call centers, surveys and other VoC tools, promo/contest tools belong here.
  • Marketer facing applications include business intelligence, analytics, dashboards, asset and project management.

These apps are distributed by various delivery networks. Some have their own, dedicated ways of reaching the consumer (e.g. via email) but most will live on content networks, frequently fully integrating with them. Taking the earlier TV example: the once monolith and isolated domain is dissolving into the various layers already facing users: e.g., a freestanding mobile (video) app, a smart tv app, or integrated into various video platforms on the front-end. On the back end, it will mesh into the marketing analytics/measurement platform and asset/content management systems.

Similar to mobile apps the benefit of Marketing Apps is the combination of single-minded focus on specific opportunities while avoiding the recreation of isolated “point solutions” as apps are connected to a common platform and all the 1st & 3rd party data available.

Marketing Apps improve effectiveness as the consistent approach – of possibly very diverse apps – is rooted in a common (data) platform and deliver a fluid user experience while enhancing the 360 visibility of the consumer. Speed and simplicity of execution improves efficiency. Renting only the apps necessary for the time they are needed means additional savings for marketers.

You will be able to build modular toolboxes, quickly swapping specific solutions in and out yet enjoying the benefit of a unified backend system for process management and measurement. Today marketers still must know how “the watch works” but soon apps will just “tell the time”. They will just get the app that ‘Recruits participants for a Clinical Trial” or “Engages Influencers in Category X” and not worry about the mechanics.

Brace for the flood of Marketing Apps. Developers will find many new contexts and use cases in which marketers, brands, and other constituencies can interact with consumers and users providing rich experiences. We don’t know what they will be like and probably will have to invent a few new categories and names. But rest assured, the Appification of Marketing is on its way.


Data: The New Gold Fever

The next 10-15 years promise to wow us. If the analysts get it right, in the next decade, our homes, cars, and personal devices will be connected and talking to each other. This will be one of the most important changes that our industry has made since we started collecting data.



Editor’s note: Felix Rios of Ugam has never submitted a post to me before, but I sure do hope he sends more like this in the future. He nails the vision of the future of our industry that I wholeheartedly embrace, and he explains it more succinctly than perhaps anyone else (including me) has done here on the blog. This isn’t just our future, it is our emerging present, and we need to be aggressively working to align ourselves to this reality today.


By Felix Rios

I’m a self-confessed geek and an early adopter. I have the inexplicable urge to get my hands on the next technology device that promises to change the world. On a daily basis, I have more sensors and gadgets on me than Neil Armstrong during his first expedition to the moon. Google, Facebook, Nest, Apple, Fitbit and Netflix know more about my life than my mom, my wife and probably myself. I also am aware that this makes me part of a rare minority, however, this plethora of devices and interactions with applications make me a data goldmine!

When I made the conscious decision to live a connected life, I accepted that companies will be mining my data. It’s the price I’m paying to get better and more relevant products. The growth of social media platforms and Gmail may show that other consumers think the same way. We keep signing up for store cards to get discounts, for example. Those pesky store points increase slower than the Mumbai traffic, yet I better not forget to hand out my supermarket card at the checkout!

Silos on the Landscape

While the amount of data that I broadcast on a daily basis is ideal to understand my behavior patterns and preferences, it also shows clearly one of the challenges for the market research industry. My data resides in silos. And the devices, applications and manufacturers are making sure that it remains like this.

If Starbucks could see my Fitbit data, it could suggest one of the lighter options when I walk to the store. It might prompt me to buy a granola bar instead of a double chocolate muffin or a soya milk latte instead of a caramel macchiato. It could even automatically update my calorie count and the impact on my daily steps goal to compensate for the indulgent coffee break.

Siloed data is a big challenge. Both Apple and Google understand it and are trying really hard to push their own aggregation standards. Apple’s solutions are HomeKit and HealthKit. Google’s is Nest (via Thread standard) and they also recently released their answer to aggregate data from health trackers, Google Fit.

A unified standard is the foundation for a universal data aggregation platform that will allow us to get actionable insights at a level that we have never been able to get before.

Let’s Get Our Hands Dirty!

Market research panels and online communities are the natural place for us to start exploring these technologies and the natural place for them to grow and mature. First and most importantly, it is critical that the recruitment process complies with all industry regulations, guidelines and best practices. This gives participants peace of mind that their data is being used exclusively for research purposes.

It is important that we protect and nurture the trust that our participants have in us. We should remember every day that it takes years to build trust, but only seconds to break it, and forever to repair it.

At Ugam, we have experience recruiting highly specialized device panels of physicians and supporting device consumer panels. Our experience has shown us that technology is a very engaging and attractive incentive. With a layer of good and constant communication and the correct level of support, we are able to keep them highly engaged with minimum attrition and high levels of satisfaction.

Already the Internet of Things (IoT), wearables and connected device manufacturers have started to take baby steps in allowing access to some of the data via API. While the concept of aggregating multiple data sources is not alien to the market research industry, in the very near future it will be our bread and butter. As this technology becomes mainstream, our clients will demand it from us.

A few years from now, I can see life for this type of research outside of panels and online communities (e.g., IoT and wearables). I can also see a public marketplace of data where individuals would be able to select what information to share and choose the best bidder. If I am generating all this data, why can’t I just sell it directly? I can even see a band system where my data costs more depending on how granular it is. As participants become aware that their data has monetary value, it will lure more people into this vast and open behavioral data marketplace, making it richer and more dynamic.

The next 10-15 years promise to wow us. If the analysts get it right, in the next decade, our homes, cars, and personal devices will be connected and talking to each other.

In its essence, market research is about observing the few to understand the behavior of the many. Asking questions has been one of the most important ways we’ve had to get data. The new breed of devices that will inevitably be part of our lives will not only allow us to ask fewer questions, but will also allow us to make them more relevant, smarter and contextual. We have to understand this and embrace it. This will be one of the most important changes that our industry has made since we started collecting data.


This post first appeared in the CASRO CXO blog 


10 Predictions About The Future Of The Market Research Industry In The Digital Age

What is the impact of the digital era on traditional market research agencies? If they remain traditional, the impact will not be good!


By Michalis Michael

In the January 2015 edition of Research World – the ESOMAR monthly magazine – there is an article that I co-authored with Dimitris A. Mavros, the Managing Director of MRB Hellas, the third largest Market Research agency in Greece. The article is about the impact of the digital era on traditional market research agencies; our premise is that if they remain traditional, the impact will not be good!

Here are our 10 predictions about the future of the market research industry:

  1. The traditional market research agencies that refuse to change will go out of business
  2. DIY market research will catch on even more and will democratise our sector
  3. Social listening analytics will be a must-have for every marketing and market research manager
  4. Agile research will become mainstream and will be facilitated by online communities
  5. Micro surveys and intercepts will eventually replace long monthly customer tracking studies
  6. Processing behavioural data in motion and delivering real-time micro insights will be a core competence of any insights expert agency
  7. Adjacent marketing services such as customer engagement, enterprise feedback management, customer advocacy, will become solutions offered by the market research companies of the future
  8. Data scientists will be the new insight experts, utilising a lot more predictive analytics than rear-view mirror analytics
  9. The code of conduct of market research associations such as ESOMAR and MRS will be revised as it does not apply to the digital economy. If not, the new breed of MR agencies will refuse to be members of such archaic organisations, and the latter will die out
  10. Nielsen will no longer be the largest market research company in the world


Client vs Supplier next-gen market research interest


The above table from GRIT Winter 2014 more or less confirms some of our predictions; the source of this data is market research agencies and end clients of market research. The social media analytics is interesting because 47% of end clients claim to be using it whilst only 34% of the agencies claim the same. This could mean that other technology companies are being used by the end clients that are not market research suppliers.

I would be very interested to start a conversation with colleagues who have an opinion on the matter. As a company, DigitalMR holds the above positions since 2010, when it was established; in 4 years we did not have to change our minds on any of them, if anything, we see social traction confirming those positions. I am sure there will be more than two opposing views and maybe we can define different narratives and segments of us.

The bottom line is: change or perish. If you are a traditional agency it is not too late. A good first step will be to include in your solutions portfolio: social media listening and online communities. DigitalMR is looking for selected market research agencies to be its partners in certain countries and industry sectors. Please do get in touch, if nothing else, we can have a pleasant chat about the future of market research or have coffee if you are visiting London.


Has Research Quality Really Gone Downhill?

In articles about the quality of consumer insights, a common opinion is that research quality has gone downhill in recent years.  I question that perspective.



Ron Sellers

This post actually started as a reply to Scott Weinberg’s terrific Greenbook Blog post Is Online Sample Quality A Pure Oxymoron?  After doing a little writing in the reply box, I realized my comments were lengthy enough to warrant an actual blog post of their own rather than a reply.

Blogs, articles, and reader comments I’ve seen regarding research quality often have the perspective that quality in the consumer insights industry is worse today than at other times in the sector’s history.  Whether various writers blame this on DIY research, online panels, new methodologies, lack of training, or other reasons, this is a fairly common perspective.

Having been in the industry for more years than I care to admit, I have a somewhat different view.  Yes, quality is often pretty bad today, and I shudder to read lengthy lists of transgressions that Scott and others have personally witnessed.  But I question whether things are worse today than they were in past years.

First, as human beings we have the tendency to focus on recent events and situations and forget about what’s happened before.  This was really brought home to me by reactions to the New England Patriots’ recent Super Bowl victory over the Seattle Seahawks.  For those who aren’t NFL fans, Seattle was three feet away from the winning touchdown with about 40 seconds left.  Seattle has one of the most dominating running backs in the game in Marshawn Lynch, and a terrific running quarterback in Russell Wilson, so everyone expected them to use one of those two to score the winning touchdown by running the ball in.

Instead, Seattle called a passing play, and the ball was intercepted at the goal line to preserve an unexpected win for the Patriots.  After the game, a lot of the talk by pundits and fans alike focused on two opinions:

  • That was the worst play call in the history of the Super Bowl (and even that it was the worst play call in the history of sports).
  • That was the best defensive play in the history of the Super Bowl.

Now, it was a pretty bad call and a pretty great defensive play, but was it really the worst/greatest in all of 49 different Super Bowls?  I won’t get into details, but without much effort I can think of two other plays that would give it a run for the “best defensive play ever” title.  But because it’s what we just witnessed a few days ago, and because many people haven’t seen a single play from Super Bowls back in the 70s or 80s, it’s considered the best/worst ever.

We see the same things when Americans are surveyed about who is the greatest president ever.  Modern names such as Ronald Reagan and Bill Clinton generally outpoll historical greats such as Thomas Jefferson, James K. Polk, or Theodore Roosevelt.  But most respondents experienced Clinton’s presidency, while for most people Polk is just another name they might have heard of briefly in high school history.

So as bad as things are in consumer insights, are things really worse than they were ten, 20, or 30 years ago?  There’s still a problem of unqualified people doing bad research, just using a different methodology.  We still have decision makers cutting corners in order to get the lowest cost possible.  I’m guessing we’ll soon have some of the same issues with galvanic skin response, eye tracking, and any of the newer methodologies as they become more popular.

Back in the days when the phone survey was king, I worked for a boss who ordered 70% listed sample and 30% RDD sample for most studies because using listed sample was much cheaper in the phone room.  His reasoning?  Only 30% of phone numbers (at that time) were unlisted, so he was using the RDD to represent the unlisted phone numbers.  He couldn’t figure out that if 70% of phone numbers were listed, it would mean 70% of the RDD numbers would be listed, so in effect he was running with 9% unlisted and 91% listed sample.  Oh, and clients were never informed about his sample decisions, so they were unaware of the possible quality implications.

I also remember fielding a tracking study by phone in about 1988.  It had ridiculous demographic quotas and could be over an hour long for some people.  In getting it programmed, I came upon a question that made absolutely no sense to me – I didn’t even understand what it was asking.  When I questioned the client, he also had no clue and said it was worthless.  When I asked if we could change or eliminate it, he was shocked – “Absolutely not – it’s a tracking study!”  So we continued to track meaningless data for them.

I remember being a respondent for an in-person interview.  The study was about oil company advertising, and I got to listen to a variety of radio commercials with the name of each company bleeped out to see if I could identify the sponsor.  The audio editing was terrible; the bleeping generally consisted of things such as “Texa-beep” to try to hide the Texaco brand.

At the end of the survey, the interviewer asked my occupation, and I told her I was a project director at a market research company.  She looked at me and said, “I can’t put that.”  I told her that the screener had not included a security question or asked me my occupation, and she informed me that “They just know they’re supposed to ask that.”  I told her I also did some media work for the company, so she lied on the questionnaire and put me as a media liaison so that she could get credit for the interview.

I was asked by one client to falsify data to make sure that their intended advertising campaign would look good in the findings.  I was told by another client to change a question so they could get the answers they wanted, and that I had to  learn that “Sometimes you want real answers and sometimes you want to make sure you get the answers you want.”  Both of these happened back when fax machines were considered to be high tech.

When I took a corporate research job in 1993, the first thing I did was visit all of our vendors.  I monitored survey calls at one phone room and heard interviewers going completely off script and getting into conversations with respondents.  When I raised the point with the field supervisor, she was totally comfortable with what they were doing and saw no problems (needless to say, under my watch they were never used again).

We also subscribed to a number of syndicated reports, including a Hispanic tracker.  When I started digging into the data, I found that the research company was regularly reporting and graphing quantitative data from subsets of fewer than 20 people (without noting the sample sizes anywhere).  When I objected, they admitted they “probably shouldn’t do that,” but no changes were made in future waves (which is why we stopped subscribing to it).

Back on the vendor side, I took over a telephone brand tracker for a bank in 1998.  The previous vendor had first asked people an aided question about where they banked.  Only after naming about six different local banks did they ask “unaided” brand awareness.  No bias there, of course!

I could relate many other horror stories from 20 years ago as well as from last year, but you get the idea.  Are things really worse than they were in the past?  I have no quantitative way to measure that and prove or disprove my hypothesis, but I truly question whether consumer insights quality is worse today.  We had plenty of multi-paragraph concepts we had to read to people by phone, plenty of 30-minute phone questionnaires with lengthy and repetitive grids, plenty of questions which were incomprehensible, and plenty of shoddy sampling and field work back then.  We have many of the same problems today, just with different methodologies and technologies.

Is this even a relevant issue?  I would contend that it is.  For one thing, it is easy to become depressed when we believe that things are going downward, and figure there’s nothing we can do about the trend.  Can you change the whole industry?  Maybe not.  But you can darn well make certain that what you do in the industry is done properly, and you can work to point out the quality problems to those who fail to understand their importance.  If you feel the battle is already lost, it becomes much easier to throw in the towel.

For another thing, it becomes easy to blame certain methodologies for the problem, rather than human greed, sloth, or incompetence.  We have tremendous government waste in our republic, but then again so have countries under monarchies, dictatorships, socialist governments, and communist governments.  Is government waste a function of our form of government or of government in general?  Only in understanding that question can we attack the real problems rather than the symptoms.

Yes, online panel research is often atrocious, but so was a lot of the phone, intercept, and mail research that went on in the past, and so is much of the big data analysis and social media monitoring that goes on today.  We need to attack the root causes rather than the symptoms.

Finally, I want to be very clear that this post is not any sort of attack on what Scott or others have written on this topic.  Scott’s post is what got me thinking about today versus the past, but it was outstanding and I agree with the points he made.  I just wanted to bring a slightly different perspective to the discussion than we often hear about when discussing research quality, because I believe it is an important nuance that deserves some consideration.  This is an ongoing battle, not a recent development.


The Best Super Bowl Ads: Opinions vs. Quantitative Super Test

Well, it is the time of year when America’s greatest sporting event takes place. I speak of course about the race to determine which Super Bowl ad is the best.
Courtesy of Anheuser-Busch

Courtesy of Anheuser-Busch


By Rich Raquet of TRC Research

Well, it is the time of year when America’s greatest sporting event takes place. I speak of course about the race to determine which Super Bowl ad is the best. Over the years there have been many ways to accomplish this, but like so often happens in research today, the methods are flawed.

First there is the “party consensus method”. Here people gathered to watch the big game call out their approval or disapproval of various ads. Beyond the fact that the “sample” is clearly not representative, this method has other flaws. At the party I was at we had a Nationwide agent, so criticism of the “dead kid” ad was muted. This is just one example of how people in the group can influence each other (anyone who has watched a focus group has seen this in action). The most popular ad was the Fiat ad with the Viagra pill…not because it was perhaps the favorite, but because parties are noisy and this ad was largely a silent picture.

Second, there is the “opinion leaders” method. The folks who have a platform to spout their opinion (be it TV, YouTube, Twitter or Facebook) tell us what to think. While certainly this will influence opinions, I don’t think tallying up their opinions really gets at the truth. They might be right some of the time, but listening to them is like going with your gut…likely you are missing something.

Third, there is the “focus group” approach. In this method a group of typical people is shuffled off to a room to watch the game and turn dials to rate the commercials they see.   So, like any focus group, these “typical” people are of course atypical.   In exchange for some money they were willing to spend four hours watching the game with perfect strangers.   Further, are focus groups really the way to measure something like which is best? Focus groups can be outstanding at drawing out ideas, providing rich understandings of products and so on, but they are not (nor are they intended to be) quantitative measures.

The use of imperfect means to measure quantitative problems is not unique to Super Bowl ads. I’ve been told by many clients that budget and timing concerns require that they answer some quantitative questions with the opinions of their internal team, or their own gut or qualitative research. That is why we developed our agile and rigorous tools, including Message Test Express™ (MTE™).



MTE™ uses our proprietary Bracket™ method and a standardized approach to deliver quantitative results in a week’s time and starting from $9,900. So, I decided to see if it would work for Super Bowl ads. Monday morning we launched a survey which measured the relative merit of 36 Super Bowl ads. Respondents were asked to choose their most and least entertaining from groups of 3 ads. The “winners” then played off against each other until the respondent had picked their favorite. Through the use of Hierarchical Bayesian Estimation (the same math that makes Max-Diff or Discrete Choice conjoint work) we were able to produce utilities (which divide 100 points among all the ads tested) and with that quantitatively determine which ad was the most entertaining and how much better it was than the next ad.

The winner was the Budweiser Puppy Ad, which is no great surprise (if you’ve watched the coverage). The margin of victory, however, might be. As you can see in the chart above, Budweiser’s ad got over four times as high a utility score as second place Fiat/Viagra did. In fact, Budweiser’s ad was the favorite of roughly one in three respondents and ranked high for the vast majority. The ad did much better among women than it did among men (though it won with both groups) and scored well with both beer drinkers and non beer drinkers.

The Toyota “Being a Dad is more than being a father” ad came in fourth, but importantly it scored far better among those looking to buy a car in the coming months than among those who do not. Easy enough to forget that the point of these ads is to sell product!

I found it interesting that the Snickers Brady Bunch ad scored well with all age groups under 65. I had theorized that it would score better with those over 50 (who saw it as kids), but the show somehow still resonates 40 years after it went out of production. Only those too old to have “appreciated” the show when it first aired ranked the ad low.

One of my favorites, which hasn’t shown up on many lists, was the Doritos Airplane ad, which our study ranked 5th. Perhaps I’ve just been traveling a lot lately but it spoke to me.

The Victoria’s Secret ad ranked 6th and suprisingly it scored equally among both men and women.

Finally, an ad that I’ve heard a lot of commentators and Facebook posts talk about is the Squarespace “Dreaming with Jeff Bridges Ad” With apologies to the “Dude”, our representative sample ranked it as 35th (out of 36 tested).

We’ll be digging further into the results of this research over the coming days so stay tuned for more and in the meantime if you would like to know more about the methodology or the results, feel free to reach out.


2015 Super Bowl Ads Ranked from the Most to Least Entertaining
#1 Budweiser – Lost Dog
#2 – Fiat 500X Crossover – Viagra in the gas tank
#3 – Snickers – Brady Bunch – You’re not you when you’re hungry
#4 – Toyota – Being a dad is more than being a father
#5 – Doritos – Airplane seat saver with mom and baby
#6 – Victoria’s Secret – I’m in the mood for love, models posing
#7 – esurance – 2 ads: Lindsay Lohan and Walter White
#8 – Bud Light – Life-size Pac-Man game
#9 – BMW i3 – Katie Couric and Bryant Gumbel
#10 – Always – What it means to be a girl
#11 – Clash of Clans – Liam Neeson doesn’t like to lose
#12 – Coca-Cola – Spilled Coke changes the internet to make the world happy
#13 – Doritos – When pigs fly
#14 – Dove Men+Care – What it means to be a dad today
#15 – Mercedes – Tortoise and the hare fable
#16 – Nationwide – Mindy Kaling is invisible, with Matt Damon
#17 – Avocado from Mexico – First draft ever
#18 – McDonald’s – Pay by calling your loved ones
#19 – Microsoft – 2 ads: Braylon’s prosthetic legs and Estellas’ brilliant bus
#20 – Skechers – Pete Rose isn’t supposed to be in the hall
#21 – Skittles – Arm wrestlers battle over the last Skittle
#22 – Nissan – Race car driver and son – Cat’s in the cradle
#23 – Dodge – Centenarians give advice
#24 – Loctite Glue – Everyday people dancing
#25 – Turbo Tax – Boston Tea Party
#26 – T-Mobile – Chelsea Handler and Sara Silveramn one-up each other
#27 – Kia Sorento – Perfect getaway vehicle with Pierce Brosnan
#28 – Wix.com – F. Harris, L. Allen, E. Smith, T. Owen, B. Favre build websites
#29 – Mophie – When your phone dies, God knows what can happen
#30 – T-Mobile – Kim Kardashian West introducing data stash
#31 – GoDaddy – Guy working hard misses the big game
#32 – Chevy Colorado – What if your TV went out…use wifi in your vehicle
#33 – Lexus NX Turbo Hybrid – Make some noise in a glamorous parking lot
#34 – Carnival Corporation – Come back to the sea
#35 – Squarespace – Dreaming with Jeff Bridges
#36 – Cure Auto Insurance – Deflated balls
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Jeffrey Henning’s #MRX Top 10: Getting Behind the Emotions and into the Mind

Posted by Jeffrey Henning Tuesday, February 3, 2015, 14:48 pm
Of the 4427 unique links shared on #MRX last week, here are 10 of the most retweeted.


By Jeffrey Henning

Of the 4427 unique links shared on #MRX last week, here are 10 of the most retweeted:

  1. We Know How You Feel – With computers growing smarter every day, scientists and engineers are developing a way for these unemotional, cold boxes to not only track emotions in real time but identify each one, and, for some, act with the same responses we do. (From The New Yorker.)
  2. Top 100 City Destinations Rankings – Strong travel trends show that tourism in the Asian Pacific region overwhelms the rest, with Asian cities making up a third of the top 100.
  3. How data-driven is your marketing research? – Joel Rubinson reminds us all that data is the key to marketing, and with programmatic advertising, appropriate new product testing, and understanding the consumer life, effectiveness in campaigning can quickly grow.
  4. More people think we’re talking about immigration “too much” – Britons’ attitudes towards the talk of immigration swings in a bit of a different direction over the last four years, according to this Ipsos-Mori research.
  5. ESOMAR Data Protection Checklist – A quick rundown of data privacy regulations, laying them out in terms that researchers use.
  6. Top 20 Global Consumer Trends for 2015 – A white paper for those looking to learn more about consumption patterns, behavior of the consumer, and just what will catch the eye of consumers worldwide this coming year.
  7. MRIA Net Gain 2015 – Jon Puleston: Prediction Science How Well Can We Predict the Future? – If you missed last week’s Net Gain 2015, here are live-blogged notes on Jon Puleston’s presentation on prediction science and how to make predicting more precise.
  8. Emotional messaging “most impactful in wealthier nations” – Emotional messages, specifically in the form of television ads, tug harder at the heartstrings of those that are well off while those who in countries with a low-GDP are drawn to functional messaging.
  9. Social media analytics can’t tell you how to serve your customers – Alexandra Samuel of Vision Critical wants to remind companies that just because a group is vocal doesn’t mean that they represent the majority of customer.
  10. Safety First: The Road to Self-Drive – With self-driving cars already on the road, consumer feelings towards the technology vary in appeal, concerns, and emotions across all ages.


Note: This list is ordered by the relative measure of each link’s influence in the first week it debuted in the weekly Top 5. A link’s influence is a tally of the influence of each Twitter user who shared the link and tagged it #MRX, ignoring retweets from closely related accounts. Only links with a research angle are considered.




The Top 10 Disruptive Macro Trends In Market Research For 2015

Driven by unprecedented demand from the investment community, the Gen2 Advisors team has come up with a list of the macro trends that are defining our industry now, and will for the foreseeable future.
trends (1)

Courtesy of www.innoception.org



By Todd Powers, Gregg Archibald & Lenny Murphy

So far 2015 has been breathtaking in the level of interest being shown in the market research space by investors, Private Equity firms, and acquisitive companies. This should come as no surprise to anyone paying attention to the news in the past few months. PE groups have been involved with SSI, Reputation Institute, Macromill/MetrixLabs, FocusVision/Revelation/Decipher, and Research Now (off the top of my head) and of course VC firms have been injecting capital into SurveyMonkey, Qualtrics, Vision Critical, Pureprofile and a multitude of various and sundry players in the “analytics” space.

Research is hot right now, and one of the biggest reasons is because it is in a massive state of change and disruption, which creates many opportunities for investors.

At Gen2 Advisors, the consultancy I am a partner in,  it’s been a virtually non-stop series of engagements with many investment firms, in addition to the usual work with client-side organizations, start-ups, and established suppliers. In all my years in this industry, and especially during the last few years in my role as an industry analyst, I have never seen this level of demand to understand what boils down to two very basic questions:

  1. What is driving change in this space? 
  2. What does the future look like?  

Those are big questions for sure, and there is no shortage of opinions on the answers or efforts to quantify them (such as our own GRIT study). However, what’s been new in my experience for 2015 has been the volume of requests coming from the invest community, who tend to want to look at macro trends vs. the micro trends of what specific methods will be become the flavor of the week or how much of a shift we’ll see from Qual to Quant. Those questions are important for sure, but charting a five year plan as part of pre-investment due diligence requires taking a broader view first, then drilling down into the micro-trends that are important to a specific company’s product portfolio and positioning.

So, my colleagues Gregg Archibald and Todd Powers (who also graciously listen to my own ramblings on these matters) have been working on summarizing these macro trends that apply to the whole industry, regardless of whether you’re a client or supplier, quallie or quant shop, tech company or full service consultancy. Our belief is that these trends are defining our industry now, and will for the foreseeable future.

Here is our take on the 10 disruptive macro trends in market research:

  1. Transition to Insights: Marketing executives are no longer content to receive results from research providers that are little more than data-dumps. They now demand that MR provide insights, and that these insights lead to improved business decisions.
  2. Commoditization of MR: Many of the traditional methods in market research have been standardized to the point that most suppliers are implementing methods that are largely indistinguishable, and price has naturally emerged as a primary differentiator. This is nowhere more evident than in the sample or online survey arenas, but analytics is moving that way quickly as well.
  3. Diffusion of MR: In the face of new tools, many research needs are being picked up by the one that has the need – marketing. This is expected to grow as more complicated questions are more simply answered. IT, Marketing, Operations and Insights are increasingly owning various aspects of the research budget, and speed, cost, and quality are what they are looking for.
  4. DIY/Home Grown: With companies like Survey Monkey and Qualtrics leading the way, we are seeing more and more research methods offered in a do-it-yourself package, and enterprise organizations investing in proprietary data collection and analysis programs that are perfectly aligned to their needs.
  5. Automation: For years, we have seen the impact of automation in manufacturing as robots and other automated tools are used to provide better quality and/or reduced costs. Now MR is following suit, with tools finding their way into the mainstream. By creating mass efficiencies in the non-human driven processes of MR (sample, field, analysis) via highly templated, business-issue focused offerings the era of “Cheaper, Faster, Good Enough” is firmly in play.
  6. Consolidation in MR: Larger research providers are acquiring the needed skills and technologies for them to remain competitive across many sectors of the market, squeezing smaller operators in the process. The question today is that many new companies emerging are cannibalizing the revenue from the larger firms, so the next wave of consolidation may be harder for them incorporate into their businesses. 
  7. Private Equity Investment: As new disruptive tech companies cause competitive price & speed pressure and large players look for acquisition targets to sustain growth, the PE community is funding the development of a new tier of “mashups” and fast growers in the middle market. These mid-market players have less to lose from the disruptive start-ups, thus consolidation may come more from them vs. the “Big 4″.   
  8. Digital Dominance: The pervasive influence of the Internet (and it’s connected technologies like mobile and IoT) into all aspects of business and consumer life has changed almost every industry, and MR is no exception. In fact, almost all of the recent innovations in research have grown out of technologies enabled by Web-based designs.
  9. BI Disintermediation: Recently, we have seen a flurry of activity from firms that focus on the Business Intelligence (or BI) space, as they apply their particular skills to challenges in big data that have captured the interest of so many in market research.
  10. Big Data/Data Synthesis: Big data, and its associated developments like data synthesis, has been the cry and hue in marketing and MR for the past couple of years, and this trend is showing no signs of decline. With the mountains of data now captured daily by all manner of devices the challenge is connecting it appropriately and then making sense of it all.

These trends are deeply inter-connected, and while not strictly linear in relationship, there is a certain cascade effect in play, with each impacting and often reinforcing the others.

Collectively they paint a picture of an industry that is facing many challenges, but also has many, many opportunities. Data is the true currency of our world today, and although the types of data and the ways we go about collecting and analyzing it are changing very rapidly our core value proposition of helping organizations make smarter, more impactful business decision has not changed. As we navigate these trends let’s keep that in mind, because it’s the true nature of the value we bring, and that value is only increasing.


Marketing Simplification Is The New Black

Marketers need to embrace simplification and Insights teams need to learn how to bring this insight to marketers.
Marketing simplification is the new black

Consumer life has become too complex with overwhelming numbers of choices at every turn. Complexity leads to frustration, anger, regret and people making bad choices that make them feel stupid. On the other hand, simplicity leads to relief and undying gratitude. Imagine how I felt when I called a customer care number and actually got a person to answer right away without any “press 1 for this, and press 2 for that…”?!

What do people do in the face of overwhelming complexity? We create our own simplification strategy…consciously and sub-consciously…something behavioral economists might call “simplifying heuristics”. Brands per se are the ultimate simplifying heuristic and beyond that, we create short consideration sets that have an unstated but powerful natural rank order. We buy what it is on the top of the list unless it doesn’t offer a feature or price we want. We click the first thing in search results or at the least, what is on the first page. A typical supermarket can sell 45,000+ products but the typical shopper only buys 400 during the course of a year….99% of choices are ultimately irrelevant. How do we handle this? We de-select…we don’t even see 90% of the choices, like the gorilla in the basketball video…it’s called inattentional blindness by cognitive psychologists. Try shopping for groceries in a foreign country you have traveled to…you will gravitate to the global brands you are familiar with whether you buy them in your home country or not!

Consider the success of three firms who are in the simplification business, pure and simple.

  • Google. No company has simplified life more than Google, making search the way we find information, navigate to websites without having to remember the URL, and marking the beginning of how we shop. Yes, Google has simplified the shopping process among an overwhelming array of offerings.
  • Amazon. Who has simplified the complex task of shopping more than Amazon? It remembers us, our preferences, and our wishes. It shows us what others think of the products we are considering. It has made the access to “print” materials instantaneous by eliminating print. It is open 365/24/7 so we choose the store hours. It is always the closest store (right on our lap.) It even gives us apps so we can buy the product we see in a store at a lower price online .
  • Apple. Apple has led the greatest movement of reinventing how something works to make it simpler with the emergence of touch interface apps on smart phones and tablets. We now have the essence of a website distilled down into the most intuitive UI imaginable…an app.

Some, like the book “The Paradox of Choice”, have proposed reducing choice as a way of reducing complexity. That is the wrong way to go! What do Google, Amazon and Apple (with millions of apps in its app store) have in common? They do NOT reduce choice, they use big data and predictive analytics to extract a tiny fraction of choice that is most likely to be relevant to you, the user. Hence, when we talk about simplification, do NOT think reduction of choices; think relevance, personalization, and consistency across screens. Think log-in as the gateway into simplification.

Wherever there is choice complexity, an opportunity for marketing simplification exists. One of the biggest areas is TV viewing. The average person watches TV over 5 hours per day. Many of us have access to hundreds of channels, plus our DVR, plus VOD plus Hulu and/or Netflix; with overwhelming choices what do we do? We restrict ourselves to a small number of alternatives (data I’ve seen suggests that we view 10% or so of the channels we have available.) Imagine how Google search or perhaps Facebook log-in that remembers your preferences and makes suggestions could revolutionize TV program navigation. TV is becoming a big battleground for Google, Facebook, and Apple. I urge them to compete by simplifying choice rather than badges or some other nonsense.

Can we reinvent marketing approaches to simplify shopping? Absolutely! Tesco turned Korean subway stations into grocery stores where you order using your smartphone from virtual products displayed on the back wall and the groceries are delivered right after you get home. You can imagine retailers seamlessly linking online, mobile, and offline shopping with the use of unifying profiles, to create digital shopping lists that are brought into the store and the products on the list light up as you pass them based on beaconing. This is only a step or two beyond what you already see with ScanIt in Stop and Shop.

You can imagine a CEO saying to each direct report, “We are going to do everything possible to simplify our offer and brand experience. How are you going to change your team’s priorities to fulfill this mission?”

Marketers need to embrace simplification…it really is the essence of brand building. Insights teams need to learn how to bring this insight to marketers, developing a metric of what “simple” looks like to be able to test alternative marketing, product, and experience ideas against a simplification yardstick and prove its power at driving sales growth.