Research Methodologies

September 28, 2015

Can Researchers Trust Online Access Panel Data?

12 “real respondents” were asked to give their honest views on market research. What they said was both fascinating and horrifying.

Can Researchers Trust Online Access Panel Data?
Edward Appleton

by Edward Appleton

Director Global Marketing at Happy Thinking People

0

Editor’s Note: Online panel quality is perennially a hot button issue in our industry. Just here on GreenBook Blog, there are over 140 posts on this topic, many of them from this year alone. However, none are from the unique perspective of a recent client-side researcher who has now joined the “Dark Side” of suppliers, so today’s post from Edward Appleton is important on that front, as well as raising some serious concerns that our industry may wish to sweep under the rug.

Before you dive in though, I feel honor-bound to mention that I know that many panel providers are working very hard to ensure the highest level of quality in their panels already and doing a good job of it so things may not be as dire as they may appear.

It’s also important to note that there is true disruption happening in this industry that resolves many of these issues. The emergence of single-source platforms, programmatic networks, river sampling and true random sampling technology, enhanced verification technologies, and yes,  even big data frameworks, are revolutionizing how we engage, understand, and activate consumers across the entire marketing life-cycle, especially research.  Just as panels disrupted CATI and drove the growth of online research, these new models will usher in a new era of effective research that we are only beginning to realize the potential of.

Current sample providers need to address the concerns raised here, but let’s be cognizant of the fact that change is inevitably coming very soon that will solve many of these issues, while surely bringing up new ones to replace them.


I recently returned from a 2-day MRMW conference about mobile in market research in London. It was in one key aspect an eye-opener.

As a backdrop: the overriding focus was on how technology providers have ingenious plans to further “transform” Market Research – recruiting respondents differently through gaming, using beacons for “in-the-moment” insights, accessing emerging markets through low-tech mobile devices, reducing survey length…Some were more convincing than others, let’s say. Common denominator? Scaleability…..

Very few provided powerful examples – case studies* – of how mobile had actually helped surface insights that otherwise wouldn’t have been possible.

The session that had the strongest impact was one lead by Survey Sampling International‘s Global Knowledge Director, Pete Cape. He had invited 12 “real respondents” into the Conference room and asked them to give their honest views on market research.

What they said was both fascinating and horrifying:

    • money was often the first-mentioned motivator for MR participation
    • many openly confessed to multiple Panel participation
    • not telling the truth was openly admitted
    • taking upwards of 30 Surveys per month (one gent had done 80 in the last 4 weeks) was commonplace.

Many were also extremely critical of Survey design – “most of your mobile surveys are total c**p” was one succinct statement, grids were torn into, many surveys simply not working technically…..

It was a sorry state of affairs all round – Market Research being gamed by savvy pros interested in the money, disdainful of the people responsible. Curiously, the audience loved it. Much laughter. A distinct lack of embarrassment – very odd.

I was aghast, and tried to seek out Mr. Cape at the SSI stand to understand his motivation – why on earth would he do such an apparently self-destructive thing? Sadly, he was nowhere to be seen, so I am guessing as to his strategic intent.

Despite not having hooked up with Mr. Cape – yet, and perhaps he’ll comment on this blog if he reads it – here’s my take:

Online Access Panels – Change Afoot?

I can only assume that SSI deliberately wishes to disrupt the business model of online Access Panels.

Public confessions of misdeeds can be therapeutic, but in a business context, they are likely (hopefully?) more purposeful. SSI effectively let the world see that response quality in online Access Panels is very poor indeed. Why should anyone wish to buy that sort of low quality – regardless of price?

If you are an end-client or a Major MR agency, why on earth would you wish to continue to simply purchase “professional” responses? Or be associated with that in any way?

An Open Secret Isn’t Any Less Shocking

Many people in the audience were less shocked than me – shoulder-shrugging seemed the most common reaction – that’s the way it is, seemed to be the reaction… “Good on Pete for outing it” “You look shocked, Edward – are you going to write a blog about it?” “It’s true what the respondents said, I worked for years for an Access Panel…..”

Does nobody care very much? Is there such a disregard for quality in large areas of our quant. sampling industry that the concept of disruption seems to leave people unmoved? Or are the financials so ugly that a shake-out is possibly even welcomed?

We often talk and write feverishly about the future of MR – but what about the present?

At a guess, I would imagine that at least 50% of all quant. survey work uses access panels – what about all the major MR companies such as TNS, Ipsos, GfK with reputations to manage – is this all so new?

Take it to the client level: what about all the multi-million-pound decision making taken on the basis of this….”c**p”? Do they know about this slight quality issue?

UK retailer Gerald Ratner’s remarks confessing one of his jewelry products was “total c**p” had catastrophic consequences for his eponymous jewelry chain, as some in the UK might remember.

“It’s True…but Keep it Amongst Yourselves”…Beg Pardon??

I checked out Social Media – Twitter first and foremost – for mentions of the disruptive impact of the SSI piece, using the Conference hashtag #MRMW. The result: very few mentions indeed.

Twitter, of course, isn’t representative of an MRX universe; perhaps the topic is too explosive – or perhaps people simply would prefer to pretend the session hadn’t happened. Just shut up and “move on”. A conspiracy of silence?

So what’s my take-out?

Well, recruitment quality is and has been an issue in many areas of the Research Industry for decades.

But if what SSI’s “real respondent” session suggested is in any way representative of the online Access Panel universe, then it has broad implications – “Quality of Response” has been “outed” as very very low. Online access panels aren’t robust.

Worse: many of the more sophisticated parts of the MR toolkit relating to attitudinal measures – derived importance, conjoint, predictive analytics, mixed-modal implicit/explicit measurement, Bayesian statistics….- are pretty irrelevant if the fundamentals are effectively rotten. As for mobile? It hardly matters if it’s “in the moment” if that’s a fake moment.

“The King is dead” is normally accompanied by the phrase “Long live the King” – so I will watch the SSI space and see if the disruptive session was actually deliberate, launched with strategic intent, to see “what’s next.

If not, or if no further explanation is forthcoming, my blog floats off into the ether unnoticed, then I would suggest that the legitimacy of such online Access Panels is seriously questioned – and as such should come with a clear health warning to all future users.

I do hope I was not the only person to be shocked by the session – naivety to me is preferable to cynicism.

Curious, as ever, as to others’ thoughts.

0

data qualitymarket research supplierspanel providerssamplestate of the industry

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

More from Edward Appleton

Interview with Deborah Ancona: Seley Distinguished Professor

Research Methodologies

Interview with Deborah Ancona: Seley Distinguished Professor

Qualitative Research takes the front seat in this interview with Deborah Ancona.

Edward Appleton

Edward Appleton

Director Global Marketing at Happy Thinking People

Is There a Difference Between Consumer Insights and Market Research?

Insights Industry News

Is There a Difference Between Consumer Insights and Market Research?

Edward Appleton explores the difference between research and insights.

Edward Appleton

Edward Appleton

Director Global Marketing at Happy Thinking People

Edward Appleton’s Impressions of #IIeX EU 2017

Edward Appleton shares what stood out at last week’s IIeX Europe.

Edward Appleton

Edward Appleton

Director Global Marketing at Happy Thinking People

A Renaissance of Qualitative Research

We’re witnessing not just a renaissance of qualitative research, we’re seeing a transformation.

Edward Appleton

Edward Appleton

Director Global Marketing at Happy Thinking People

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*