Can Researchers Trust Online Access Panel Data?
Editor’s Note: Online panel quality is perennially a hot button issue in our industry. Just here on GreenBook Blog there are over 140 posts on this topic, many of them from this year alone. However, none are from the unique perspective of a recent client-side researcher who has now joined the “Dark Side” of suppliers, so today’s post from Edward Appleton is important on that front, as well as raising some serious concerns that our industry may wish to sweep under the rug.
Before you dive in though, I feel honor bound to mention that I know that many panel providers are working very hard to ensure the highest level of quality in their panels already and doing a good job of it so things may not be as dire as they may appear.
It’s also important to note that there is true disruption happening in this industry that resolve many of these issues. The emergence of single source platforms, programmatic networks, river sampling and true random sampling technology, enhanced verification technologies, and yes, even big data frameworks, are revolutionizing how we engage, understand, and activate consumers across the entire marketing life-cycle, especially research. Just as panels disrupted CATI and drove the growth of online research, these new models will usher in a new era of effective research that we are only beginning to realize the potential of.
Current sample providers need to address the concerns raised here, but let’s be cognizant of the fact that change is inevitably coming very soon that will solve many of these issues, while surely bringing up new ones to replace them.
By Edward Appleton
I recently returned from a 2 day MRMW conference about mobile in market research in London. It was in one key aspect an eye-opener.
As a backdrop: the overriding focus was on how technology providers have ingenious plans to further “transform” Market Research – recruiting respondents differently through gaming, using beacons for “in-the-moment” insights, accessing emerging markets through low-tech mobile devices, reducing survey length….Some were more convincing than others, lets say. Common denominator? Scaleability…..
Very few provided powerful examples – case studies* – of how mobile had actually helped surface insights that otherwise wouldn’t have been possible.
The session that had the strongest impact was one lead by Survey Sampling International‘s Global Knowledge Director, Pete Cape. He had invited 12 “real respondents” into the Conference room, and asked them to give their honest views on market Research.
What they said was both fascinating and horrifying:
- money was often the first-mentioned motivator for MR participation
- many openly confessed to multiple Panel participation
- not telling the truth was openly admitted
- taking upwards of 30 Surveys per month (one gent had done 80 in the last 4 weeks) was commonplace.
Many were also extremely critical of Survey design – “most of your mobile surveys are total c**p” was one succinct statement, grids were torn into, many surveys simply not working technically…..
It was a sorry state of affairs all round – Market Research being gamed by savvy pros interested in the money, disdainful of the people responsible . Curiously, the audience loved it. Much laughter. A distinct lack of embarrassment – very odd.
I was aghast, and tried to seek out Mr. Cape at the SSI stand to understand his motivation – why on earth would he do such an apparently self-destructive thing? Sadly, he was nowhere to be seen, so I am guessing as to his strategic intent.
Despite not having hooked up with Mr. Cape – yet, and perhaps he’ll comment on this blog if he reads it – here’s my take:
Online Access Panels – Change Afoot?
I can only assume that SSI deliberately wish to disrupt the business model of online Access Panels.
Public confessions of misdeeds can be therapeutic, but in a business context they are likely (hopefully?) more purposeful. SSI effectively let the world see that response quality in online Access Panels is very poor indeed. Why should anyone wish to buy that sort of low quality – regardless of price?
If you are an end-client or a Major MR agency, why on earth would you wish to continue to simply purchase “professional” responses? Or be associated with that in any way?
An Open Secret isn’t any less Shocking
Many people in the audience were less shocked than myself – shoulder-shrugging seemed the most common reaction – that’s the way it is, seemed to be the reaction…. “Good on Pete for outing it” “You look shocked, Edward – are you going to write a blog about it?” “It’s true what the respondents said, I worked for years for an Access Panel…..”
Does nobody care very much? Is there such a disregard for quality in large areas of our quant. sampling industry that the concept of disruption seems to leave people unmoved? Or are the financials so ugly that a shake-out is possibly even welcomed?
We often talk and write feverishly about the future of MR – but what about the present?
At a guess I would imagine that at least 50% of all quant. survey work uses access panels – what about all the major MR companies such as TNS, Ipsos, GfK with reputations to manage – is this all so new?
Take it to the client level: what about all the multi-million pound decision making taken on the basis of this….”c**p”? Do they know about this slight quality issue?
UK retailer Gerald Ratner’s remarks confessing one of his jewelry products was “total c**p” had catastrophic consequences for his eponymous jewelry chain, as some in the UK might remember.
“It’s True…but Keep it Amongst Yourselves”….Beg Pardon??
I checked out Social Media – Twitter first and foremost – for mentions of the disruptive impact of the SSI piece, using the Conference hashtag #MRMW. The result: very few mentions indeed.
Twitter of course isn’t representative of a MR universe; perhaps the topic is too explosive – or perhaps people simple would prefer to pretend the session hadn’t happened. Just shut up and “move on”. A conspiracy of silence?
So what’s my take-out?
Well, recruitment quality is and has been an issue in many areas of the Research Industry for decades.
But if what SSI’s “real respondent” session suggested is in any way representative of the online Access Panel universe, then it has broad implications – “Quality of Response” has been “outed” as very very low. Online access panels aren’t robust.
Worse: many of the more sophisticated parts of the MR toolkit relating to attitudinal measures – derived importance, conjoint, predictive analytics, mixed-modal implicit/explicit measurement, Bayesian statistics….- are pretty irrelevant if the fundamentals are effectively rotten. As for mobile? It hardly matters if it’s “in the moment” if that’s a fake moment.
“The King is dead” is normally accompanied by the phrase “Long live the King” – so I will watch the SSI space and see if the disruptive session was actually deliberate, launched with strategic intent, to see “what’s next.
If not, or if no further explanation is forthcoming, my blog floats off into the ether unnoticed, then I would suggest that the legitimacy of such online Access Panels is seriously questioned – and as such should come with a clear health warning to all future users.
I do hope I was not the only person to be shocked by the session – naivety to me is preferable to cynicism.
Curious, as ever, as to others’ thoughts.