Among the many challenges market research firms face, none is more severe or important to solve than the mounting difficulty in securing a representative sample of survey research respondents. The recent UK elections provide another proof point. Smartphone-friendly surveys help, but at the end of the day the real trouble is that panel companies are unable to demonstrate compelling value to their most in-demand audiences and their samples are generating ever more biased data. The only way panel companies can ensure future relevance is by thinking and investing like brands, expanding reach and diversity, offering meaningful engagement, jealously protecting their flock, deeply and respectfully understanding their members, and demonstrating a new commitment to quality.
By Jonathan D. Deitch, Ph.D., Global Head of Sampling, Ipsos
In the dark corners of the bar after a long day of conference presentations, back in prehistoric times before online panels but even as recently as pre-social media, conversations among researchers always turned to one subject: the disasters that had befallen each of us. As a former colleague once quipped, when you’ve worked in the business long enough, you know where all the bodies are buried.
Nowadays the conversations turn to a different subject: the trouble with panels. To be clear, nobody believes there’s a problem acquiring and retaining people who came of age before the Internet and time-shifted TV. The problem is everyone else. Scour the panel companies (or full-service shops with panels) for people under 35 or the parents with kids and you’ll find precious few. Racial and ethnic minorities, too. They aren’t joining traditional research panels any more.
The adoption en masse by these subpopulations of portable, always-connected devices is only a symptom of the industry’s pathology, one which device-agnostic surveys helps to alleviate.
The real disease, though, is that in general panels don’t offer compelling value to these people. The situation is critical in the U.S. and many countries in Western Europe (including the UK) and is becoming so elsewhere.
The impact of this crisis will be a real change in how sample is sourced, how much it costs to access real respondents, and on the future of panels themselves.
The problem is easily summarized. Research participants who have even a modest interest in rewards quickly figure out that, to earn enough points, they have to take way too many surveys, most of which aren’t compatible with their preferred device or the amount of time they’re prepared to dedicate for meager nonfungible compensation.
Those who feel more altruistic or simply want to speak their mind have plenty of direct outlets to engage with their favorite brands or vendors. These outlets often provide recognition in the form of return contact. Likewise, most Internet polls show results immediately. Users get the instant gratification of knowing if their answers are “right”, or at least how others have responded. The research industry rarely offers this. Studies are blinded for fear of self-selection bias. Results are hidden, the confidential property of our clients.
If the slow and steady decline of panel participation is an infection, the industry’s response has been to continue to apply leeches. Acquisition techniques are evolving, but panel management and retention practices have remained static. Engagement is still largely transactional. Competition grows fiercer by the day. Attrition is a big problem.
In fairness, panel companies are caught in a Bermuda Triangle of sorts. They face a sea change in respondent behavior driven by constant connectedness, one they are powerless to control. Their clients understand the need for change but many are unprepared or slow to accept it. Their managers (or shareholders) are concerned about financial risk given the high levels of uncertainty. Like a drug-resistant disease, no existing medications will help.
Building the Panel of the Future
Panels will decay into mere email lists (or social media “likes”) without new thinking and action around every aspect of the respondent lifecycle, from acquisition, to participation, to engagement and retention.
1: Reach and diversity are essential
Having fished the same waters for too long, panel companies have been forced to look to other sources to expand the top of the recruitment funnel. One of first was incentivized networks, or the “river”. Without a doubt, river/non-panel sources will become more important, which requires rebooting the conventional wisdom for many researchers.
Managed properly, the only real difference between a panel respondent and a river respondent is that the latter has decided not to join a rewards community. This is from whence we come methodologically, yet there are clients who consider river respondents to be of lower quality than panelists when, methodologically, the sources are equivalent: neither can be linked back to the population with a known probability of selection. It is ironic that these same people ask what suppliers are doing to combat “professional” panelists. Nobody has found the Goldilocks respondent yet, though we’re all looking. At least in the short term, and probably for longer, the river will continue to supply audiences that are increasingly shunning panel.
Firms will need to invest heavily in making programmatic acquisition work as well. The challenge in a programmatic world is not so much the right demographic profiles, but finding those willing to participate. Buying on impressions (CPM) is an easy way to lose a lot of money if the people one acquires don’t take surveys. The ability to recruit richly at scale and evolve as the market evolves will be an asset that provides competitive advantage.
Implied in this greater attention to reach and diversity is the need for new measures of quality. Panel size and response rate are becoming meaningless. Both can be artificially manipulated in ways that do little to really help us understand bias. Response rate simply isn’t applicable in a non-panel environment. What matters now is feasibility across an ever-evolving panoply of sources. Reach and diversity are the only ways to increase feasibility and reduce coverage bias.
2: Incentives and engagement must be meaningful
Reach and diversity help with the question of “who”, but not “why”. Panels are in desperate need of a new value proposition.
The easiest target for change is the proprietary incentive system, that is, those rewards that only have value through single panel participation. Common sense suggests these are of little interest to the audiences the industry needs the most. It seems safe to conclude that the level of participation required to earn a meaningful reward is too high. Companies that solve the incentive problem will allow respondents to accumulate points in multiple diverse ways and have fungible currencies. There are panel companies that do this already. Incentivized ad networks that form the backbone of many river sample offerings do this as well.
There are more interesting challenges on the incentives and engagement front, though. One is the impact that shorter, device-agnostic studies will have on retention. The burden on the panelist will be less onerous, but the incentive will be lower, too. This seems to still point to the need for transferable currencies.
Another is the possibility of differential incentives by respondent type. The only relationship in the value chain where payment is not a function of demand is between the panel and the general population respondent. Automation and access to data on costs by source could easily lead to highly personalized rewards by profile that improve participation, with the only caveat that companies will need to pay greater attention to fraud.
The third, and perhaps the toughest, challenge is providing the intangible engagement that is generally lacking in most panels. While panels have become more “social”, they don’t compete with other networks for stickiness. Whether it’s sharing how the data were used or permitting comparisons with a peer group, or offering interesting content, panel companies need to appeal to the latent feeling of “interestedness” respondents have that leads them to participate in the first place.
In his recent Greenbook article, Jan Hofmeyr reminds marketers of the importance of working on both loyalty and acquisition. This applies in equal measure to panel companies. They need to acquire and reacquire/retain participants in ways that will force them to pay attention to—or to simply pay more for/to—the people they need.
3: Suppliers need to stop putting up with abusive buyers
More has been written and said on the move to device agnostic questionnaires in the past year than any other topic related to sample. Panel suppliers are shouting the loudest, exhorting researchers to move at a less-than-glacial pace to overhaul their designs.
Panel providers will, of necessity, start protecting their respondents. They will be able to measure survey pain by and construct “buyer ratings”, which, thanks to automation, will be easily calculable as a function of length of interview, device compatibility screenouts, abandon rate, and perhaps others. These pain measures will affect pricing as one would expect: buyers whose questionnaires fall short of the panel company’s standards will pay higher CPIs. At some level, suppliers will turn away revenue, either implicitly through pricing or explicitly by refusing to bid. This is happening already, and frankly it’s long overdue.
Similarly, companies will improve the routing experience, the process of shunting respondents hither and yon from study to study in search of a match, which still largely stinks. In part, it’s because the technology doesn’t move fast enough. In part, it’s because the industry has as yet been incapable of creating standards for passing basic demographic information, thereby requiring respondents to indicate their sex and age over, and over, and over. Many in the industry recognize the importance of standards. Work began at Samplecon and will continue at the Online Sampling Forum prior to the CASRO Technology Conference in late May.
4: Respondents will be deeply and respectfully profiled
Companies that broker survey respondents continue to explore ways in which they can add value. Profiling is one nut that many hope to crack. From micro-surveys to data enrichment to analytics that enable suppliers to infer behavioral and attitudinal propensities from survey responses, there are many exciting prospects.
Having said this, it will surely become more and more difficult to collect profile information directly. The marketing and research industries will be forced to take a far more serious approach to privacy. Privacy is a throwaway thing for researchers. The fact that panel companies generally collect this information without meaningful compensation is proof. Researchers will need to carefully watch the trends in personal data security. Mounting concerns over privacy suggest a future in which, at minimum, there’s little trust for unknown data collectors.
More intriguing are the embryonic marketplaces for personal data, one of which explicitly speaks about cutting out the market research middlemen. Entrepreneurs are helping individuals become aware of the value of their personal information and facilitate its exchange. It’s too early to know whether this will take off, but it needs to be watched.
5: Quality practices will need to evolve
Any research company of appreciable size has automated speeding and straightlining detection methods. These algorithms arose as reactions to clients’ punishing research designs. In a grid-less, device-agnostic world with shorter questionnaires, speeding and straightlining are no longer meaningful problems.
As the recent UK elections have proven, extant practices for ensuring representativity and response quality are insufficient. That the pollsters have suffered this massive a setback should strike fear into the hearts of other researchers, as the pollsters are generally more stringent in their methodology and write simpler, shorter questionnaires.
Panel companies can demonstrate additional value and potentially charge a premium by developing bias correction methods. Two avenues are worth exploring.
One is examining the relative consistency of responses. The technology exists to be able to automatically detect whether Panelist ABC’s responses are consistent with others of her demographic, attitudinal, or behavioral cohort and then decide what to do. Parameters could be set based on tolerance for some numerical deviation, at which point the respondent and his data would be eliminated or quarantined.
Another avenue of interest is asking questions that have gold-standard references. Standard practice is to weight data to correct for demographic imbalances, but more is possible. The essence of this type of bias correction would be to benchmark against gold standard data and calibrate responses to reduce bias.
Cost and Conclusions
I’ve written elsewhere that technology will, in the short term, drive down the supplier’s costs by eliminating manual labor. It will expose market liquidity and reduce the information asymmetry between buyers and suppliers, creating greater pricing parity.
But make no mistake, the marketplace that panel companies and the industry have taken for granted is that of general population research participants. The implication of the above points is that the cost of securing an in-demand respondent is likely to rise. For people with no relationship with the participant, the cost will be direct and steep. For panel companies, the cost will be the investment needed to build compelling value. They will be forced to compete like brands for loyalty and interest, which for many will be an existential challenge. This is one reason why brand-based social communities are flourishing: they are able to capitalize on pre-existing affinity.
It’s not outside the realm of possibility that the market for sample will evolve like the market for bonds. Bond quality is inversely proportional to risk: the higher the quality, the lower the risk. The only hitch in the metaphor is that higher quality bonds have lower payouts down the road, while higher quality sample will cost more up front. As mentioned above, there is (or will be) enough information being automatically transmitted to permit this sort of calculation.
It’s impossible to predict how the factors that are driving sample challenges will resolve themselves. What’s sure is that the forces are inexorable. The worst assumption the industry can make is assuming the future looks like the past. Research companies of all stripes, private and public, need to understand that they are not facing marginal cost decisions. Their actions require new thinking upon which sustainable futures can be built.
Copyright © 2015. The Author. All rights reserved.