Counteracting the Dirty Little Secrets of Online Panels

By Ron Sellers

In my last post, I detailed some of the issues and concerns in online panel research (from the Grey Matter Research report More Dirty Little Secrets of Online Panel Research), including the following:

• Panels that send respondents 50 – 60 survey invitations per month
• Overly long questionnaires and/or questions that are impossible to answer
• Panels that allow respondents to complete survey after survey in one sitting
• Absurdly low incentive amounts
• Surveys that are in the field a mere matter of hours
• Panels that insert their own questions before yours
• Studies that were irrelevant to their target population
• Panels that carry advertising on their website or give respondents sales messages under the heading of a “survey”

There are some really, really bad panels out there.  The question is, how can we as researchers navigate these treacherous waters?  In short, what can we do to avoid these bad panels and practices?

First and foremost, we must pay attention to the field.  Quite frankly, fieldwork is not exciting.  One might even say it’s boring (apologies to field specialists out there!).  But it is also critically important.  It’s much like the chassis in a car – forgotten under the sleek exterior and plush interior, it’s still what holds the entire car together.  It’s not enough to hand your project over to a research vendor or a panel company and wait for the resulting data.

Decide what you actually value in a panel.  Do you care what the incentive is?  Do you care what the field time is?  Do you care whether your project is subcontracted to two other panels in order to fill the quotas?  Do you care whether or not they use a router?  Why or why not?  Think strategically on this.

Ask questions of the panel provider.  What incentive will you pay respondents?  How long will you keep the field open?  Will I be able to get the data from the mid-interview terminates and/or the unqualified respondents?  Can you complete this entire study off your own panel or will you be subcontracting it?  If I’m using profile information from the panel (e.g. I only want married respondents), what proportion of your panel has that information attached and how recent is the information?  Make sure you’re comfortable with the answers.

Then set expectations.  For instance, Grey Matter Research has clearly communicated to all of our panel vendors that all respondents must come from their own panel (no subcontracting), that they are not allowed to insert their own questions before our survey (no portal use), and what the field time is to be on each study.  We review and test all survey programming before the field.  We get the data from the unqualifieds and mid-interview terminates, looking for any bias in who was invited to take the study and/or who didn’t make it all the way through.  We review the data afterward, looking for speeders and straightliners (even if the panel company promises to do that).  If they deviate from our instructions, they don’t work with us any more.

When the survey is in the field, monitor the progress.  Be involved.  A research vendor can’t effectively put a survey into the field for 12 hours and skim a convenience sample off the panel if you’re monitoring the progress of the study and have agreed that the field time will be four days.  Set expectations with the vendor for what you want out of the field management.

Pay attention to research on research.  In addition to Grey Matter’s reports on panels, there are other resources out there, such as the Grand Mean Project from Mktg. Inc.  Read.  Listen.  Talk to other researchers.  Talk to the panel companies themselves (or make sure your vendor is doing these things).  Read panelist comments at websites on which respondents discuss panels and panel members (such as www.surveypolice.com).  Learn.

Do the research the right way, regardless of the methodology.  A nonsensical question is a nonsensical question, whether it reaches the respondent through an online panel, a mail survey, their smartphone, or a landline phone call.  I will never remember exactly how many times I have rented a car in the last 12 months, no matter what you want out of me as a respondent.  And I will lose interest after 40 minutes, or after a few lengthy, complex grids.

Page 1 of 2 | Next page