Research Methodologies

April 14, 2017

The High Cost of Cheap Sample

The fact is, bad or “cheap” sample can give you information that is dead wrong.

Rob Berger

by Rob Berger

0

By Rob Berger

“At least it will give us directional information. I mean, how wrong can it be?” I’ve heard those remarks and many like them bantered about when people rationalize using cheap sample sources.

The fact is, bad or “cheap” sample can give you information that is dead wrong. So wrong that the information it provides is directional—it’s just pointing in the wrong direction.

We’re into the third year of a study where we test the reliability and validity of a popular and well known consumer survey service. We compare the findings of that research with the same data tracked by the Pew Research Center, who are well known for their methodological rigor. The question we are tracking is about the use of social media sites and apps.

The consumer survey results and the Pew Research Center findings could hardly be more different. For starters, the consumer survey suggests social media usage is half the level that Pew and many other sources say it is at.

The consumer survey source says, for example, that 39% of online Americans are using Facebook. Pew puts that number at 79%. For other social media sites the differences are equally stark. Furthermore the consumer survey data would suggest great volatility in the use of social media—with vast surges and declines. Pew shows a slow steady growth.

Why this discrepancy? We believe a lot of it has to do with why people are completing the survey. The consumer survey source obtains respondents by working with publishers to intercept people who are seeking to access “premium content” on their sites. Potential respondents are asked to answer a few questions in order to get access to content. In a whitepaper on this topic we consider a host of potential reasons, but the one that seems to be the most important is respondent motivation.

These people are not taking the survey because they want to. They are doing it to get to their desired content. They don’t have any stake in how they answer—in fact, the question is just a nuisance. When you treat people like that, it is no surprise that the data they give you may be inaccurate. Why would they bother? After all, they have just been frustrated in their pursuit of something they want.

That’s why we take respondent engagement so seriously. Whether it be on our Market Communities—recruited to be representative of the US and Canadian populations—or our client’s specially recruited Insight Communities, we take care to ensure respondents know that their opinion is valued.

We’ve researched why people respond, and we work hard to let respondents know their opinion is important to us and that their input makes a difference. We strive to provide them with feedback on what we’re learning and we try to expose them to interesting new things.

When we invite people to join our communities we mean community in the fullest sense of the word. That’s an important part of what allows us to collect accurate and consistent information to inform our client’s decision making.

People conduct research to help them make better decisions. When the research is wrong, they make bad, even terrible, decisions. In those cases the data is hurting rather than helping them. That makes cheap sample very expensive.

To learn more, download our whitepaper The High Cost of Cheap Sample: Evaluating the Reliability and Validity of a Publisher-driven Online Sample Source.

0

consumer researchdata qualitypanelsrespondent engagement samplesurveys

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*