Research Methodologies

August 7, 2012

The Black Hole of Research: Dull but Critical

Fieldwork to many researchers is frequently the least exciting part of a project, but it is a critical part of your success as a researcher.

Ron Sellers

by Ron Sellers

0

There’s one area of both qualitative and quantitative research that too often is a black hole.  Researchers can hand their precious project over to someone and expect the results back, but feel they don’t need to worry about what goes on in between.  Unfortunately, what you don’t know can kill you.

Well, maybe that’s a bit melodramatic – but what you don’t know certainly can kill your project.

I’m talking about The Field.  Whether it’s recruiting qualitative respondents, getting panelists to complete your survey online, gathering completed phone interviews, or any other methodology, fieldwork to many researchers is frequently the least exciting part of a project.  Vendors are often selected primarily by price, with the attitude that The Field is pretty much a commodity.  And The Field is a practice that too often is just handed over to someone else, such as a panel provider or a focus group facility, to be completed while the researcher turns his or her attention to reporting or designing a methodology for the next project.

But what goes on in the field is a critical part of your success as a researcher.  There’s very good fieldwork, and very bad, and it’s up to you to know the difference.

For instance, when you contract with a panel provider to have your questionnaire completed, do you know (or care) whether they subcontract part of the sample to another panel?  Whether they insert some of their own questions right before yours, potentially biasing your sample?  Whether they’re going to put it into the field for just one day, skimming a convenience sample off the top in order to move on to another project?  Whether they use a router, or direct invitations specifically to your study?

When you contract with a phone center, do you know whether the work will be done domestically or off-shored to a center in India or Guatemala, where the interviewers will be probing on your open-ends with English as a second or third language?  If the focus group facility tells you it’s not possible to fill your quotas, have you explored exactly why?

Keeping close tabs on what happens in the field is not optional to good research.  It should be a collaborative process, rather than something you simply hand off.  There are some pretty bad phone centers, focus group recruiters, and online panels out there.  There are also things you can learn in the field that will make a difference in your project.

Here’s one example.  Grey Matter Research conducted a mixed-modality survey about non-profit giving a couple of years ago.  In reviewing the data, we found a serious inconsistency:  people who said they had not given to any non-profits in the past 12 months often reported having given toward Haiti earthquake relief just a few minutes later in the survey.  In trying to figure out what was happening, we spoke with the phone interviewers.  They told us respondents were explaining that with the massive Haiti earthquake, they felt they were giving toward a country, so they didn’t consider that to be helping a non-profit.  Without that feedback from the field, we would not have been able to figure out a serious anomaly in the data.

Here’s another.  Years ago, we had to recruit focus groups in a market we hadn’t used before.  We selected a well-known, highly rated facility for the recruiting from our client’s customer list.  Unfortunately, they struggled to get it recruited.  Rather than simply open up the quotas and qualifications (as the facility repeatedly demanded), we dug into what was going on.  Turns out their recruiters were mostly single mothers who dialed while their kids were in school.  Since this was a consumer recruit, that meant most people were getting called during weekdays, while they were away from home.  We learned a hard lesson about the need to investigate the field practices of a new vendor before handing them a project – even when that vendor has a good reputation.

How many callbacks is the phone center making on each number?  Are they using true RDD sample, or listed sample because it’s much more productive (and therefore easier and more profitable)?  Did they randomize the list you provided before they started calling, or are they dialing alphabetically because that’s the order of the list you gave them?

How many other questionnaires has your online panel provider offered their panelists today – is yours the fifth one they will complete in the last two hours?  How complete and recent is the typical panelist profile – are you relying on outdated or partial information?  Is someone cleaning the data you get, looking for straightliners or speeders who completed a 16-minute questionnaire in six minutes?

Is the qualitative recruiter really using your screener as you wrote it, or is that just a suggestion for the recruiters?  When respondents ask them what the call is about, do they say, “Oh, we’re recruiting a focus group among people who use hand lotion regularly,” thereby telling respondents just how to qualify for the groups?  Are they just sending out an e-mail blast to their database, telling respondents they’re recruiting a group among women with post-nasal drip, or are they actually contacting and screening people in an unbiased manner?

In short, how much attention are you paying to The Field – this rather unglamorous but critically important element of your research?  If the answer is not “a lot,” your projects are at risk.

0

qualitative researchquantitative researchresearch fundamentalssample

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.

More from Ron Sellers

Are the Fraudsters More Sophisticated Than the Researchers?

Research Methodologies

Are the Fraudsters More Sophisticated Than the Researchers?

It’s amazing what some people will do in order to make a buck-fifty. Two recent studies have brought to light how sophisticated panel fraud has become...

Ron Sellers

Ron Sellers

Still More Dirty Little Secrets of Online Panels

Research Methodologies

Still More Dirty Little Secrets of Online Panels

Nearly half of your panel data is trash. Here is how to fix it.

Ron Sellers

Ron Sellers

Can Political Polls Really Be Trusted?

Can Political Polls Really Be Trusted?

When political polls fail to predict the exact outcome of an election, maybe they’re not wrong…maybe we are.

Ron Sellers

Ron Sellers

Panel Quality Stinks and Clients Are To Blame

Research Methodologies

Panel Quality Stinks and Clients Are To Blame

Why should panel companies improve their results when clients accept the status quo and won’t pay for better?

Ron Sellers

Ron Sellers

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*