Insights Industry News

May 5, 2017

This Is A Test. This Is Only A Test.

Experimentation is an often underutilized tool in research that should be used more often to test ideas to see if they truly work.

Steve Needel,

by Steve Needel,

0

Words to live by if you grew up in America during the Cold War (and anyone over 40 is reminded of this in our current political climate).  However, this is not about politics but about understanding a simple fact of a researcher’s life – sometimes you have to test an idea to see if it will work. Unfortunately in our line of work we don’t have a lot of “facts”. Here are some common things we don’t know:

  • Does social media make a difference? There’s some research that says yes and some research that says no. Worse, some of the studies that say a social media push is only effective in specific ways for specific things. Not very helpful, is it? But we want to believe that social/digital media matters, so we keep on trying.
  • How should we advertise? Never mind the newer question of digital versus traditional, which is a whole other issue. What should our ads look like? How should they be delivered (reach, frequency)? On what device(s)? How often should they be changed? You would think that with all the years of research on this topic we would have a pretty good blueprint by now.
  • Does eye-tracking matter? Beyond the simple, “if you don’t see it you don’t buy it” axiom, we haven’t been able to show that attracting more attention to a product increases its sales. Again, we all believe it, we just haven’t proved it.
  • Aren’t all the answers in Big Data? Gee, if I had a dollar for every time someone says this, I could retire soon. Sometimes the answer is in Big Data, sometimes it’s not. Sometimes a database has a great covariance story; every time a brand does this, here’s what happens. That’s a pretty good indicator that if you try it, you’ll come up with the same outcome. More often, you find the data is equivocal, usually because there are additional factors you aren’t considering. Second, you want to try something new, something no one else has tried before. Big Data won’t be that useful in that case.
  • Should brands be stocked horizontally or vertically? This is actually something I know about and the answer, of course, is it depends on the category. Again, not helpful.

 

Experimentation needs to be in our research toolbox. It’s always been a hallmark of the scientific method, just as observation (think ethnography or data-mining) is and just like hypothesis formation is. Whether you are doing a simple online A/B test, a virtual reality test, a controlled store test, or a live test market, sometimes the solution to a whether a marketing idea will work is to test it. Over the 25 years we’ve been doing our research, we’ve found that, contrary to expectations:

  • You can charge more for your product than you may think.
  • Making packages more convenient for consumers may not improve sales.
  • SKU reductions can improve sales for the brand and for the retailer.
  • Shelf signage and displays are not always a good thing – they can actually hurt your sales.

 

Well-designed experiments need not be costly nor do they need to be time-consuming. They can:

  • Provide a causal, rather than correlation-based answer to your question; you know the differences you see are due to the test variable.
  • Reduce the risk associated with a marketing action. How many new products would perform better had they been test marketed?
  • Resolve disagreements on how to market a product when competing ideas exist; test them both and see which wins.

 

Harvard economist Sendhil Mullainathan has a great quote:

No one would say, “Hey, I think this medicine works, go ahead and use it”. We have testing, we go to the lab, we try it again, we have refinement. But you know what we do on the last mile? “Oh, this is a good idea. People will like this. Let’s put it out there.”

Test more.

0

big databrand strategyresearch fundamentals

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

More from Steve Needel,

What If We Gave a Pandemic and Nobody Came?

What If We Gave a Pandemic and Nobody Came?

Since the start of COVID, marketing research blogs have been forecasting dire consequences for the industry. Enough already.

Steve Needel,

Steve Needel,

Data, Data, Everywhere And Not A Drop To Analyze

Research Methodologies

Data, Data, Everywhere And Not A Drop To Analyze

An argument for the importance of quality over quantity in data.

Steve Needel,

Steve Needel,

How to Model Data Incorrectly

Research Methodologies

How to Model Data Incorrectly

A rebuttal to the call to return to simple linear data models.

Steve Needel,

Steve Needel,

That’s a Fact, Jack

That’s a Fact, Jack

A rebuttal by Steve Needel defending the validity of significance testing.

Steve Needel,

Steve Needel,

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*