Survey research has been declared dead before, yet it is still with us. In the ’90s, for example, plummeting response rates were often portrayed as the death knell for survey research. Recently Big Data, Social Media and Biometrics are viewed by some as a grave threat to its survival. But has its death been exaggerated once more? Are traditional survey data utilized to their fullest? Are the New Data in reality an opportunity for survey research?
Editor’s Note: It’s easy to assume that with all of the new tools and data sources coming into play in the insights space that the stalwart survey is no longer relevant. I am not in that camp, and judging by the success of Survey Monkey, Google Consumer Surveys, and myriad other variations on survey models in the marketplace many others see the value of this tool. That said, I am skeptical that the survey will remain the primary tool used in the research process; we are witnessing a transition period now where new approaches are augmenting survey data and I fully expect at the end of the transition for the roles to be reversed and surveys will be used to augment and fill in the data gaps when other data sources are primary.
Today’s post by first time GreenBook Blog author Kevin Gray is a wonderful reminder of how useful the survey is and will remain to be in the foreseeable future. It’s an honest appraisal of the pros and cons of the technique and a great primer on how to think about the role of surveys in the new insights era we are in. It’s worth noting that Kevin is a 25 year veteran of the space, a well respected researcher, and a voice of reason on many discussion threads in the various MR-focused LinkedIn groups. He’s someone I respect immensely and I am thrilled to share this great first post with you; it moves the dialogue forward for us all.
Here is Kevin’s post in defense of the humble survey. Enjoy!
By Kevin Gray
Imagine a world in which every consumer’s shopping and purchase behavior, Social Media and other public online activity are monitored in real time, and that there is an ever-expanding data file for each and every consumer available to marketers.
At some point in the not-too-distant future few technological hurdles may remain in the way of this marketing utopia. There are of course privacy concerns that might prevent this (arguably dystopian) world from coming into existence. But, for the sake of discussion, let’s assume that world is here. Would you, as a marketer, still need to survey consumers?
Yes! One example would be when a product or service is not yet on the market. For instance, you may have a product prototype, or just an idea about a product, and you would like detailed consumer reactions about it. Concept screening is another example. The requisite data are simply not “out there” waiting to be collected.
Back to the Present…
In today’s real world vast amounts of data from sources other than consumer surveys are already available to marketers. Let’s think about customer records for a moment. Customer records are “hard” data on individual customers. They may include detailed transaction and demographic information. These data are core to certain specialized analytics, for example market basket analysis and recommender systems. Many kinds of customer records have been mined for years - this is not a new development and predates Customer Relationship Management.
A limitation of analytics which only make use of customer records is that intangible but important variables such as brand awareness, image and attitudinal data, are absent. To a degree, these can be inferred from past behavior and demographic data that may be attached to each customer record. Social Media, call center records and other external data sometimes can also be leveraged. However, specific information crucial for brand building and other essentials of marketing for individual consumers is mostly lacking. Consumer surveys can be used to plug these gaps.
“But Surveys Are Flawed!”
Consumer surveys are not without their critics, as any marketing researcher knows. One concern is that consumers often can’t answer questions important to marketers with great precision. It’s difficult to remember exactly how many times we bought Brand X in the past three months. We also may not really know why we bought Brand X. Honest answers to psychographic questions can be challenging and socially-desirable responses can contaminate the data. Respondents, understandably, may also be reluctant to reveal information they regard as personal, such as income.
These are long-standing issues in survey research. They can be seriously exacerbated by substandard research design, amateurish questionnaires and poor fieldwork execution. A flawed questionnaire administered to the wrong consumers, combined with a low response rate is a marketing research nightmare.
Why do surveys go wrong? There can be many reasons. The respondent group we interview may be too narrowly-defined because of mistaken presumptions about consumers of the category, for instance. Questions may include legacy items no longer meaningful or items made up on the fly that are ambiguous or even nonsensical to many consumers. In the case of multi-country studies, psychographic items developed with U.S. consumers in mind, for instance, may be irrelevant to consumers in other cultures.
Survey Data Analytics
That all said, there is a legal adage stating that “hard cases make bad law” which may bear on this discussion. Irretrievably shoddy surveys are not the rule, fortunately. Most survey research is valuable to the end users and does fill gaps left by other data sources. Let’s elaborate a bit. Marketers need to drill down and look at the interrelationships among many variables. These analyses are especially useful when they are able to tie together specific kinds of data for individual consumers. Here are a few examples of how marketers can use consumer survey data (and also move beyond standard Field & PowerPoint studies).
· Post hoc segmentation is widely-used to identify consumer groups for targeting purposes. With recent advances in statistics and machine learning, we are now less bound to the customary “Cluster & Pray” approach to segmentation, in which cluster analysis is applied to attitudinal data and candidate clusters cross tabulated against demographics and key consumption behaviors…in the hope they will connect meaningfully.
· Generating a massive number of crosstabs, besides being costly and inefficient, risks finding sexy results that are really flukes. Post hoc segmentation is an alternative. It can also serve as an exploratory analysis tool to develop knowledge and insights about the various ways different kinds of consumers behave. Used this way, segmentation can be thought of as a kind of multivariate crosstab.
· In addition to post hoc segmentation, multivariate analysis can be used to profile pre-defined consumer groups, such as heavy, medium and light category users. A form of data mining, this latter approach to is sometimes called a priori segmentation. There are now many methods at our disposal for these sorts of analyses.
· Key driver analysis comes into play when the objective is to identify important associations between predictor variables, e.g., product attribute or satisfaction ratings, and one or more target variables, such as purchase intention for a brand or overall satisfaction with customer service. It’s typically done with all predictors – independent variables – considered simultaneously, and when performed competently is superior to simple correlation analysis or multiple cross tabs.
· Perceptual mapping gives us rich insights into how consumers see the market. Consumer segments can also be “mapped.” Mapping comes in many flavors. Some methods, like Correspondence Analysis, are particularly useful for revealing brand image profiles and reducing brand size effect, whereby the big brands dominate the map.
· New Product Development (NPD) was briefly referred at the outset. Concept screening, product testing, simulated test marketing and conjoint studies are methods commonly employed in NPD. All require consumer surveys.
It’s not Either/Or (with apologies to Søren Kierkegaard)
Luckily, there is no stark “to-survey-or-not” choice facing researchers. Rather, there are many ways to combine data from diverse sources and capture the synergies among them. For example, if you have a customer database it’s now quite easy to draw a sample of your customers and interview them. In this way the “hard” and the “soft” are integrated into one data file.
Taking this idea a step further, you can also incorporate your customer sample into a Usage & Attitudes study conducted among a more general population. Respondents would thus include your customers and users of competitor brands. You will not have access to competitors’ customer records, but with modern statistical methods it’s possible to impute missing data for competitors’ customers (though this should be done with care).
Survey data, such as advertising and brand awareness and brand image, can be tracked over time and integrated with sales and other marketing data from various sources, including Social media. Simple correlations and graphics can uncover important patterns among these variables. Sophisticated market response/marketing mix modeling, in addition, can be used to assess marketing activity in detail, including competitors’, and improve marketing at both the strategic and tactical level.
Another approach, still quite new, is tracking Social Media activity (with permission) of respondents who have also participated in a consumer survey. One application is in advertising research.
While the foregoing are not fresh innovations, they probably are not utilized to their fullest. This brief overview has highlighted quantitative consumer surveys but it also seems doubtful that synergies among qualitative research and the New Data have been fully explored.
American writer Mark Twain once famously declared that the report of his death was an exaggeration. Survey research also has died before but remains alive and well. New data sources will not make it irrelevant – they will make it more valuable.