Editor’s Note: In preparation for the last GRIT report Ron Sellers conducted a series of IDIs with client-side MR professionals. Some of those have been turned into interviews for this blog series on client-side views, and today’s is another amazing example of the very pragmatic and progressive view of the evolution of the role and process of strategic insights espoused by many client leaders. Stacey Symonds offers a very grounded and reasoned view into how Orbitz is embracing the best of the industry, both “traditional” and “emerging” and it should be required reading for all suppliers.
Stacey will be one of over 40 clients joining us on stage at IIeX North America in June and I for one can’t wait to chat with her more in person!
By Ron Sellers
This interview features thoughts and observations from Stacey Symonds. Stacey is Senior Director of Consumer Insights for Orbitz Worldwide. In her current role, she partners with a range of internal business leaders to integrate the voice of the customer into day-to-day as well as long-term development. She has over 15 years of experience in client-side customer insights, brand strategy, and market analysis in the automotive, retail, financial services, and travel industries. Stacey holds an M.A. in Applied Social Research from the University of Michigan and a B.A. in Textile/Apparel Management from Cornell University. She serves on the External Advisory Board of the A. C. Nielsen Center for Marketing Research at the University of Wisconsin-Madison School of Business, and lives in Madison, WI with her husband and two energetic kids.
Ron: First of all, what research or consumer insights methodologies or approaches have you used over the past 12 months or so?
Stacey: Well, it’s been a very wide range because in my role I have responsibility for all consumer insights at Orbitz, so that can range from brand tracking to product development where we’re doing some things like workshops with consumers to do some co-creation, to online discussions, to bulletin board discussions. So really it runs the gamut. I would say I skew toward quantitative studies, though, like discreet choice and other trade-off modes.
Ron: Are there any methodologies or approaches you’ve intentionally stopped using, or even considering, over the past 12 months or so?
Stacey: I’m not a huge fan of traditional focus groups with the one-way-glass kind of interactions. I just think there’s too much of a group dynamic that goes on. There are very specific cases where they might be okay to use, and I rarely have those cases. I need decision-making data. I don’t feel like I can get it from that mode. I’ve used them more in other places I’ve worked, but here that’s something I definitely don’t tend to use anymore.
Ron: Is that something you’ve become more uncomfortable with because of some of the new techniques that are out there, or is that totally independent of these new techniques?
Stacey: It’s actually independent of the new techniques. For me, it’s more that I think we were using a screwdriver to hammer in a nail. When I first got to Orbitz a little over three years ago, they were using focus groups for concept evaluation and progression. When you just don’t get a definitive sense of which concepts should move forward, it becomes very subject to interpretation, versus doing a quant study where you can actually do some rotation and get some clarity on what is working. Actually, to your point, though, one thing we are doing is a qual/quant hybrid, which takes the place of a focus group. It gives you a larger sample size and then also lets you explore language and how people are feeling about things, so you kind of get both sides. And in that format, too, the consumers are not sitting in a room, but they’re virtual. So you don’t get that group influence unless you want it to come into play. You can control it, versus in a focus group were you don’t have control over it, and it’s a small sample size, where you just get this influence that you may or may not want in your decision making.
Ron: With all the new consumer insights methods that have emerged over the past five years or so, where do you see the market research or consumer insights industry headed in the next few years?
Stacey: I think there definitely is a move to quantitative. I’ve seen that in a variety of forms, even taking data that might previously have been called qualitative and making it quantitative. Text analytics is a good example. I think that’s definitely moving into a more mainstream space.
The market research that I have seen used most is information that helps you make decisions. And I feel like quantitative helps you do that in a more deliberate and consistent way. So I think that’s been an evolution for sure. Very fast-turnaround, self-service methodologies are here to stay. I don’t think they’re going anywhere.
I think there may be some discussion, too, about the evolution of panels in communities. It’s something I’ve kind of struggled with – where I fit on that continuum. Because I feel like when you empanel somebody, either for just a generic research panel or for an online-community-type engagement, you change something. The moment they become a part of that community, you change how they feel about the company and how they answer questions, so I don’t know how that’s going to evolve, but I bet it will in some meaningful way. Either we have to just acknowledge that and just keep using it because it’s convenient and easy, or we find other ways to do it, like what I think Google’s consumer surveys is doing in going out to a broader audience where they can actually get to people that you may not have in other forums. You may get a rawer, more honest perspective on things when you do that, versus going to a friendly kind of a panel environment.
Ron: It’s interesting – you’re saying you see more of a move to quantitative, and of course the example you gave was something that’s like big data in that it’s existing information that people are now quantifying. But what about the traditional quantitative surveys, whether using an online access panel, a panel you create, a phone survey, even a mail survey? You’re saying there’s a move to quantitative data, but at the same time there are increasing concerns about the representativeness of traditional quantitative methodologies. I think it was Pew that recently estimated that the typical phone response rate now is nine percent. It’s not random-probability sampling when you’re using an online panel. So how do you deal with the lack of representativeness, even as you’re saying more and more you do quantitative?
Stacey: First it’s recognizing that issue where it exists and making decisions given that fact. So I caveat the heck out of things now to make sure people understand. There’s always been a say/do gap anyway. So even if it was a completely unbiased perspective, what somebody tells you is not always what they are going to go do.
For one thing, I argue that we should use multiple methods, and we do. It’s observational plus it’s surveying. But the other thing is I do feel like we potentially need to adapt how we ask questions, and therefore how surveys are done. The method might be still useful where you can focus a person’s attention on something for a period of time, whereas if you’re in social media, you have no control over what they talk about, how long it gets talked about, how deep they go, or how much they understand it.
I think there’s still a role for the survey in the world today. But I do think they need to be made more engaging so that a broader cross section of people will want to do them. So instead of sitting there and saying, “Gosh, I have to sit here and answer 30 questions, text question after text question,” we have more visually engaging, more interactive methods I think could help counter some of the trends we’re seeing here and still give us useful output.
Ron: Where and how do you learn about new research methods and new approaches?
Stacey: Some of it is definitely going to conferences. I think attending a couple of major market-research-focused or advertising-research-focused conferences definitely helps a lot. And definitely scanning publications that are out there, whether GreenBook or Quirk’s, is a really good resource as well. Then just the grapevine. Certainly, the first time I ever heard about Google Surveys was somebody at work who’s actually not even a researcher. They happened upon it when they were surfing the Internet.
Ron: When evaluating an approach you haven’t used before, what are the factors you look at to determine whether it’s something you believe is valid or something that you kick to the curb because you just don’t think it’s usable?
Stacey: Well, a couple things for me. One, I try to avoid the “shiny new object” syndrome. Just because something is new doesn’t mean we should go do it. And I think partially that’s because I’m not dealing with a huge budget. I think even if I did have a huge budget I’d want to use it very judiciously. So I don’t try things out just to try them. I actually want to be fairly educated first to make sure it’s something that could really fill a gap in what we know.
The other is actually more of a cultural reason than anything else. I want to make sure I’m in tune with what my organization needs and can tolerate, in terms of risk and acceptance. I am in a very data-driven organization, but consumer insights is a new practice for them over the past couple of years. So if I went in and said, “Okay, here’s this new, really complicated technique; we’re going to try this,” I think they might be open to it, but whether or not I could really sell it in… I think it would be harder to do if I didn’t feel confident that it really was something I knew could add value. So I’m not so speculative about most things.
Ron: The industry obviously has a lot of what I’ll call the traditional approaches, such as IDIs or focus groups, intercept interviews, ethnography, and all the different survey methods. Which of these, if any, do you feel are still valid and useful today for your work? And which them also do you feel will be valid and useful five or ten years from now?
Stacey: I would say definitely the face-to-face research and survey techniques are just moving online. I think we still use them, but we’re doing it in a way that’s more flexible for the person involved. We have done group discussions by webcam. We’ve used it for ethnography. Where you want to go really in depth with somebody, I still think qualitative questioning is one of the best ways to do that. We just might do it through an online bulletin board exchange, an online diary, or something like that, instead of sitting in a room and doing a one-on-one interview. That I think has changed for sure. And going forward I wouldn’t expect that to revert back. That’s just sort of how I feel about focus groups and where I think those are.
I think the idea of co-creation or discussions with consumers, there’s something to that where you can do it either in person or virtually where you can make it less subject to group think, depending on how you structure it. So that’s definitely something I feel like is evolving into just a little bit of a different space.
Ethnography, I still feel like that’s a great tool. And I think that approach is one that also is evolving with technology. So ethnography now is: How do people use mobile phones? How are they using websites? All of those things. I think it’s the same principles; it’s just being used in a different channel.
Ron: You’re talking about how a lot of different things are moving from in-person to online. How much of that do you think is driven by, or unique to, the fact that the customer interaction with a company like Orbitz is online? Shoppers are not going to the Orbitz store or the Orbitz dealership or taking a package of Orbitz home with them. Do you think it’s influenced partly by the type of company you work in? Or do you feel like if you were working at Post Foods or Walmart you’d have a lot of that same perspective?
Stacey: I think I would still have a perspective that digital is a way to do research that can get you past geographical issues. It can get you to a consumer who couldn’t show up to a group discussion or to an individual interview. You can get to them. So I feel like it’s an access issue. It is true, if I worked for Walmart, I would be in stores and doing things there which might involve talking to people in person, in a store. So there’s some of that. But I do think still the digital piece is something that lets us do more with less as budgets have gotten squeezed. It also has that added benefit that it just takes some of these barriers away that we’ve had before that really might’ve narrowed our ability to reach different types of consumers.
Ron: We talked about both traditional and non-traditional research approaches, but what about some of the stuff that’s a little bit more out there; a little newer? Eye tracking, facial analysis, mobile MR?
Stacey: I’m involved with a project from the ARF (Advertising Research Foundation) which tested ads with various methods of neuromarketing research and biometrics including EEG, Skin conductance, fMRI, and eye tracking along with traditional survey techniques. It’s almost like a bake-off to figure out which of those is better.
I think the cost/benefit equation is still difficult for any of the really new techniques, especially neuro- or bio-related. These methods do tell you if someone has a reaction to something, but it is not always clear if it is positive or negative reaction and how that might impact the effectiveness of your advertising. Eye tracking has been around the longest, and it does tell you how much time someone spends looking at something so you can optimize your use of space and visuals more so than with other methods.
Ron: What about facial analysis or facial coding? Do you have any experience with or thoughts on that?
Stacey: No. That was not part of this study, but we are actually using for the first time now. It’s the same thing as with those other techniques. They’re so subject to interpretation. Yes, there are principles and some things that people have observed over time, but I am still not sure I could go to my CEO and say, “Hey, because this person has this facial expression, we should go do something different.” That said, we may be able to use facial coding to uncover where people say they have a negative stated reaction to something (like more risqué humor), vs what they actually find amusing.’
Ron: Those are all methods that, to some extent, are used in partnership with traditional research approaches. Then there are the completely new approaches that are either totally separate from traditional research techniques or some people see as a replacement, which would be behavioral economics, big data, social media monitoring, and neuroscience. Which of these do you feel like are valid and useful today, or will be very soon?
Stacey: I think social media analytics can be overrated depending on the industry you are in and your context. This is a big disclaimer. We’re a price-driven industry. We don’t have a lot of people out there debating the benefits of Orbitz versus Expedia, in general, or at least it is not done in public. The discourse is more, “Here, I got a promotion code for Orbitz; do you want to use it?”. We do use it for customer service issues or getting reaction to new campaigns. In other cases though, the dialogue is typically not meaningful enough for us to use it in a robust way for generating insight.
Now, if you are like a P&G, for example, I could see where that could be more useful to you to see how products are being discussed and uncover issues. But, again, you don’t control the conversation; you’re just observing it and you’re just trying to figure out which takes a tremendous amount of time to filter through what are people talking about? And it’s probably 1% valuable, 99% not. And the time and effort it takes you to get to that 1%, my sense has been I don’t know if it’s worth it. But again, it could be different in different industries, so I think that one is probably context-dependent.
I think mobile research was one of the other approaches you mentioned. That one also depends on what business you’re in. For us, that’s actually just part of my normal mix of tools I use, because we do have mobile apps and that’s very important to our business. That, to me, is just a channel. It’s not really a new technique, per se. It’s still a survey; it’s just on a smaller device and has different usability. But it’s still a survey at the end of the day.
Ron: What about big data?
Stacey: Big data, to me, can take a lot of forms. I like to use it in a very focused way. So I make sure I triangulate, because you can’t trust any one data source these days; you need to have several to be able to really understand what’s going on.
For example, if I want to understand why people are canceling hotels, I want to look at all the people in our big data who have canceled a hotel. How far ahead of time did they cancel it, and how did they cancel it? Was it on a phone? Was it on a desktop? And then also do a survey of people who canceled and ask them why they did it. So I feel like using big data in focused ways is meaningful.
It certainly can be valuable to have a complete and accurate record of things, because everything’s in there for click-stream data, for example. If we want to understand how many people are leaving our site after visiting a certain page, we have that data, and we have to go and query and find it. And I think actually those front-end tools that sit on top of big data are where the value is. If you can get good tools to do that and help you quickly get to answers, that’s when big data also becomes more useful, is when your access points really enable you to surgically go into it versus trying to deal with all of it. I think that’s maybe where there’s more value as well.
I think just complete, open data mining – potentially you can learn some things, but you have to have the resources to be able to do that. I think it’s hard for a lot of companies to have somebody whose job it is to explore without knowing what they might find. That’s just something we don’t really have the luxury of doing. But if other companies do, that’s great. I just think it’s a huge amount of effort to uncover something without having some type of business question you’re trying to answer with it.
These companies that came in and started with big data figured out there was something missing, and that was, I think, the why, and that’s where consumer insights comes in. Why are people doing this? Why do we see these patterns of behavior? And it’s become a part of their decision making.
Ron: Some of these new approaches require fairly specialized skills that a lot of traditional researchers don’t necessarily have. Do any of these tend to make you nervous about your own skill set or future in the industry?
Stacey: I would say not really. I have sort of an eclectic background anyway. I have a master’s in applied social research, which was sort of a multidisciplinary program at the University of Michigan, and it has a lot of different aspects to it. It has statistics, business social psychology and it had research methods and the cognitive psychology of asking questions. So I feel like I actually had a very good background that enables me to go in different directions. It enables me to go to the quant stuff. It enables me to go to behavioral economics. I would say I don’t feel too worried.
But I do think that if you want to go into research today, you should have a pretty broad base to draw on. If you just learn traditional survey questioning and not the digital world of web analytics and big data, that will probably set you up for failure. It’ll be much harder for you to figure out what’s going on.
I do think, too, there is a flexibility gap sometimes. I find research can be a very logical, process-oriented activity, but I think there’s actually a need for more rapid thinking and flexibility in the field. It may be harder for some researchers to move there, especially if you’ve spent your whole life doing a certain type of research and a certain function, especially in a bigger company. I think it’s harder to adapt after you have been in that place. It certainly depends where you are, too. I happen to be in a job where I have to know all these techniques and use them. That’s just part of my job. For me, I feel confident I can move between methods pretty well.
Ron: When you consider all these different new techniques and approaches, which of these do you feel tends to be more true about our industry today: A) Too many research professionals are dragging their feet and need to get on board with these new approaches, or they’re simply going to be left behind; or B) Too many research professionals are abandoning proven methods and jumping on the bandwagon of these new approaches without sufficient proof that they’re valid or meaningful?
Stacey: I think I’m going to be totally in the middle. I think I’ve seen some of both. And I don’t know which I agree with more. I think there’s a little bit of each of them. I don’t think we should forget about what’s good about traditional research. I feel like sometimes principles are disappearing, like how to ask questions well. That means we’re not asking things in the most effective ways.
I think some of the move to new methods is also causing people to focus less on the fundamentals. That is an issue. On the other hand, these new methods do represent where we’re going. And I think if you want to be successful going forward, especially in today’s world, where people do have to move between companies or between industries, if you’re not open to some of these new things, I think it will be difficult to keep up.
Some of the core suppliers – some of the biggest suppliers in the world – still do rely on some of the really traditional techniques and bear a significant cost structure because of it. And I just think they’re also probably going to have to evolve a bit in order to be successful in the long term.
Ron: When you deal with people who say things such as, “In-person qualitative is dead,” or “Survey research is obsolete,” how do you respond to those folks?
Stacey: I feel like nothing is absolute in this world. At the end of the day, there are different applications where different methods make a lot of sense. And I think the industry might also be different. But I tell people, at least for us, it’s like being a reporter. If you don’t understand all the W’s and the how about something, then you don’t really understand it. You can’t get the what, where, why, who and how from just observing somebody do something, just from social media analytics or just from click-stream data. You need to get the why. And sometimes the only way you can get a why is to ask somebody, or you could just decide, “Okay, I’m just not going to know it.” And then you’ll be less educated for it.
It’s not an all-or-nothing kind of a thing. I think there are benefits in these things; these techniques. There are benefits in focusing a discussion on a topic that somebody may not otherwise think about. I just think to help businesses grow, we’re going to have to use the best of what’s out there in order to really be successful. And some of it is going to be more traditional and some of it’s going to be new. I think that’s where the good intersection of this industry is.
Ron: One thing I have seen with a lot of these newer approaches is that it’s not just, “Here’s a new tool,” but it’s, “Oh, you do this and you can replace this other thing.” For example, “If you do social media monitoring, you can replace your surveys. You don’t ever have to do a survey again.” What does your reaction tend to be when you see that message coming from vendors offering these new tools?
Stacey: You know what? I actually think it erodes their credibility with me personally to say there are these absolutes out there. At the end of the day, it’s sales, so I am pretty skeptical of that. I need to have things be more proven before I’ll buy into them. And I would never buy into that idea that there’s one solution to everything. It doesn’t make sense to me logically, just given my experience.
Ron: With some of these newer techniques that you’re using or you would consider using, are you looking to your traditional research vendors; vendors you have used for traditional research in the past? Are you looking for them to work with you on those approaches, or are you looking for new vendors that specialize in the newer methods?
Stacey: I think it is a little of both. It is difficult and disruptive to change vendors all the time, especially if you have brand tracking, ad testing, or other things like that. On the other hand, I kind of watch. I kind of plant some seeds and I watch and see if the companies I’m working with are adaptive or not. I’ve left working with some companies because they do not seem to be adapting. For me, it’s also about the relationship.
But I feel there are some vendor partners I work with that have clearly evolved, and I’ve worked with them for more than a decade now because they evolved and are proactive about my business needs in a really positive way, and others are kind of left behind because I feel like they’re not evolving, or the way they’re evolving is not as value-added for me. If they’re not moving into these new spaces, in a way, then I feel like it’s sometimes easier to do things myself than it is for me to go through a supplier who does it for me. So it’s trying to find that balance.