Editor’s Note: Client-side researcher Jason Anderson continues his quest to create a “No BS Zone” within the research blogosphere by speaking out on the reality of some of the biggest issues impacting the business of market research. Although his context is within the gaming industry, I suspect his experience translates across most categories. As a supplier-side researcher, it’s both refreshing and a bit intimidating to have our dirty laundry aired this way, but it is my belief that you can’t fix a problem unless you understand the cause and in this post Jason gets to some of the root business model issues impacting MR. This one is sure to spur some debate and I’m looking forward to hearing what you think! Make sure to check out the comments section to follow along and contribute your own thoughts.
By Jason Anderson
Last month, I saw a presentation from VGMarket (a boutique research shop in the video games space). If you hit up their website, you’re going to see something rarely revealed by a supplier: their prices. Yes, for a firm quote that includes all of the idiosyncrasies of your particular study you’ll need to talk with someone, but if you’re just trying to size up a hypothetical project you can avoid salespersons entirely.
You’re also going to notice that their prices are…well, they’re cheap, by industry standards. I rarely get quotes for gamer-targeted focus groups for $3k/piece. But this isn’t an article about a specific vendor per se — it’s about the seven “truths” they expressed about games research practices.
I agree with all of them. They’ve given me permission to post and comment upon their presentation, so here are 7 distortions that too many companies accept at face value.
BS Story #1: Focus group pricing is fair.
Actually, most of the time it’s not. If a standard focus group or playtesting group is 6 to 8 respondents for 2 1/2 hours, a typical invoice for a US-based group adds up to $6k to $10k per group. In reality, the recruiting is typically outsourced to another firm for $80 to $120 per respondent. Add on some fixed costs for facility, and that’s lots of healthy profit margin.
If you are conducting at least 4 or 5 standard groups, there is no financial justification for a vendor to charge more than $3k to $5k per group, including expenses. I recently hired a quality firm to conduct 8 groups and 20 individual interviews for under $30k. And I got great quality output. Quotes for the same service from other vendors were as much as twice as much.
Lesson: when is the last time you looked underneath the hood of your vendors’ financial model? Are you paying an appropriate margin for the value added? If you’re paying $60k when you could be paying $30k, what is the extra value delivered for that extra $30k?
BS Story #2: You can always trust that your focus group respondents meet the criteria
While I didn’t need a vendor to tell me this for me to know it to be true, it’s refreshing to hear. Most of the time you’ll work with your vendor to develop a screening survey, and you probably rely on that vendor to manage the integrity of the recruiting process.
Most vendors don’t manage the recruit directly, however. They outsource to another firm and use the screening survey as the delivery contract. Those recruiters (now two steps removed from the project) won’t have industry-specific experience; they will have survey script-reading experience.
This is particularly challenging in the games business. Not surprisingly, gamers are great at “gaming” the system and figuring out the right answers to qualify (and earn their money). Weeding out the liars requires in-depth knowledge of games and game genres, and the ability to challenge respondents to prove their claims. For example, if your recruiter isn’t a gamer, and their target is people who have “played required game X for at least 5 hours,” how can you validate their gaming experience?
You can’t. And if your vendor isn’t paying attention to these details (either by doing their own recruiting or micromanaging their subcontractors), you get participants with the wrong background. Which leads me again to wonder why I’m paying twice as much for half the value.
Lesson: Contract management 101. Make sure ownership of recruiting process and quality control is clearly defined and there is explicit accountability.
BS Story #3: You can always trust your online survey sample and its pricing.
VGMarket’s presentation captures the problem succinctly: “Because the end customer has zero visibility into online recruiting practices, the online sample business has devolved into a cesspool of incestuous outsourcing, excessive mark-ups, and rampant fraud.” Those aren’t client-side words, they’re supplier-side.
Why are things this way?
- Repeated outsourcing to sub-agencies and sub-sub-agencies results in costs of $3 to $5 per respondent at the bottom of the food chain, which gets marked up 4x or more by the time the invoice reaches the client.
- Respondents belong to multiple panels, so it’s not uncommon for the same person to take the same survey multiple times as it’s distributed through multiple channels. (Before suppliers start countering with the importance of having solid panel management, I can say that I have witnessed this duplicate-response issue in nearly every panel-based survey I field.)
- “Survey farms,” primarily located in China, Russia, and Brazil, will deliver fraudulent completes for $1 per respondent for any recruiting criteria.
Lesson: You need to know exactly where your sample comes from. And your vendor needs to be transparent about the sourcing.
BS Story #4: Market research requires a lot of time
While some types of research require more time for analysis and processing, focus groups and crosstabbed surveys do not. For me, project turn-around time is a primary factor in whether or not to “do it myself.” If I have good internal attendance at a playtesting or focus group session, I have 48 hours (best case) before the other observers in the room begin distributing their own notes and findings, and I lose control of the message.
In these situations, I need topline reports in 24 hours. I need final reports in 4 days. My business moves fast, and I presume this is true in many other industries.
Lesson: Vendors supporting fast-moving industries need to be able to keep up with the pace of their clients’ businesses.
BS Story #5: “Business Hours”
Game development is notorious for “crunch time” — that point in the project where deadlines and the 40-hour workweek fall painfully out of sync with each other. In these situations, 8-hour days become 14-hour marathons and weekends happen once or twice per month. If I’m working that hard, so should my vendor. Just in the past month, I’ve had to:
- Repair an 18-country email campaign from a hotel room the morning of my brother’s wedding;
- Manage two surveys from a Blackberry at 5am the morning of my TMRE presentation;
- Edit 200MB of focus group videos from a Dropbox while en route to the airport;
- Fix audio wiring problems in my playtesting lab 30 minutes before start time;
- Go from “we need to do Project X” to “Project X is finished and decisions made” in 5 days;
This isn’t about “oh, woe is me” — this is just the way things are in games. I’m 24×7 on call. I need to be able to contact my vendors at 1am on a Saturday and get the same level of service that I would during “working hours.”
Lesson: Your vendor’s working hours should match your own. If you’re in a 24×7 role, you need 24×7 support.
BS Story #6: Research for games is the same as research for other products.
I’m not disparaging research in other categories. All industries have their own vocabulary and norms. Games are extremely sensitive to the lexicon that has evolved over several decades, however, which makes it difficult (if not impossible) to moderate a discussion with only a superficial understanding of the category.
Gamers have invented hundreds of words and acronyms: LFG, GG, LB, RT, UDUDLRLRBAStart, pwn, 1337, raid, crit, epic, headshot, metacritic, FPS, RTS, MMO, RP, frag, ping, PK, mule, 4X, easter eggs, boss kill. It’s endless. So when your participant says “I don’t like playing RP because it takes too long to find a PUG on the LFG channel,” can you translate?
Lesson: You need to work with people that understand your product category. This is easier for some categories than others. In the case of games research, it’s not good enough to have one analyst or moderator that happens to be “a gamer” — they need solid experience in your particular genre.
BS Story #7: It’s OK to make big decisions based on small sample sizes.
Because game development can be so fast-paced, product teams are easily tempted to use the results of a few focus groups or playtesting sessions to make large decisions. The reason: they assume it will take too long to achieve a larger sample, or it will be too expensive, and so it’s better to act with some feedback than none at all.
I suspect this issue may be more common in the entertainment industry than in packaged goods or other products and services with longer shelf life. But it’s still possible to recruit 80 or 100 respondents at a reasonable cost and deliver both quantitative and qualitative feedback. Yes, your confidence level will be lower, but I believe it would be better to do a smaller number of somewhat significant studies than a larger number of purely directional interviews.
Lesson: Make sure your internal client knows how a study’s results can and cannot be applied.