It is tempting to try to reduce the industry’s challenges to bite-sized pieces, as if using big data, understanding millennials, or telling better stories might turn the tide. These things need to be done, but the issues are bigger.
Editor’s Note: Earlier this year JD Deitch wrote what I consider to be one of the best deconstructions of the systemic challenges facing the market research industry I have ever read. I was so impressed I asked him to come to IIeX in Atlanta to present his thoughts as well. Last week JD asked if I’d like to post a follow-up, and I jumped at the opportunity. As we close out 2014 and turn our eyes towards 2015, this should become a foundational POV to inform your strategy for the foreseeable future.
Jonathan D. Deitch, Ph.D. (Connect on Twitter @JDDeitch)
About ten months ago I wrote an article entitled Disrupting Market Research: Five Keys to Survival. In it, I observed that, like most sectors, traditional market research is being disrupted by new techniques and environmental factors that pose an existential threat to laggard firms regardless of size. I concluded that, to merely survive, research companies needed to stop talking about value and start delivering it, and that this would only happen if they created game-changing assets and approaches that gave clients the tailored services they wanted.
My argument was predicated on changes in spending as data penetrates the business world, changes that have effectively ended the monopoly on budget and methodology traditional researchers once enjoyed. From 2010-2012, spending on traditional MR remained virtually flat, while spending on new techniques grew at a healthy double-digit rate.
With the recent release of 2013 estimates from ESOMAR (as estimated by Outsell) I thought I’d update the original article and reevaluate its themes. The data continue to point to the same conclusions. The industry is innovating and incumbents are facing stiffer headwinds. Worst of all, research participation is now at the edge of an abyss.
2013 Spending: Traditional MR still losing ground, online trends down
Spending on traditional MR in 2013 grew by a scant 2% compared to 8% growth in what ESOMAR calls the ‘expanded market’. Traditional MR thus continues to lose ground to new techniques at what is roughly the same pace as previous years. Moreover, the share of spending on online research declined by 1 point, meaning about $600 million shifted to other modes. ESOMAR attributes these factors to price competition in online research and competition from other methodologies, explanations which are entirely consistent with my observations in January’s article.
For sample and panel companies, one would expect similar topline headwinds as traditional MR companies are their core clients. Nevertheless, that sector grew by about 5%. There is evidence that sample/panel companies are diversifying revenue sources, e.g., by going directly to end clients or otherwise leveraging their technology investments.
Meanwhile, new sectors of the insights industry continue their strong growth. Online analytics, web and social media research, social media communities, and marketing reports and research grew by at least 10%. 2013 saw a very healthy 13% increase in the survey software sector as well, which seems a clear vindication of DIY.
2014: More of the same?
In my original article, I postulated that industry growth would top out when clients, recognizing they were spending ever more money on insights, would start making zero-sum decisions, i.e., they would redirect budget from traditional techniques to new methodologies. While the “legacy” MR industry may have staved off the worst in 2013, there are ill winds blowing in 2014.
Dark clouds for traditional MR firms
In July, two of the biggest traditional MR firms announced first-half revenue shortfalls and warned of full-year consequences. The capital markets delivered swift judgment. In October, one the biggest conglomerates showed a revenue decline in Q3 and warned of weaker sales to come. The weakness appears to be hitting hardest in mature markets.
While they don’t report their finances like publicly-quoted companies, the “upstarts” show no signs of flagging. The leaders of the insurgency might well be not one but two DIY (do-it-yourself) survey software companies, both of which have putative valuations of over $1 billion, making them worth more than all but the biggest players in the industry. At what point do they “acqui-hire” a full service firm for some additional research bona fides for clients who might need higher-level consulting?
Trends to watch: Research “off the rack”
Keeping with the DIY theme, for years now insights clients have been complaining about the haute couture world of full-service research with its high prices, slow delivery, and methodological ostentation. In response, researchers have argued back that DIY solutions are like buying factory seconds: you get what you pay for.
Enter “Ready-to-Wear” Research. A UK-based company offering services in Europe, North America, and Australia, now allows customers to buy, with a credit card, the industry’s leading intellectual property for concept testing, copy testing, and brand measurement (with other products coming soon) in explicit partnership with the IP owners. By buying “off the rack”, clients don’t get every bell and whistle, but they do get reliable results using proven techniques, faster and at a lower price, which is exactly what they have been saying they want. This hybrid approach—not exactly DIY but not exactly full-service—gets sample via an API from a large global sample company. The whole thing oozes efficiency, explicitly addresses the issue of quality, and appeals to clients’ own personal consumer experiences of getting just what they want while executing the transaction quickly and conveniently.
This company is not alone in its efforts to create a more templated solution. Many research companies are trying to find the sweet spot between their full service offerings and “lighter” designs that are faster and cheaper. The limiting factors are twofold. One is the operational challenge of simplifying. The other is the deep concern that the lighter product will cannibalize revenue.
Trends to watch: Automation
With sample and panel providers leading the way, the industry is now headed full speed into programmatic execution. From lower operations payrolls to highly efficient and cost-effective matching and delivery of respondents to studies, automation holds a lot of promise.
Suppliers are now offering pricing advantages for using self-service APIs instead of their human project managers. Automation may be both blessing and curse, though. In the longer term, automation must surely reduce the information asymmetry between sample buyers and suppliers (which currently favors suppliers) as it promises far greater price transparency and liquidity. This, in turn, should promote greater pricing parity and create both top- and bottom-line pressure. Of course there is more to sample than just price—respondent quality and a firm’s reliability are important considerations, too—but at some point this information will become equally available and get priced in.
On the acquisition side, panel builders continue to recruit in the fully automated and magnificently intricate digital media ecosystem. The process is entirely data-driven and managed in real-time: predictive models determine, in a matter of milliseconds, whether the person currently looking at the screen (whose demographics are inferred and appended from big data) is likely to be a panel joiner to determine whether they should bid for the ad spot and show creative that has been optimized for the body viewing the screen. The competition for “eyeballs” is fierce. The advertising and affiliate networks and co-registration marketing companies who have traditionally provided the industry’s panelists are also showing a willingness to experiment with new techniques to drive conversion.
Tipping Point: Smartphones and Young Respondents
Despite all the talk by both suppliers and end clients, and despite more than adequate research-on-research to provide guidelines, the industry is moving at a snail’s pace in its adoption of mobile-first, device-agnostic surveys—and it’s starting to materially affect feasibility in younger audiences.
Even a casual review of research company websites and apps provides ample evidence that there isn’t nearly enough inventory of smartphone-friendly completes. Users give middling ratings to mobile apps and remark that “there are never any surveys available”. In app, users still get diverted to the big-screen versions of surveys and websites. They watch as the “please wait” icons spin, only to finally deliver the frustrating news that the survey is closed.
The industry has reached tipping point particularly in the US, where somewhere between 30-50% of new panel recruits (perhaps more depending on the weight of mobile in the media plan) are joining from their smartphones. The consensus is that better than half, and perhaps as much as two-thirds, of all emails are opened on mobile devices, which is important as email remains the primary method of informing people about surveys. The numbers skew higher for those under 35 as smartphone penetration is higher.
This is a massive slice of the available universe, one that panel companies have paid for and their clients would love to speak to, that is simply unavailable for projects that are not adapted to smartphones. Being tablet-friendly isn’t enough either: both the user experience and demographics are different. In the US at least, tablet penetration is around 40% and is skewed toward higher incomes, whereas smartphone penetration is 20 points higher and, importantly, shows no significant difference with racial minorities and less important differences by income.
This problem is of particular concern for trackers and normed studies. While one can understand their reluctance to change, it is no longer avoidable. There are only two sustainable options: they either (a) get shorter and adapt their data collection to be device agnostic or (b) they tolerate feasibility declines that lead to trend breaks and bias. Researchers and clients need to confront these issues so they can plan and act deliberately.
If the industry is fortunate, the need to adapt to smartphones will drive greater attention to the user experience and lead to shorter surveys, more reasonable designs, more focused questions, less cumbersome response mechanisms, and ultimately higher participation rates and better data. There really is no other alternative.
Five Keys to Survival: They Still Make Sense
In the context of the above developments in the industry, the Five Keys to Survival from my original article remain relevant.
1: Make money for clients
At the moment, this appears to be taking the form of leveraging technology and tools for greater speed. With data and analytics continuing to be built into execution, the need to undertake a distinct “research project” is shrinking. While this need won’t disappear, it is clear that understanding how clients execute and developing offers that fit their execution is essential.
2: Create new assets
At the moment, this is all about automation and tools. From “ready to wear” research to sample automation to horizontally-integrated field/project management platforms, insights firms are investing in technology that enables them to streamline their own execution to save time and money. If there is nothing else certain about the future of the industry, it is that it will be far less manual than it is today.
What remains to be seen still is whether firms will be able to develop new data assets either from the data itself or the analytics on top. A quick scan of the job boards shows a healthy market for data scientists, particularly in the new expanded sectors of the market. Yet while big data is interesting and necessary, it is not sufficient. Even Google is using survey data to inform its own models. It’s hard to see how the insights of the future would not be driven by a combination of techniques, both innovative and traditional.
Equally uncertain is the state of consumer panels in mature markets. The situation is especially disquieting among young audiences. The industry and its clients would be wise to remember that, still, the decision to join a panel or to participate in research studies remains a function of the participant’s perceived value given the cost. All the technical sophistication in the world will not overcome a poor experience, and the bar for what constitutes “acceptable experience” is always rising.
3: Embed Quality
The industry continues to make great strides in terms of understanding the drivers of quality in research design and sample management, particularly on connected lifestyles and mobile research experience. The work is as good as it’s ever been. Self-service tools are getting better, too.
The real issue here though, still, is around translating the research-on-research into action to improve research participation. The solution is a tricky one for incumbent firms. End clients need to agree to more sensible designs, yet these typically imply not only lower price points but also difficult change. Suppliers aiming to avoid big change may, for a time, be able to maintain revenue, but will unquestionably see margins shrink as studies become increasingly costly to field. It is a quintessential prisoner’s dilemma, and escaping it is the biggest challenge the industry and its individual firms face.
4: Ooze efficiency
I’ve mentioned several examples above which illustrate this point. Efficiency via automation is clearly the focus for many companies as they seek to improve speed and cost. These efforts will surely continue in 2015. Expect to start seeing real differences (and consequences) between those who do this well and those who don’t.
5: Reboot the business model
Are we on the cusp of this heading into 2015? It seems so. News has just broken of a merger between two analytics and insight firms. These companies combine a variety of assets, experience, and global presence to form a wholly integrated, and quite innovative looking, global top 10 competitor. It will be interesting to see if they get traction.
It is tempting to try to reduce the industry’s challenges to bite-sized pieces, as if using big data, understanding millennials, or telling better stories (all big topics in the past year’s conferences) might turn the tide. These things need to be done, but the issues are bigger. I noted in the original article that the biggest mistake firms being disrupted can make is to think marginally when more profound action is needed. Firms need to act strategically by envisioning the business they would build if they were starting from scratch.
Copyright © 2014. Jonathan D. Deitch, Ph. D. All Rights Reserved.