Reality: Mobile Surveys Will Beat Traditional in 2016
By Patrick Comer
If the current trend continues, then mobile interviews will exceed PC conversion in 2016 further disrupting survey platforms and research designs that aren’t mobile friendly. Ink has been spilt over the onslaught of mobile respondents and the total lack of preparation or even care that research agencies and survey designers have for the user experience. I remember a CASRO Tech in NYC when a researcher suggested that we should ‘keep mobile respondents’ out of surveys lest they mangle norms. All of this reminds me of the challenges political researchers face as the number of cell phone-only households grow. My hunch has been that while progress has been hard to see for a number of years, a lot of good work has been done in the background in 2015 to resolve this problem. When I looked at the data, I found that mobile conversions are close to reaching parity with PC conversions. 2016 is the tipping point.
The suggested improvements to be mobile-friendly break down into a few key areas:
- Interview Length: The mobile respondent session seems ideal for a shorter interview length. With many PC based surveys lasting 15 – 25 minutes, researchers should shorten the interview to match the new environment.
- Survey design: Are the questions and survey-flow taking into account the mobile user in terms of screen size but also usability? For example, complex grid questions are the bread and butter of some survey methods, but are unworkable on mobile.
- Responsive Design: Rendering the traditional survey on the mobile browser can be awful. So introducing responsive design to the survey tool allowing any screen size to be presented well.
How well have buyers and sellers improved the mobile user experience and by what method can we judge the change? We pulled the past two years of data from the Fulcrum Exchange* looking for the difference between the conversion rates of mobile and non-mobile respondents. You’ll see incremental improvements began to accelerate during the second half of 2015.
* all data comes from the Fulcrum Exchange from January 2014 to January 2016 including almost 30 million interviews over 24 months and used by over 200 buyers and 300 suppliers across 80 countries.
Mobile interviews are growing fast
Over the past two years, we saw an increase in the total number of interviews (2.6x overall growth). In that same time frame, the number of interviews from mobile respondents grew by almost 700%.
% Mobile entrants and completes – getting to parity
Not only are the number of mobile completes growing but more mobile survey takers are showing up. In early 2014, 13% of all survey sessions coming were from a mobile device. By the end of 2015, this number had increased to 23%.
More importantly, the % of all completes that are taken on a mobile device has risen by 2.5x over the 24 months period reaching 20% of all surveys on the platform. Many will say that that only 23% mobile entrants is low compared to other platforms. My guess is that panel companies over-index historically to PC users given that most surveys wouldn’t render on the mobile device. Is this a learned response?
Rapid improvement in mobile conversion rates
In early 2014, PC users were 76% more likely to complete a survey than a mobile user. Now, we are reaching parity between the two. What’s important is that overall conversion rates in Fulcrum have doubled in the past 24 months… so it’s a tripling of mobile conversion rates in the same time frame. The improvement was even more dramatic over the course of 2015 alone.
What is driving this change?
Length of interview (LOI) is getting shorter not longer for the first time in forever. We looked at the number of surveys under certain LOIs: less than five minutes, less than 10 minutes, and less than 20 minutes. Across the board we see shorter LOIs, but the largest growth area was in the “<5 min” from 2% to 7% of all surveys.
The volume of mobile respondents in surveys is rapidly increasing. Half of all surveys in January 2014 had ZERO mobile responses compared to only 20% now.
So what’s going on here? Why are mobile conversion rates improving?
First, no matter what they say, more surveys are being engineered with the mobile respondent in mind. Responsive design is driving a lot of the change along with questionnaire design in terms of types of questions included. Maybe more importantly the length of interview is decreasing which gives mobile users greater opportunity to complete a survey.
I wouldn’t be surprised to see mobile conversion parity in 2016 and then the most interesting transition will happen… mobile users will convert better than PC. My impression is this will be the turning point for survey design and we will see a rapid decrease and under-performance of traditional surveys as more users and suppliers focus on mobile friendly. Compelling news for those 40 minute trackers… you know who you are… the conversions against those will plummet creating additional delivery and cost challenges.
Additionally, sample suppliers are getting very savvy at programmatic or real-time provisioning of respondents to surveys. Rather than just demographic matching on the fly, mobile conversion has now become one of the de facto variable considered. So a mobile respondent originally intended for a survey with a low-mobile conversion rate may be routed to a mobile friendly one. Obviously, this has positive effects on the respondent experience and supplier delivery but what are the unintended consequences? Is the research biased against mobile responders if the questionnaire isn’t mobile friendly? Perhaps we will see researchers try to implement a mobile quota cell representing the current percentage for mobile penetration.
So we are seeing more questions than answers but the trending is clear: 2015 was the year when mobile respondents were treated better with both buyers and sellers actively working to drive parity. If you haven’t transitioned to mobile friendly by the end of 2016, then you’re research goals will start to suffer.