How TNS Is Validating Mobile Globally
Editor’s Note: Edward Appleton is doing a series of posts focused on the client-side view of mobile research, with an emphasis on use cases and best practices learned so far. This is the fourth post in that series that we’ll be publishing over the rest of the month. Parts 1 – 3 can be found here.
By Edward Appleton
TNS is one of the leading global Market Research agencies, with operations in over 80 countries, and part of the Kantar Group – a leading provider of insights and intelligence. Millward Brown, the Futures Company, The Added Value Company, Kantar Worldpanel, Kantar Retail and Kantar Media all form part of the group.
TNS is taking the lead with a serious look at mobile in market research as a global opportunity.
It was the main sponsor at the recent MRMW Conference in London (Market Research in the Mobile World – 10/2013 – http://bit.ly/1aeYQ6d); they have presented papers and case studies at various industry Mobile Events over the past 2 years.
More importantly, they are building an evidence-based approach to understanding mobile research. They execute annual large-scale quantitative Global studies on mobile – TNS Mobile Life (http://bit.ly/1bloPNd) – covering 43 countries, interviewing just under 38.000 mobile users globally. They are beginning to document the pros and cons of the different types of mobile surveys (Apps, WAP, SMS, USSD).
This empirical approach of global exploration and validation is extremely beneficial to the research industry as a whole, as it will create confidence amongst Client side researchers.
Their approach extends to a healthy skeptical attitude to what they see as massive hype surrounding mobile as a panacea to all marketing ills.
A recent blog on the ESOMAR site by Sam Curtis (http://bit.ly/1etPZFf), provocatively entitled “The Big Mobile Lie”, shared evidence from TNS studies showing mobile devices being hardly used by shoppers at the point of purchase once usage data is broken down to category level – including pet food, alcohol, tobacco, OTC medicines.
So: what’s hype, what’s “for real”? What does TNS’ empirical approach currently reveal about how to best approach mobile research – for what types of study, how best design, what caveats?
I caught up with Sam to understand TNS’ views better.
Mobile: Hype or Seachange?
TNS actually shares much of the underlying excitement for mobile witnessed by many media and marketing organisations.
With the rise of Smartphones, the potential of mobile to change many industries radically is real – retail (“showrooming”), finance (mobile banking, mobile wallets), health (m-health) are examples where swift change in value chains is already happening.
Smart phones transform the device into something completely integrated, offering all sorts of experiences in one go. It offers consumers superior convenience and flexibility to explore, interact, buy – wherever they are, when they want, whilst at the same time offering entertainment (music, videos, games) connection to friends via social media, and the opportunity to keep up to date on events.
My sense is that TNS sees both the excitement and disruptive threat of mobile – something that will likely destroy traditional business models – and wish to be position themselves as the go-to insights consultancy offering marketing business advice on a range of issues beyond research.
Significant investment and senior talent within the Kantar group is being focused on understanding how best to use mobile in research, according to Sam.
In the research world, they estimate that within 2 – 3 years between 20 – 30% of all data collection will be mobile, a massive shift from today’s lower levels.
Equally, they see little evidence yet of a mobile “shopper revolution” – many FMCG items are frequent purchases, habit-driven, with little use of the mobile to check prices or find out more about the brand at the POS.
So – mobile is a sea change, but with significant dangers of unsubstantiated claims – hype – leading marketers to potentially invest indiscriminately into mobile as the hot topic of the moment.
Forces Driving Change
For research, TNS sees the following issues and trends that make them view mobile data collection as a strategically important topic:
- Emerging markets will go mobile first, skipping the phase of laptop/desktop usage which characterizes many developed markets. Emerging markets are huge growth opportunities for many major multinational companies. Missing out here on growth fueled by mobile isn’t an option.
- Dropping engagement levels on online panels. Mobile is potentially an answer to what is a widely recognized (if poorly documented) issue - relatively high churn levels of people dropping off MR panels, with recruitment of new users proving increasingly difficult. TNS sees mobile as a way of attracting new, particularly younger audiences to participate in market research.
- Recall gaps. Our memories are flawed; data sets relying on recall are often not complete and therefore lack accuracy and granularity. Mobile is an “in-the-moment” medium. Not only can we capture the “what” more accurately, but also the way we feel at that point in time. Mobile overcomes the recall gaps.
The most powerful driver to mobile, however, is one that is relevant to the research industry as a whole: the need to do shorter surveys.
Researchers have long been aware that conducting long surveys – arguably anything over 15 minutes – is in danger of being counter-productive: respondents’ engagement levels drop, their enthusiasm for future survey participation falls, and data quality suffers as a result.
Some of the key reasons for long surveys surviving in a world of ever-decreasing attention spans are as follows:
– Comparability: any change in methodology will likely result in the data changing. If you have tracker studies running for a number of years, with internally accepted KPIs, then switching data collection mode needs careful transition, calibration planning, and internal stakeholder sensitization. It is a bold but important call, that requires senior Insights staff that are both respected, well established and confident.
– Knowledge Needs: Clients (often Marketing staff who are the budget holders, but with little or no formal MR training) naturally wish to squeeze every last question in “once we have them there”. Micro-surveys are a relatively new phenomenon – Google Consumer Surveys was launched in 2011, and was limited initially to the US and Canada. The notion of “chunking” will be new to them – education is called for.
– Commercial forces: many Agencies are reluctant to push-back on Survey length for fear of losing a piece of work. This is a spiral towards lower quality and needs to be broken out of, with the more commercially stable industry leaders playing a key role.
– More Complicated Scripting: scripting a mobile survey is no doubt more complex due to the myriad types of devices and operating systems that a given survey has to work with. This requires accessing sufficient staff with the relevant skill sets, investing in that resource. This can easily raise the base level cost of a business, which Agencies may or may not feel comfortable about passing on to clients.
The barriers are significant.
Mobile, however, has witnessed such rapid and widespread adoption amongst consumers worldwide that it represents an irresistible force that Companies – Client and Agency – will have to react to. The phase of “simply being prepared” is probably behind us, knowing how to integrate mobile into the research mix is an imperative.
If – as TNS suggests – mobile actually represents an opportunity by which the respondent experience for all types of MR engagement can be both improved and made more predictive, it has the potential to transform – revolutionize even – the way we conduct huge swathes of survey research.
Mobile: Shorter, with Higher Predictive Validity
TNS is currently conducting research-on-research studies in selected geographies and categories – focusing on brand equity, advertising tracking, customer satisfaction studies – to establish potential differences in response patterns between mobile and laptop/ desktop. They also are aiming to find questions that are the most predictive of actual behavior.
Key survey metrics – including Purchase Intent – are compared across the different devices to actual subsequent purchasing behavior. Analysis is at the more critical respondent level.
The topline meta- finding is startling: shorter, more relevant surveys have a higher predictive validity than longer ones.
This has major implications for interview length across survey type.
TNS can pinpoint which questions in a given survey type are more predictive, correlate well to actual behavior, and eliminate questions that are redundant.
Work TNS has conducted in 3 countries across two categories show the following questions to be redundant, as they have a low correlation with individual purchase behavior:
- aided awareness
- brand familiarity
- brand satisfaction
- purchase intent
- recommendation (NPS)
- brands bought – past 3 months, regularly, most often
- brand attitudes (i.e. ‚this is a brand i trust‘)
Parallel to this work on mobile validation, they are working broadly with clients to implement “survey-length-reduction” principles:
- Respondent Level Validity (only asking questions that respondents can answer accurately)
- Principle of Redundancy (detecting respondent level correlations within-survey and eliminating unnecessary questions by using auto-fill)
- Relevance (only asking questions about the few brands and attributes that participants really care about)
Brand and attribute lists are reduced radically, only one brand-rating question is asked per brand.
The resulting shorter survey (an equity study in this example) can be conducted in 3 minutes with a respondent-level validity of R = 0.62, and an impressive correlation to market share of R=0.90+.
Their conclusion: shorter surveys can give higher predictive validity.
Trackers and brand equity studies that could traditionally require 40 minutes of respondents’ time can be reduced to well under 10 minutes – as little as 3 in the case quoted.
This opens the door for many trackers to transition to mobile, saving time, eliminating respondent fatigue, reducing cost, and delivering better results.
The insight that shorter is more powerful is intuitive: we pay more attention if we are allowed to talk about the few things we care about in a category, and not get forced to answer multiple questions about things we may have no opinion about. This is still the case with many trackers.
It is as TNS states nothing less than a revolutionary approach to surveys – whatever data mode participants choose to respond.
In summary, best practice in mobile could drive best practice across mode type. There are probably few types of surveys that cannot, once condensed intelligently, using sufficient computer power and intelligent programming, be conducted on a mobile device.
Mobile: Increased Accuracy
Alongside forcing Survey designers to think carefully about asking fewer questions, mobile helps solve the known problem of memory gaps. It delivers more accuracy.
This is particularly relevant for diary-formats, but also for any out-of-home buying or consumption occasion.
Delivering better accuracy can also be challenging: the emerging picture can be radically different from one delivered by memory-reliant desktop/laptop.
The following case looks at how drinks are ordered in a pub, and shows how different key outputs can be.
Case Study: Molson Coors On-Trade Lager Drinking
TNS was asked by Molson Coors to take a quantitative look at drinking in pubs in the UK (the on-trade), which in their view was an area thin on robust insights.
Industry wisdom – and one shared by the UK Government – was that alcohol consumption is price-sensitive, and that raising prices leads to a fall in consumption. Molson Coors wished to validate this, understand better what the key reasons driving brand choice were, including price, as accurately as possible. Mobile seemed an ideal vehicle for research.
TNS used a split-design approach, comparing online laptop/desktop data collection to mobile. They recruited 147 lager drinkers in the UK; and asked them the same questions regarding their drinking habits at a recruitment stage, (using a laptop), then complete the same questions whilst in the pub on their mobile.
The key findings showed clearly that using mobile gives a different response:
- 3.8 influence sources were mentioned in the recruitment survey – in mobile there were only 1.4
- A sharp swap of priorities was noticeable: price and special offers were stated as the most important influences in the survey completed on a laptop/desktop, whereas the in-the-moment responses via mobile showed “brand” to be the most important factor.
- Special offer hardly figured as an influencer in the mobile survey.
The two versions challenge received opinion.
Whilst the “truth” isn’t necessarily accessible via a single-mode research approach – no triangulation was undertaken or at least made public – price and promotions appear to be less important in marketing in the on-trade as a traditional approach suggested, whereas the role of the brand is up-weighted.
This has huge implications for the Molson Coors marketing mix – suggesting that brand-building activities for online preference are key, the marketing mix needs to respect this. Targeted TV advertising (one example) should be given serious consideration.
The conclusion from this case: mobile is immensely valuable in getting marketing a step closer to out of home experiences as they are felt and recorded at the time.
This greater accuracy can lead directly to:
- more efficient marketing-mix
- a high likelihood of an improved ROI.
Mobile – What Types of Research?
Sam sees two main areas where doing mobile research has clear advantages
- Path-to-purchase journeys
- Touchpoint analyses
i) Path to purchase journeys
TNS’ global study on mobile usage – Mobile Life – contains data showing that at present, mobile is used relatively more extensively for in-store intelligence gathering than purchasing.
There are clear differences by level of market and mobile maturity, with retail mobile usage increasing with rising smartphone penetration levels.
In Europe, 38% of mobile users have used their device at some point to research an in-store purchase whilst only 16% have actually used their mobile to buy something. Younger respondents are more likely to browse on their mobile in-store. The use varies strongly by category – usage levels on an individual trip basis, as Sam points out in his RW Connect blog, are often very low. TNS refers to “channel not fulfilling potential”.
Overall, the data suggests that some browsing has shifted from in-home (desktop) to in-store (mobile).
Using mobile to understand better how consumers react to in-store promotions or shelf-displays is therefore of value; a need which is likely to increase.
Mobile research should be executed in categories where the extent of in-store mobile browsing merits attention – books, DVDs/games and computers top the list, albeit still at single digit penetration levels.
A further consideration for mobile path-to-purchase studies is length of purchase cycle.
Engagement levels and diary entry enthusiasm can be held high for a relatively short time period, typically up to 4 days. After this, interest levels and reporting intensity drop.
For categories with particularly long decision making cycles, mobile tracking is not yet easy to execute successfully in TNS’ experience. As mobile panels become larger and more robust, this is likely to become easier.
Mobile diaries work well by using “near-the-moment” self-reporting: respondents are asked 4 – 5 questions at the end of each day, until they make a purchase.
ii) Touchpoint analyses
Mobile offers the following advantages:
- all touchpoints are recorded
- event sequencing can be monitored
Respondents’ behavior can be linked to a touchpoint – so allowing researchers to see precisely what prompts (advertising) or interactions (advice, recommendations, expert opinion) lead to an event, a purchase act.
The journey can be mapped over time.
Mobile allows a better understanding of the context in which the decision was made, as the recording is much nearer to the time it happened, and less reliant on erroneous recall.
“Touchpoint correlation” can also be identified: patterns can be identified of where the use of two or more touchpoints occur in clusters frequently. This allows researchers to probe the causality, engage in the diagnostics of “can you tell us more about…..” as an improvement on the direct and often blunt technique of “why did you….?”
Mobile Surveys – Good Practice Tips
TNS talks of “good” rather than “best practice”, because:
- the medium is still relatively new
- insufficient R&D work has been done across audiences, geographies, categories
It is premature to issue a clear and comprehensive set of survey design guidelines.
Their overall comment on the state of mobile research is critical, that there is insufficient recognition that simply transferring an online survey onto mobile whole-scale isn’t the answer. To quote Sam: “too many mobile surveys look like a regular survey.“
Their current guidance on how to shape good respondent mobile survey experience is as follows, based on pilot studies completed between 2011 – 2013.
- Short Interview length: maximum 10 minutes, but ideally much shorter, at 3 minutes.
This conflicts with all the knowledge needs a given project may have. It would require clients breaking up a large single surveys into multiple smaller ones. This means Agencies need to show Clients options that deliver on cost and timing – doing multiple shorter surveys instead of one longer ones.
This is an area that as yet is not standard practice in the development of proposals, and represents an opportunity for the industry. Education is needed.
- Eliminate repetition
Questionnaires often – in TNS’ experience – cover the same areas twice, possibly more often. The drive to brevity requires any potentially overlapping questions to be merged.
For continuous surveys, this involves showing Client stakeholders relevant analysis (factor analysis, predictive analytics) and proof of redundancies, levels of inter-question correlation levels.
- Filter for Relevance
Respondents should only evaluate brands that are in their consideration set. They don’t need to see a whole battery of say 8-10 brands or a list of 30 product attributes. This makes survey tasks – of image evaluation for example – much shorter, and manageable on a significantly smaller screen.
Filter questions are key: only ask questions on the attributes that respondents care about, for example, or the brands they have in their relevant set.
Grids are much simplified by following this guideline, and responses correlate more strongly with actual behavior.
- Gamify to suit your audience
Making tasks more intuitive, and fun is a way of holding attention levels high – TNS sees gamification as an area of opportunity.This can be very simple gamification – removing a 1 – 5 touchpad option to a slider for example.
Challenges for Mobile Transition: Audience Biases
TNS currently see limitations in the size and scope of available mobile-enabled panels.
Where mobile panels are available, they see certain audience skews:
– more tech-savvy
– less older respondents
– less younger males
No doubt this is a moving target, with many panel providers working actively to increase the number of their participants who are opted-in for mobile research.
Given the current paucity of mobile-enabled panels, TNS currently adopts a partnering approach, linking up with companies outside of the traditional MR space that can deliver potentially interested MR audiences with the right approach and incentives. These partnerships are becoming increasingly fruitful in Emerging Markets like India.
Mobile & Passive Monitoring?
Kantar has been collecting “clickstream” data – respondents’ complete use of their mobile – for over three years now in North America. They record telephone calls (only duration, not the content of the calls) pics taken, sites visited, ads exposed to – recorded passively by an App they download. Cell tower GPS information reveals where respondents are.
A pilot is also testing an audio recognition app that can listen to and record sounds in the immediate vicinity – similar to the music-recognition app Shazam.
Such tracking allows a comparison between claimed and actual behavior – and is powerful in detecting potential contradictions – useful for insight generation.
As a method, passive mobile monitoring is something TNS is treating as an area of huge potential. There are still challenges in persuading respondents to join these panels over privacy concerns, even though no personally identifiable information is stored.
Whether sufficient consumers will consent to total tracking in future, thereby permitting some degree of representativeness, is an open question.
Unless barriers are overcome, passive monitoring may remain of niche relevance for research.
- With its ongoing global mobile research program amongst tens of thousands of mobile phone uses, TNS is well positioned be a global leader in understanding the habits, preferences and movements of the mobile consumer.
- Using mobile as a vehicle to help convince the industry (clients and Agencies) to shift to shorter surveys with higher predictive validity has multiple benefits. It also has the potential to revolutionize the way surveys are conducted, possibly signaling the demise of the long, tedious 25 + minute tracker.
- Mobile should improve touch-point and customer journey analyses. Improved efficiencies in media and marketing mix planning should result from mobile’s ability to register more touch-points, better understand how touch-points interrelate and then lead to an event such as purchase, and the context in which someone felt good or bad about a brand or a marketing message.