Does (Panel) Size Matter?
Editor’s Note: USamp is one of several firms that are redefining the model for what a panel provider looks like in the new MR paradigm. Their incorporation of self-service survey tools, a mobile platform for data collection, and the merging of both traditional panel, social sample, and river sampling has set the stage for not just a new panel business model but also a whole new series of questions on what metrics should be important in judging panel partners. One of those is how do we factor in traffic metrics into a portal-based sampling model. It’s certainly something that is an increasingly relevant consideration and I suspect it will only increase in importance as new research models emerge and take hold.
Matt Dusig, CEO of USamp has been challenging the industry to look at these new ideas and I think it’s a conversation that we need to have industry wide. Not only do these questions apply to panels, but also to social media research, gamification, mobile, MROCs…heck, just about ANY newer research approach is impacted by these ideas. This isn’t meant to be a uSamp advocacy post; I think the questions Matt raises are important and pertinent and I’m happy that he asked me to guest post this on GreenBook Blog. I’ll be very curious to see what type of debate this generates!
By Matt Dusig, CEO of USamp
Most sample companies market themselves according to panel size, quality of respondent data, variety of the traffic sources and customer service excellence. We announce the size of our panels to show scale, and to impress clients. It’s how the industry has defined success over the past decade; however I want to offer up another metric for your consideration: traffic.
I think the new discussion shouldn’t only be about panel size, but about traffic and transparency. As sample companies move beyond the pure process of building panels, daily/monthly web traffic becomes a core metric. Traffic is a foreign term to most in the market research industry. Traffic = website visitors. What good is a website or any online business without visitors? But what good are visitors without the right technology to manage the entire process? From registration to rewards, from deep profiling to quality measures, from routing technology to creating a positive experience—all require excellent technology. And if we don’t honestly and transparently convey the entire process to our clients, then we’re doing our clients and ourselves a disservice.
So how do we measure traffic? Why is traffic important?
Let’s say you want to recruit people from Facebook to take a survey. You don’t really care about the fact that Facebook has a database of 750 million people. You care about how many people will be exposed to your ad during the days your survey is in field. It’s all about visitors and traffic.
When you want a sample company to find panelists for an upcoming survey it doesn’t matter how big the panel is; it matters how people will respond to survey invitations and how much traffic will visit a sample company’s sites during the days you’re in field.
I think the real hard-hitting questions to ask sample companies are as follows:
a. How much traffic do your websites see DAILY?
b. What % of traffic is US vs. International? c. Of the daily visitors, what % are new visitors vs. returning visitors?
2) Completion and Experience:
a. What % of your daily visitors who log into your site successfully complete a survey on that day?
b. What % of people who click from an email invitation successfully complete the survey they were invited to?
c. How long is the routing process for a panelist (the process of bouncing a person from question to question before they get routed into the appropriate survey).
3) Activity and Quality:
a. How do you define an active panelist?
b. What do they have to do to stay active?
c. If someone isn’t joining your panel, how do you monitor activity to maintain quality?
All this talk about traffic and transparency is not to underestimate the importance of size. But I want to recognize that there are more dimensions to a panel than size—and more important questions that clients should be asking their suppliers. Clients should be very concerned if sample vendors aren’t willing to share their traffic sources and all panel attributes. They have a right to know what they are purchasing. I invite fellow suppliers to join uSamp’s mission to make the market research industry more transparent. Who’s ready to reveal?