Who Are The 50 Most Innovative Companies in Market Research? The Q1-Q2 2017 GRIT Report is here!

The Q1-Q2 2017 GRIT Report is now available, including the rankings of the Most Innovative Suppliers and Clients.

I’m immensely pleased to announce that the 21st edition of the GreenBook Research Industry Trends Report, using data collected in Q1 & Q2 of 2017, is here!

This is the edition that features the “GRIT Top 50” rankings of research suppliers and buyers perceived to be most innovative, and it’s quite the list! In the infographic summary below you can see the Top 10 from each list, but you can access the full rankings here. Over the past seven years, the GRIT Top 50 ranking has become one of the key metrics many companies use to understand their position in the marketplace. At its core, it is a brand tracker using the attribute of “innovation” as the key metric, and we’re thrilled so many organizations in our industry use this metric as part of their strategic planning and marketing positioning.

In addition to the GRIT 50, we keep looking into issues surrounding automation, sample, and research budgets. The report also dives into the skills and resources needed for educating the researcher of the future in post-secondary programs as well as market positioning of the U.S. MMR programs.

Last but not least, we are unveiling the GRIT Benchmark, designed to give research organizations a set of comparison points against which they can measure their development and performance. Later this month, we’ll release an interactive online tool using these benchmarking data, so stay tuned for that!

This GRIT Report is based on the biggest sample ever, with 2,637 completed interviews globally. Although the sample sizes are still not robust enough outside of North America and Europe, overall – and in relative balance with market sizes – we are at a point where GRIT may already be more on the “representative” side than just “largely directional”. However, we won’t claim representability yet so treat the findings as VERY “largely directional”. In keeping with the spirit of transparency and collaboration, we are making all GRIT data available to everyone for further exploration via our partners OfficeReports.

As a research effort, GRIT allows us to “walk the talk” and, as always, we do better in some ways than in others. We feel the same pain that many clients and suppliers do in trying to migrate to new modes or incorporate best practices in mobile-friendly designs. At 15 minutes GRIT is longer than we’d like, but compared to many comparable B-to-B studies we’re pretty proud that we keep it within a reasonable range while also covering all the topics that we do. However, as always we hear lots of feedback from the industry on design, question areas, etc… and we appreciate the suggestions. For those who participate in the survey, we appreciate your contribution as we continue to fine tune the study.

And our objective? A report that goes deeper to explore the key drivers of our industry, offers better guidance, and helps chart the future as a strategic planning tool. We hope the GRIT Report is a touchstone for you and your team to understand what is happening, what it all means, and what you should do about it.

GRIT is a community effort and our authors, commentary providers, sample partners, advertisers, and most especially research partners make this all possible. Special thanks go out to the organizations who helped with data collection and analysis, including Ascribe, AYTM, Bakamo Social, G3 Translate, Gen2 Advisors, Lightspeed, mTAB, Multivariate Solutions, NewMR, OdinText, OfficeReports, Research Now, Researchscape International & Stakeholder Advisory Services.

As always, I think you will find the report informative, provocative, and useful. Enjoy!



Key Takeaways from IIeX

Missed IIeX North America 2017? Get caught up with Susan Petoyan's key takeaways.

By Susan Petoyan, CEO, Imagine If Research & Insights

Were you able to make it out to Atlanta for IIeX 2017? If you did, I’m sorry I didn’t have a chance to catch up with you. Like you, I was busy listening, absorbing, and processing everything that was happening all day and night throughout the entire conference. Here, I want to share some general observations, things that ‘hit’ me or “hit home” with me.  Will you share yours too?

1) The growth of this conference over the years is incredible and highly encouraging because you see companies embracing innovative technologies more and more. This gradual change elevates not just those companies that attend the conference but the entire industry as a whole. And this is one of the most important reasons why I support this conference.

2) Together with the growth, apparently come the suits. I remember in 2013 when I attended IIeX NA for the first time, a visual scan of the 300+ attendees would reveal a more relaxed, business- or casual choice of apparel. I sometimes refer this as “silicon valley or start-up chic.”

This time, it hit me that there were many, many more people in formal business suits (mostly men). The sales people have descended on the conference, was the conclusion.  

And this is a good thing – because it shows that the industry believes in the opportunity in innovation –  as long as a genuine spirit of that new, fresh, and somewhat non-conformist thinking does not get lost among the suits. On the last day, I hesitated but put on my ripped jeans. Good thing, most of the suits had already left.

3) This year I could also tell that attendees from the corporate side may have been “hiding” – overtly. While this happens in most conferences, it baffles me to see junior or mid-level corporate researchers conceal their name cards, or avoid eye contact, or just lay low altogether. As a former client, I do appreciate the deluge of sales pitches, offers to connect, or business card requests…

In one of the speaker sessions, I was standing next to a young man, by the door, whose sole job was to hand out business cards as people were exiting. He said, “Can I force my card on you?” As an ethnographic researcher, I marveled at how few people said no. But I digress.

For client-side attendees, aren’t there more effective ways to address the sales deluge than to hide? We are here because this is an innovation-focused conference. Innovation will only come into an organization if there are champions – like you – open and willing to embrace innovation, despite the work and effort it may entail.

4) Despite the sheer amount of new, directly or indirectly MR-related tech, platforms, apps, etc. there is still a dominating question in the air about what does this all mean? I love sticking around for the last day of the conference where people like Lenny Murphy, Simon Chadwick, etc. tackle the biggest themes – after the “shiny object” overload. And like in the past, this time too, some major issues were raised – including the importance of actionability – with which I couldn’t agree more. Yes, as a client or a provider, you now have a few more ways to get to the answers. Faster, cheaper, automated, AI-driven, etc. And this should be absolutely celebrated!

But what are you going to do once you have the answers? I call this the last “hundred feet” of research design, harkening back to my sales and retail marketing colleagues from a former life. 

What problems are these technologies solving and how are you going to ensure these new tools indeed drive real behavior change in your organization or client?

I wish there is more discussion of this topic after the conference, as we all go back to our offices, and at the conference next year. Actionability, the ability of a data-based project or initiative to drive change in a large, complex and often unwieldy organization – is the holy grail (methinks) for all of us in the industry, no matter which side you’re on, or what new shiny tool you might be exploring.

5) Can I confess something to you? I suffered from FOMO at the conference. There was so much great content across the four tracks, exhibitors to explore, and people to talk to, that I seriously struggled in deciding what to cover and what to skip. All this on top of the many great friends, former colleagues, familiar faces and new people that I wanted to meet.  

Still one of the key things that I loved seeing was the robust discussion around the importance of story. Despite all the new methods, tech, trends, etc., seeing more and more people focused on the (re-emerging) importance of great storytelling with research and data is reassuring. Heck, even McKinsey talked about story! The reversal of the “80-20 split” from “time spent on gathering/analyzing data” to “crafting/telling a story” is perhaps the biggest message that we all should embrace.

Whether it’s an AI-driven emotion reading platform, a dashboard specifically designed for the perennial CMO, or tech that embeds the presenter inside the deck she is presenting, the power of a good story transcends it all.  Good stories come from good design.  One reminder I brought home to my team was always to ask:

“What is the story?”

Big thanks to the organizing team for this an excellent conference!  And if you’re reading this, I look forward to seeing you at IIeX 2018!



Jeffrey Henning’s #MRX Top 10: Amazing Amazon Outlook, plus Conference Previews

Of the 3,244 unique links shared on the Twitter #MRX hashtag over the past two weeks, here are 10 of the most retweeted...

By Jeffrey Henning

Of the 3,244 unique links shared on the Twitter #MRX hashtag over the past two weeks, here are 10 of the most retweeted…

  1. Millennial Myths and Realities – Ipsos MORI offers a nuanced commentary on Millennials, noting “We are generational researchers and believe in its power to predict future directions, but that doesn’t mean we want to explain everything as generational. In fact, the opposite is true: we see our job as separating the different types of effect and not to claim generational impact when the evidence is weak.”
  2. And the Winner Is… The Exit Poll – In the wake of the UK General Election, Jane Bainbridge of Research Live recaps the performance of polls.
  3. ESOMAR Congress 2017 – Read all about the program for this year’s Annual Congress in Amsterdam in September.
  4. Eight Reasons to Attend the June 27-28 Sentiment Analysis Symposium – Can’t get to Amsterdam? Seth Grimes invites you to his Sentiment Analysis Symposium in New York next week, which will cover natural language processing, speech analytics, facial coding, and emoji for applications from CX and MRX to finance.
  5. Ready for the Digital Revolution? 3 Tips from the Web Tomorrow Event – Jilke Ramon of InSites Consulting recaps Web Tomorrow, noting the important of challenging your status quo and finding the right technology fit while not ignoring the human touch.
  6. Does It Sell? Content Marketing Tips – Isabel Gautschi and Sean Campbell of Cascade Insights have eight questions you should ask to gauge the effectiveness of your content marketing, starting with, “What are buyers searching for on Google?”
  7. Who are IIEX’s Biggest Influencers in 2017? – Maura Woodman of Affinio looks at top influences and niche influencers at IIEX.
  8. Amazon Now Ranked Fourth in BrandZ Top 100 2017 – A week before Amazon announced its acquisition of Whole Foods, Nigel Hollis of Kantar Millward Brown wrote, “For me, … it is the return to bricks experimentation that is most interesting. Amazon’s foray into physical bookstores has been going since 2015 and is positioned as simply a better way to discover books, but for a more radical approach we must look at its grocery business.”
  9. What is Artificial Intelligence? – Andrew Jeavons of Mass Cognition provides historical context about AI with some caveats for market researchers: “Any AI must have a goal, an outcome that it can ‘navigate’ to and so be termed intelligent”; “the target state could be market share, or ROI or any number of KPIs.”
  10. Silos: Good for Grain, Bad for Market Research – Keri Vermaak of Infotools provides three tips to overcome silos: design for synthesis from the outset, use a single platform, and provide broad access to the data.

Note: This list is ordered by the relative measure of each link’s influence in the first week it debuted in the weekly Top 5. A link’s influence is a tally of the influence of each Twitter user who shared the link and tagged it #MRX, ignoring retweets from closely related accounts. The following links are excluded: links promoting RTs for prizes, links promoting events in the next week, pages not in English, and links outside of the research industry (sorry, Bollywood).


Connecting with Consumers in the Modern Age

In response to "Cheap, Fast & Easy" by Michalis Michael, this article continues the discussion of the changing research landscape through democratization of research, affordability, and self-service.


By Zach Simmons

Michalis Michael, of DigitalMR, recently wrote a piece, “Cheap, Fast & Easy,” that made me shout a resounding “YES!” from my desk. In the article, Michael talks about a couple of key points that I’d like to dissect: democratization of research, affordability, and self-service (online). These points fit into a larger shift that’s happening in market research; by improving and speeding up brand’s ability to have real-time conversations, we’re able to focus on consumer engagement and connection.

Democratization of Research

For far too long, consumer research has been inaccessible to smaller brands and startups. We all know how critical research is to a brand’s potential for success, so the barriers to entry for smaller companies are particularly unfair. By allowing research to continue being so inaccessible, we encourage brands to forego research and in favor of guesswork. This should be alarming for us as an industry.

However, democratization of research doesn’t have to stop there. Increasingly, we’re seeing many of the world’s leading brands shifting toward a democratization of research within their organizations as well. They’re taking the formality out of research and encouraging their teams to speak to consumers directly. This is helping teams to build empathy and humanize consumers in order to build and market the best, most relevant products and campaigns. Different organizations call this sort of program by different names; we call them Consumer Connection programs.


Of course, one of the largest barriers to entry for democratization is affordability. Smaller brands and (seemingly) less integral members of the team won’t be included in the research process if it’s too expensive. And anyone who has ever worked on a research project knows just how expensive research can be! Fortunately, we can leverage modern technologies to help decrease some of these costs. By automating parts of the research process, we’re able to significantly cut unnecessary costs.


Automating the process also helps to make research solutions self-service and on-demand. When teams are empowering teams to connect directly with consumers, they are granted an opportunity to ask consumers questions according to their own timelines and needs. Because Consumer Connects are an ongoing initiative, so it’s important that brands can access consumers whenever necessary. The tools that they use for this, therefore, have to be self-service and on-demand. Naturally, that means that these tools have to be online.  

Michael cites SurveyMonkey as a success story of self-service market research. He’s absolutely right. SurveyMonkey has taken the hassle out of creating surveys, making it easy for anyone across a team to create and share a survey in just a couple of minutes. Quant has been doing this for nearly a decade, so why has it taken qual so long to catch up? Qualitative research should also be self-service, on demand, and simple. Of course, traditionally qualitative research is more challenging to pull off. However, with the advancements of modern technologies, this no longer has to be the case.

By automating the process and leveraging ubiquitous technologies like webcams, web browsers, WebRTC, VoIP, NLPs (the list goes on and on), the qualitative process can be dramatically simplified. What’s more is it can be done not only asynchronously (with solutions like text-based communities), but synchronously (with video conferencing), meaning that brands are still able to connect face-to-face with consumers, but online, instead of in-person. Not only is this more convenient for both parties, but it also means that brands are able to speak with consumers contextually, within their own home and work environments, opening the door to a whole new set of possible insights.

A Changing Research Landscape

Consumer research is in a period of transition. It’s no longer as easy to talk about the qual/quant binary, because, increasingly, the landscape of research is changing. Today, qualitative research isn’t just focus groups and IDIs; brands are requesting closer, more regular contact with consumers. They want to understand them not just as respondents, but as consumers and as people. It’s not about conducting rigorous interviews or uncovering the perfect insight, but rather, about allowing teams and marketers to simply have a conversation with consumers. In trying to gain a holistic understanding of consumers, it’s important that brands build empathy and understand consumers within their contexts and cultures.

Who Cares About Evidence?

Why bother with evidence? Because it improves the odds that what we believe is actually true. But not always.

By Kevin Gray


A close contact of mine who studies the sociology of science recently commented that, when the data do not support the researcher’s hypothesis, all too often it is the data that are rejected. In the political realm, logic and evidence are routinely subordinated to belief and ideology or, in the more elegant words of Mary Wollstonecraft: “But what a weak barrier is truth when it stands in the way of an hypothesis!”

Science, by the usual definitions, does not appear to come naturally to human beings and most of us struggle with it in school. While we’ve put men on the moon, it took us tens of thousands of years to accomplish this. Moreover, we want the world to be like…the way we want it to be. Damn the evidence! This is understandable, especially when science is shrouded in mumbo-jumbo and math, which most of us also hate.

And, naturally, egos, reputations, money, political expediency and a host of other flesh-and-blood variables can conspire to defang evidence.

Moreover, all evidence is not equal. Evidence can differ in quantity and quality. Recommending that a client make a decision potentially involving millions of dollars should normally require more than one number or the opinion of a moderator based on a couple of focus groups. So how can clients tell how much, or what kind of evidence, is enough? As far as I know, there is no simple answer.

Let’s go back to the drawing board and start from the beginning. Some results can be calculated precisely or are determined by rules. Others can be estimated probabilistically with statistics and machine learning tools. However, decision-makers are often confronted with situations in which they must rely on their gut.

For example, there are situations in which we know with certainty, or near certainty, that something will or will not happen. We peek outside our window and see clear blue skies all around and we know it won’t rain in the next five minutes. There are immutable physical laws governing this and we know we can run out to the mailbox without getting wet. This is the first type of decision.

There are also circumstances where we don’t know with certainty what will happen but can estimate the outcome pretty accurately. Say we’re at the stadium and the baseball team we’re rooting for is up 5-0 in the 9th inning. We can look up historical data on our tablet and, if we’re clever and quick enough, estimate the probability our team will win the game. We won’t be right all the time but we can beat a coin toss most of the time and know whether or not we can head to our car. This is the second type of decision.

Finally, there is the third type in which we have no pertinent data and there are no physical laws driving what will or will not happen. For instance, say a New Product Development person has a rough idea for a radically new kind of product, and we must decide whether to pursue it or not. In this case we are ignorant, in the parlance of decision theorists, because we have no applicable data or benchmarks at this juncture. It’s in these sorts of situations where using our gut instinct – our unconscious intelligence in the words of psychologist Gerd Gigerenzer – is our only real option. We may decide to commission marketing research or we may shelve the idea.

The first type of decision is very straightforward, and decisions that are essentially clerical in nature are increasingly being automated. If the decision rules are clear and unambiguous, humans aren’t required except perhaps as auditors and to ensure bugs haven’t crept into the software. For the second type of decision, where things are not as clear cut but probable outcomes can be estimated, analytics of various kinds, including Artificial Intelligence (AI), can be very helpful. Over time, they may reduce the need for human decision makers but this is hotly debated.

Let’s now turn to the third decision type. This last kind of decision is what we’re naturally best equipped to deal with by virtue of our genetics. There are no definitive rules and we lack the data to estimate probable outcomes with satisfactory accuracy. Here, gut instinct – the product of millions of years of evolution – cannot be replaced. AI cannot find and analyze data that do not exist. This is the kind of decision we most often encounter in real life, the world outside of blogs, sales pitches, academic papers and textbooks.

Different sorts of evidence come into play in these very dissimilar situations. In the first instance, we only need data that are required by the decision rule. The second is less clear. What we need depends on our model, which we have previously judged as adequate for our purposes, given the data we have or can obtain. However, there normally are many decisions required to develop that model and typically competing models we could have used instead. Furthermore, what is “adequate?” Decisions such as these cannot be automated and, as statisticians know, a considerable amount of judgement is required in data analysis.

This leads us back to the third type of decision process. We cannot calculate or estimate the outcome but somehow sense what is the appropriate course of action, which may be not to make a decision. Veteran marketing researchers know that major corporate decisions often fall into this category (though there may be sleight of hand performed with numbers and analytics). It is also the natural habitat of politicians. These gut-feel decisions aren’t just about which pizza toppings to order or what should we watch on TV tonight. Although I’m a marketing science person, I concede that there are situations in which analytics just get in the way. There is no point in trying to quantify illusions. We have to make a decision and all we really have is our personal or collective gut to go by. Analysis paralysis wastes time and time, after all, is money.

The terms deterministic, probabilistic and intuitive can be used to describe these three fundamental kinds of decisions. As you’ll have already surmised, many decisions are multifaceted and combine two or more of these decision processes. Though not an attorney myself, it is my impression that legal decisions mostly are of the first and third types. There are also circumstances in which the outcome itself is a matter of dispute. Is this what we really want? Would that outcome be right? Opinions may clash as to what the best decision is and ethical considerations might weigh heavily. There may be no right or wrong answer, in other words, and AI (or consultants like me) cannot come to the rescue.

The first and third types are the ones we’re generally most comfortable with. Engineers trust their equations and salespeople trust their gut. The second kind of decision process is hazy and a no man’s land for many of us. It is the natural habitat of statisticians, who must learn to cope with ambiguity and turn it to their advantage in order to thrive. Statistical science, after all, is a form of legalized gambling.

Hence, evidence matters some of the time but not all of the time. Moreover, humans frequently misconstrue conjecture as evidence. We also readily reject evidence that contradicts our opinions, and cherry-pick data and analytics to support decisions we’ve already made. More data and more sophisticated analytics will not change the world overnight.

This brief article has been an enormous simplification of a complicated topic that straddles philosophy, statistics and countless other disciplines. I haven’t even quoted Donald Rumsfeld, but hope you’ve found it fun and useful!

Behavioral Economics: Measuring Social Desirability

In this piece we are going to take a quick look at a method to measure a certain trait, Social Desirability, without the ‘self-selection’ error.

By Michael Lieberman

A common finding in almost every Behavioral Economics experiment that respondents believe that they are ‘above average’. Self-description is rarely objective. In this piece we are going to take a quick look at a method to measure a certain trait, Social Desirability, without the ‘self-selection’ error.

When we ask people (employees, customers, or any other kind of respondent) for their opinions, one of the key points in ensuring reliable answers is an objective question. Having an objective question includes the words within the question itself, but also the environment in which it is asked, and who is doing the asking. The whole point is to make sure that when a question is asked, there is not an answer that seems “better” than another.

The reason that this is so important is that people have the tendency to answer questions in way that allows for a more positive self-reflection. . This tendency exists between people, but more importantly it varies between the situations that people are in when asked the question. This is the reason researchers insist on how and where questions are asked.

This tendency, referred to as Social Desirability, does not mean respondents are necessarily going to be untruthful. It does mean that you may only be hearing about what their overly positive self-reflection more than an assessment of how your respondents actually behave.

This is not necessarily new information when you consider that most respondents will tell you brand name is not important, and that each purchase decision is a careful calculation of value.

There are a great number of tools that are used to get around this issue but I thought it would be fun to share how this tendency is actually measured.

Social Desirability Tendency

If you are interested in seeing where you fall, take the 20 question quiz below BEFORE reading the scoring key. This is a test that is used by Psychologists when studying behavior and has been in use for a little over twenty years.

For what it’s worth, I’m not sure I would recommend giving this quiz to customers as a way to assess their answers.

Social Desirability Quiz

Indicate the extent to which you agree with each of the following statements. Use a 7-point scale to indicate your response, with 1 = “Not True” and 7 = “Very True”.

1) I sometimes tell lies if I have to

2) I never cover up my mistakes

3) There have been occasions when I have taken advantage of someone

4) I never swear

5) I sometimes try to get even rather than forgive and forget

6) I always obey laws, even if I’m unlikely to get caught

7) I have said something bad about a friend behind his or her back

8) When I hear people talking privately, I avoid listening

9) I have received too much change from a salesperson without telling him or her

10) I always declare everything at customs

11) When I was young, I sometimes stole things

12) I have never dropped litter on the street

13) I sometimes drive faster than the speed limit

14) I never read sexy books or magazines

15) I have done things that I don’t tell other people about

16) I never take things that don’t belong to me

17) I have taken sick-leave from work or school even though I wasn’t really sick

18) I have never damaged a library book or store merchandise without reporting it

19) I have some pretty awful habits

20) I don’t gossip about other people’s business

Scoring key: Give yourself one point for each 1 or 2 response to odd-numbered items and one point for each 6 or 7 response to even-numbered items.

The test developers found a mean score or 4.9 and a standard deviation of 3.2 for female college students and a mean score of 4.3 and a standard deviation of 3.1 for male college students. Those who score higher are more likely to present themselves in a more positive light.

Source: Lockard, J.S. and D.L. Paulhus (Ed.). Self-Deception: An Adaptive Mechanism. Prentice-Hall, 1988.


It is important to understand that people who are more prone to present themselves more favorably than others are typically unaware that they are doing so. If we look at the different items on this test, few of us can say that we have absolutely have never tried to cover up a mistake. However, somebody who rates high on Social Desirability may be more inclined to exaggerate the truth and indicate that this point is true for him or her.

Now, if we did actually know which of our respondents rated high for this tendency we may be able to adjust our interpretation of the results; but honestly it would likely be a challenge to get this information in most, if not all, of our studies. What is more useful though is to understand that this tendency exists and that we should know that it is a matter of to what extent respondents are prone to it, not if.

So when we ask a question such as, “To what extent do you make a purchase on impulse?”, a low score could reflect a desire for the respondent to present themselves in a rational light just as much as it could reflect behavior. As a result it is more difficult to interpret the results to this question than it would be for a question such as “How often do you make purchases that are only on your shopping list?”

In marketing research, perceptual validity of questions has been the subject of many discussions. There are certain methodologies to correct for Social Desirability (e.g. division by the standard deviation). However, the issue of Social Desirability, and data validity as a whole, remains on the front burner of branding image and product research and likely will be as long as research relies on Q&A.

Gradient Boosting: The Coolest Kid on the Machine Learning Block

Gradient boosting is a technique attracting attention for its prediction speed and accuracy, especially with large and complex data.

By Jake Hoare

Gradient boosting is a technique attracting attention for its prediction speed and accuracy, especially with large and complex data. As evidenced in the chart below showing the rapid growth of Google searches for xgboost (the best gradient boosting R package). From data science competitions to machine learning solutions for business, gradient boosting has produced best-in-class results. In this blog post I describe what it is and how to use it in Displayr.


Gradient Boosting

Gradient boosting is a type of boosting. Boosting is a method for combining a series of simple individual models to create a more powerful model. Start by fitting an initial model (either a tree or linear regression) to the data. Then build a second model that focuses on accurately predicting the cases where the first model performs poorly. The combination of these two models is expected to be better than either model alone. Repeat the process many times. Each successive model attempts to correct for the shortcomings of the combined ensemble of all previous models.

The best possible next model, when combined with previous models, minimizes the overall prediction error. The key idea behind gradient boosting is to set the target outcomes for this next model in order to minimize the error. The target outcome for each case in the data set depends on how much a change in that case’s prediction impacts the overall prediction error.

If, for case xi, a small change in the prediction causes a large drop in error, then the next target outcome is a high value. Predictions from the new model that are close to its targets will reduce the error.

If, for case xj, a small change in the prediction causes no change in error, then the next target outcome is zero because changing this prediction does not decrease the error.

The name gradient boosting arises because of setting target outcomes based on the gradient of the error with respect to the prediction of each case. Each new model takes a step in the direction that minimizes prediction error, in the space of possible predictions for each training case.

Time for a drink

Maybe that explanation was a bit heavy, so let’s relax. To be more specific, how about a glass of wine? The data set I will be using to illustrate Gradient Boosting describes nearly 5000 Portuguese white wines (described here). You can replicate the analysis used in this post in Displayr. Displayr uses the R xgboost package and you can do your complete analysis, from data import to charting.

As a preparatory step, I split the data into a training and testing set. This is done with Insert > Utilities > Filtering > Create Train-Test Split. This allows us to fit the model to the training set and assess the performance of the model within the testing set.

Next, I add the gradient booster with Insert > More > Machine Learning > Gradient Boosting, and select quality as the Outcome variable. Quality is the median of at least 3 evaluations made by wine experts, on a scale from 1 to 10.

I use all 11 other variables for the Predictors. These describe the chemical composition of the wine. I produce an Output of Importance with the gbtree Booster. Boosting normally uses an underlying tree model, although a linear regression as also available as an option. Finally, I set the filter to Training sample in Home > Filter (Data Selection), and the result is shown below. The importance scores tell us that the alcohol content is by far the the most important predictor of quality, followed by the volatile acidity.


Performance evaluation

Now let’s find the accuracy of this model on the test data. To do so:

  1. Select the output from the gradient boosting.
  2. Select Insert > More > Machine Learning > Diagnostic > Prediction-Accuracy Table.
  3. With the prediction accuracy table selected, choose the Testing sample filter in Home > Filter (Data Selection).

The prediction-accuracy table illustrates the overlap between the categories which are predicted by the model and the original category of each case within the testing sample. The overall accuracy, shown beneath the chart, is 64.74%. However, there is one more trick to enhance this. By checking the Grid search box on the original model inputs, the boosting algorithm varies its internal settings in an attempt to find the best settings for this data. This takes more time to run, but accuracy increases to 65.69% as shown below.


How good is this model? For comparison I’ll repeat the analysis with CART instead of gradient boosting. (In Displayr, this is done using Insert > More > Machine Learning > Classification and Regression Trees (CART)). The accuracy is much lower, at around 50%. Since alcohol is used for the first split of the tree, the resulting Sankey diagram below agrees with the boosting on the most important variable. So next time you buy the most alcoholic bottle of wine, you can tell yourself it’s really quality you are buying!

The examples in this post use Displayr as a front-end to running the R code. If you go into our example document, you can see the outputs for yourself. The code that has been used to generate each of the outputs is accessible by selecting the output and clicking Properties > R CODE on the right hand side of the screen. The heavy-lifting is done with the xgboost package, via our own package flipMultivariates (available on GitHub), and the prediction-accuracy tables are found in our own flipRegression(also available on GitHub).

You can replicate this analysis for yourself in Displayr.

Originally posted here.

Social Data Analysis and Insights Requires Labeling Laws

Social data is rapidly being adopted by more and more researchers given the robust, raw and real-time nature of it.

By Rob Key

With the advent of machine learning and AI rapidly transforming market research, brands and researchers are faced with a dilemma:  how do I trust the data that is bring provided to me via a black box algorithm?

Indeed, According to the Forbes Insights and KPMG “2016 Global CEO Outlook, 84% of CEOs are concerned about the quality of the data they’re basing their decisions on.    And with good reason, poor data quality is pervasive within the insights industry, especially in the area of unstructured social and voice of customer data where accuracy challenges are significant. But poor data quality necessarily leads to poor insights, and poor business decisions make in these insights.

There is some good reason for this: human language is complex.  Sarcasm, slang and implicit meanings abound.  Different techniques are often used to analyze the data (ranging from more traditional “rules based” techniques, machine learning/AI and Deep Learning).   This has led to a great variance in data quality and vast inconsistency in the way the data is coded for sentiment, emotion, relevance and more.  Studies have shown that some techniques pervasive in the industry achieve only 60 percent precision and about 15-30 percent relevancy through primitive Boolean queries.  Other more contemporary approaches are yielding different results.  To complicate matters further, not all analysis is done at the same level.  Some solutions provide record level analysis, while others provide highly granular “facet level.”  Non standard performance testing techniques and marketing hype simply add to the confusion.

In the meantime, social data is rapidly being adopted by more and more researchers given the robust, raw and real time nature of it.   This vast, unstructured, unprompted discussion can be a gold mine for researchers who know how to harness it.  It is being increasingly utilized in areas ranging from brand tracking to post segmentation, market mixed modeling and more where researchers are demanded to provide “better faster, cheaper” insights. In too many cases though, users of this data are forced to “eyeball” results or simply accept the data quality being provided “as is.”   The effort to QA data with often millions of records is simply not realistic.

Clearly the status quo is untenable.   For the industry to grow to the next level it must enable insights professionals and organizations to more confidently leverage and mainstream this data.   This requires opening up the black box and openly providing “labels” for classifier performance.

This of course should seem reasonable to anyone involved in consumer insights.  Yet, it would represent a bit of a revolution in the world of social and voice-of-customer analysis.  While traditional survey approaches have long provided margin of error and confidence intervals, there has not been a corresponding accuracy for social and voice data provided by text analytics solutions.   Yet, those performance scores exist.

The F1 score (and its cousins F2 and F 0.5) provide performance scores on two measures:  precision and recall.   Precision is essentially how well the algorithm matches the gold standard of “accuracy,” while recall measures the numbers of “signals” gleaned from the data. Historically, when precision goes up, recall goes down; and vice-versa   One can be more accurate if evaluating a smaller number of signals. The more signals analyzed, the more chance for error so precision often declines.  Showing a precision number without a recall number can be misleading and has led to much misinformation and misunderstanding in the marketplace.  Of course one can get to 90% precision if only analyzing the most obvious solutions and leaving the harder-to-analyze as mixed or neutral. The combination of precision and recall results in the F1 score, which you can read more about here.

While this sounds obvious and simple, there are a few complexities here that will require the industry to collaborate and achieve consensus. One area is that we first need to first agree to how to measure precision in social data. Human beings themselves when asked to evaluate a social conversation for expressed sentiment and emotion often differ. In fact, in our experience and depending on the complexity of the category, we see humans only agreeing on average 65-80 percent of the time. Analyst bias is rampant.  Context matters.  At Converseon we advocate a precision score measured against three independent humans who view the same record and agree.  This also requires approaches to ensure inner-coder reliability where there is a different of opinion.  How well that algorithm matches that is, in our view, the human gold standard for language analysis and should be established as a benchmark for measurement.

Labeling of classifiers with this scoring would provide confidence to researchers, help demystify performance claims from different vendors, expand the use of this valuable information, and help ensure this data is used responsibility and effectively.  In fact, inputting into models or reporting insights based on this data without these scores can be quite misleading and dangerous.

Expressions of opinion through social platforms is powerful and transformative.  Whether in politics, culture or simple expressions of brand and product opinions, consumers are demanding to be heard.  And we, as an industry, have a responsibility to analyze and report on those opinions accurately.    On the flip side, brands also have every right to know that the information they’re receiving from solution providers in this space is also clear and reliable, and at what precision and recall level.

For social data to be more broadly leveraged, no longer should users of this data be required to blindly trust results, or have to try to evaluate the data quality themselves, before using for important insight and modeling work. In fact, broad adoption of this data into critical functional areas requires nothing less.

The Solar System of Social Media

If social media were a solar system, what would it look like?

By Michael Lieberman

If social media were a solar system, what would it look like? Would there be a sun, an earth, a moon?

In a survey respondents were asked which Social Media sites they regularly used. With this data we were are able to map a network, with the size of the spheres representing market share and the thickness of the lines representing the relationships between spheres.

The red spheres represent are our major players—the center of the social media universe. No surprise, Facebook is the Sun, YouTube is Jupiter, and Twitter is, say, Saturn.

The blue spheres are Social Media sites with the second highest proportion of users; the second tier. Green spheres the third ring. The small planets in black are kind of like Mars, out there somewhere, existing but small.

The ultimate utility of this map is that in one glance a client can grasp the social media solar system. Yes, MeetMe is small, but it a better connection to Instagram than to Google Plus. Great information. Ask.fm is an anonymous question and answer platform website used regularly by lots of young people in Ireland and around the world. Its strongest connection is to Facebook, which is the worldwide leader. Bebo describes itself as “a company that dreams up ideas for fun social apps;” Though not a central player, its strongest relationship is with Google Plus. This seems intuitive given that Google is the largest distributor of apps, and terrific information for the executives at Bebo.

These types of visuals can be employed not only for the solar systems of Social Media, but to brand space of any product, purchase path behavior for click throughs, or the shape of attitudes around client behavior. They can be used in place of correspondence maps or set to configure the political structure of the parliament of the United Kingdom, the United States Senate, or a local school board. Send us your data, we can map it. Then tell you the story.

Visualization of data networks, social media, industry structure, and even now enourmous transational datasets are now coming online to make sense of the data deluge and convey the results of analyses through emerging, open-source programs. This kind of analysis is not limited to Social Media, but also can be applied to other megadatasets, consumer sales data from any major corporation, major supermarket, Walmart or survey data. It is a great new tool that, together with our analytic skills, we can deploy to give our clients a full picture of their product solar system.

Network anlaysis expands marketing research industry core competencies such as segmentation, pricing, conjoint analysis, regression modeling, forecasting, data mining, project management, and overflow reporting. As the industry moves from the reporting to the consulting phase, this techniques offers a powerful, simple, and easy to explain summarization. As it is said, a picture paints a thousand words.

We expect the availability of tools such as Network Analysis to have a positive impact on brand research. As mentioned above, visualing your brand solar system is a fruitful area of study, as is research into approaches for jointly analyzing megadata and text-content data. That is, companies have learned to harness the power of thought leaders, experts, and influencers to promote their products. Brand solar system visualizations will play a central role in the forthcoming drama.

What is your brand solar system?

Making Your Data Hot: Heatmaps for the Display of Large Tables

A heatmap, which replaces the numbers with colors or shades proportional to the numbers in the cell, is a lot easier than a table for our brains to digest.

By Tim Bock

Sometimes tables are just too big to read. The table below shows the personality attributes that people associate with different iconic brands. A table too big to read easily and too big to show elegantly on a web page, in this case, leaves only the first page visible. A heatmap, which replaces the numbers with colors or shades proportional to the numbers in the cell, is a lot easier for our brains to digest.

While we were unable to display all the data in the table above, the heatmap below shows it all nicely.  By replacing the numbers with colors, all 42 columns and 15 rows fit in one compact view.

An additional benefit of the visualization is that it is an image. With a table, our brains need time to process each of the numbers and work out their implications. A heatmap, on the other hand, allows our brains to readily detect the differences in intensity. You can still view the numbers in these examples by hovering your mouse over the heatmap.

Making patterns in heatmaps easier to spot

While we can hunt out patterns in this heatmap, it is a painful process. Reducing the size of the visualization makes patterns easier to see. They are not, however, simplified. A common solution to this problem applies both here and to tables: compute the average of each row and column, and sort the table from highest to lowest (see below).

We can see from this that the personality traits that are near-synonyms for appeal are at the top, indicating they were most likely to be associated with these brands (all of which are successful). We can also see that the car brands appear mainly on the left. This highlights that years of personality-based car advertising has had some effect.

Using cluster analysis to improve the heatmap

While reordering of the rows and columns has improved the visualization, it has not highlighted patterns showing the relationship between the personality attributes and the brands, which is the main goal of the analysis. We can improve the heatmap further, by clustering rows and columns so as to group together similar rows and columns. The visualization below has dendrograms showing these groupings.  The visualization is now, at last, paying off in terms of allowing us to see interesting conclusions.

To name two:
• On the left, we can see a cluster of luxury brands, starting with Calvin Klein through to Porsche.
• The next set of brands contains more rugged outdoorsy brands, including jeans, shoes, and just one car brand: Toyota. There are, of course, many more insights that now leap from the visualization.

You can play with the R code used to create these examples in Displayr.


Thanks go to Michael Bostock, Joe Cheng, Tal Galili, and Justin Palmer, who did the heavy lifting in creating the wonderful d3heatmap package used in this blogpost, and to Michael Wang who tweaked it so it did precisely what I wanted.

Originally posted here

The Giant That Was Greenfield Online

How the Tentacles of the Former Sampling Behemoth Still Survive Today

By Matt Dusig

In every industry, there’s a pivotal, innovative, company that forges new paths to do something that’s never been done before. In 1998, PayPal created the consumer-friendly, money transfer solution that you know today. The early founders and group of leaders within PayPal have been referred to as the PayPal Mafia because of the influence they wield around Silicon Valley. After the sale of PayPal to eBay, this team went on to create many incredible businesses that you use today. Elon Musk, founder of Tesla and SpaceX, was a co-founder of PayPal. Reid Hoffman, also a co-founder, created LinkedIn.  A web designer and engineer from PayPal created YouTube and a few PayPal engineers created Yelp. Three of the PayPal founders created The Founders Fund, a venture capital firm that has invested in Airbnb, Lyft, Facebook and Spotify, to name a few.

In the Market Research industry, I think similar analysis can be applied to people who got their feet wet in market research and online sampling at Greenfield Online. If you started in Market Research after 2009, you’re probably not familiar with Greenfield, which was founded in the late 1990’s. It was the primary driver in the creation of panels online and made the movement towards online market research possible. Greenfield’s history is a windy road, having gone public in 2004, and then embarking on a buying spree gobbling up competing sampling firms around the world.

One of those acquired companies was my first sampling company, called goZing, that I co-founded in 1999 with my Innovate partner, Gregg Lavin. After goZing, Gregg and I went on to create uSamp/Instantly in 2008, sold to SSI in 2016.

In 2005, Greenfield also bought Ciao, a European sampling firm and comparison shopping service. Greenfield was sold in 2008 to Microsoft for $486 million, solely to own the comparison shopping service, and Microsoft quickly sold the sampling assets to Toluna in 2009.

As I assessed the value of this blog post, it became clear that ex-Greenfield’ers still run much of the sampling industry today.

Here’s where some of them are now: 

    • Gregg Lavin and Matt Dusig sold goZing (’99) to Greenfield, created uSamp (’08) and Innovate (’14) – all pioneering sampling firms of their time.
    • George Llorens, also a co-founder of Innovate, was a Regional Vice President at Greenfield, before becoming EVP of Global Sales at uSamp.
    • Toluna is the current owner of the Greenfield Online brand and assets.
    • Michael Anderson, was an early sales leader at Greenfield and went on to become SVP, Head of Sales, N. America for Toluna.
    • Mark Simon was a Client Development Director at Greenfield and is currently the Managing Director of North America for Toluna.
    • Sandy Casey, former VP of Greenfield is now SVP of Global Supply at Toluna.
    • Hugh Davis has been credited with creating the first email-based panels at Greenfield. Hugh co-founded sampling firm Critical Mix.
    • Keith Price, was EVP of sales at Greenfield and became President of Toluna N. America before also co-founding Critical Mix.
    • Jonathan Flatow, the COO of Greenfield Online, is now the Managing Partner for Reimagine Holdings Group (owner of Critical Mix), an investment firm focused on market research, growth-oriented, technology-enabled service companies.
    • Andy Ellis held numerous senior level positions at Greenfield Online, and is now the COO of Lucid, the pioneering programmatic sampling platform that just raised $60 million.
  • SSI
    • David Zotter was an early Director of Research & Development at Greenfield and is currently the CTO of SSI.
    • Frank Kelly was the SVP of Marketing and Strategy for Greenfield, and is currently the SVP of Global Marketing and Strategy for Lightspeed GMI.
  • CINT
    • Richard Thornton was a VP of Europe for Ciao/Greenfield and is now Deputy CEO of Cint.
    • David St. Pierre was the CTO of Greenfield and is currently the CTO of TrueSample.
    • Jennifer Weitz was the Vice President of Global Supply at Greenfield and is currently the Chief Revenue Officer for Imperium.
    • Michael McCrary was a Senior Vice President at Greenfield, before becoming MD of N. America for Cint and President at Federated Sample. He is now the founder/CEO of the sampling platform PureSpectrum.
    • Terence McCarron was SVP of N. America sales at Greenfield, went on to become the MD of N. America for Cint before founding his own company called OpinionRoute.
    • Doug Guion was VP of N. American Operations for Greenfield and is currently President of Acturus.
    • Beth Rounds was the SVP of Marketing at Greenfield is currently the CMO of Dapresy.
    • Rudy Nadilo, an early CEO of Greenfield, is now President of Dapresy North America.
    • Dana Stanley was the Senior Director of Client Development at Greenfield and is now the COO/CRO for the GreenBook.

There are hundreds of people who still work in online sampling today (too many to list) who gained experience in market research through their time at Greenfield. So, maybe there isn’t a Greenfield Mafia like the PayPal veterans, but the accomplishments in market research, from the people above, are still impressive.

Over the past 20 years, many of us have built our careers around surveying and sampling, and it’s amazing to see how today’s leaders are so connected to the past. My hope for the future of market research and sampling is that today’s leaders remember that change is the only constant in life and we all must all push forward, and continue to Innovate.

Did I miss someone you would add to this list? Please add them to the comments of this post.