1. Greenbook 2
  2. Greenbook-Mobile-6.29.16-

A Bad Week for Neuroscience

Recent finding have created serious doubts and cast aspersions on neuroscience.

Neuroscience

By Dr. Stephen Needel, Advanced Simulations

For those of us in the business who are scientists, or have pretensions of being scientists, neuroscience is a ridiculously compelling concept. Science uses the term sui generis (because we can’t help but use Latin terms profusely and indiscriminately), which means “cannot be reduced to a lower concept”.  The ability to define the neurological processes, the lowest level of an individual’s response, to marketing concepts is a holy grail for us. If we can map the process to the response, we get away from asking people questions and all the uncertainty, all the biases, and all the controversy in our scientific endeavors.

So it is with both great sadness and a certain amount of smug self-satisfaction that I read two publications recently that raise serious doubts and cast aspersions on neuroscience. The first is an article in Proceedings of the National Academy of Sciences (no, I haven’t heard of it either) by Eklund, Nichols, and Knutsson called “Cluster failure: Why fMRI inferences for spatial extent have inflated false positive rates”. I give you fair warning – this article is as dense as they come, both from a neurological and a statistical perspective; it is not for the faint of heart. Fortunately, they summarize the issue and the results in a language we can all understand. The three most common statistical packages that analyze fMRI data have a positive bias in the range of 70%. That’s worth repeating – they found that 70% of the time these packages deliver a false positive and they call into question the results of some 40,000 fMRI studies. In practical terms, when someone tells you that this type of stimulus excites this area of the brain and that means it’s good or bad, they are likely wrong.

The other publication I read was the July 2016 issue of Quirk’s Marketing Research Review. I recognize that this is an advertiser-supported publication ads_needle– it’s free to subscribers and I usually find at least one interesting article an issue in there – sometimes more. There’s an ad (on the inside back page) from a major marketing research supplier who will go unnamed, promoting their neuroscience business. The headline says, “Think your ad is good? We can make it GREAT. “ They use EEG, Biometrics, Facial Coding, Eye-Tracking, and Self-Report to “get at the truth of how people respond to your ad, so you can run your campaign with confidence.”

No, not really. They can tell you if there is or isn’t neurological stimulation and probably can tell you where in the ad the stimulation occurs or doesn’t occur. That will tell you two things – it generates some stimulation or not and does that stimulation occur when you think it should. Neither of these will make the ad great, or even good for that matter – it’s a report card. They can tell you whether it is more or less stimulating than other ads in your category that have been tested. They can tell you whether people liked the ad or not via facial coding and by asking them. That won’t make the ad great either. Why not? Simple – we don’t understand the relationship between neurological stimulation and purchasing and we barely understand the relationship between ad-liking and purchasing. At the end of the day, the question is whether advertising drives increased purchasing, and we have yet to establish the necessary linkages to define this neurologically. Research doesn’t make anything great – it tells us if it will likely be great.

I’ve argued for some time that neuromarketing is its own worst enemy, over-promising and under-delivering. Thankfully, we’ve seen less hyperbole in the last couple of years. Until this week.

Reference – www.pnas.org/cgi/doi/10.1073/pnas.1602413113

Share
You can leave a response, or trackback from your own site.

30 Responses to “A Bad Week for Neuroscience”

  1. Thomas Zoëga Ramsøy says:

    August 8th, 2016 at 8:02 am

    Dear Dr Needel

    This is an interesting piece, but only because I find it to be blatantly wrong. I find that your “smug self-satisfaction” got the better of you 😉 What you write and assert is complete and utter hogwash!

    In fact, I just wrote a GBB post on the recent meta analysis you refer to, which you can find here
    http://www.greenbookblog.org/2016/08/05/the-restless-resting-state-and-why-brain-scanning-is-still-valid/ This addresses one half of your criticism, as I believe that the meta analysis was confounded by a complete lack of understanding of the states they were measuring. If you want to avoid overclaiming and underdelivering, this would be one good place to start.

    On the second, more vague point, you state that neuromarketing companies “(…) can tell you if there is or isn’t neurological stimulation and probably can tell you where in the ad the stimulation occurs or doesn’t occur.” I am not really sure what “neurological stimulation” is here, but since the brain is always responding to events, both consciously and unconsciously, it must basically be “everything” I guess? What seems woefully amiss is a substantially nuanced view of what this “neurological stimulation” actually refers to. Are we speaking about emotional responses? That is, whether a particular ad is more emotionally engaging than another, or whether particular phases in that ad are more engaging or “flat line” dead epochs? Or could it be whether people are experiencing information overload and stress, and being less likely? And by the way, “neurological” basically refers to brain disorder, so even this concept is flawed and misleading.

    Another claim is that “we don’t understand the relationship between neurological stimulation and purchasing and we barely understand the relationship between ad-liking and purchasing.” I would say that this claim was more closer to the truth during the first hype-cycle that neuroscience went through during the early 2000s. But we’re in 2016, and even just during the past 5-7 years we have clear links that neuroscience methods not only predict individual choice (see for example these articles: http://goo.gl/qNl2qa and http://goo.gl/ROnBgy), we also see that coherent responses in a small group can predict cultural effects — which is also becoming a new discipline in cognitive neuroscience and psychology (see my earlier blog post on this here: http://www.greenbookblog.org/2014/09/10/brain-scans-cultural-popularity-and-sample-sizes/). To say that we don’t know requires that you actually know the neuroscience, and this is clearly an outdated, simplistic view.

    What is most troubling is the bandwagon ill-intended approach taken in the blog post. It seems that there is an over-eagerness in trying to kill off neuroscience as a “bad thing” for the marketing industry. In truth, I actually find that most MR methods are archaic, lack any substantial support in the literature (for crying out loud why are people still using NPS as an industry KPI???). What I find to be the REAL reason behind such attacks are rather a badly covered interest in criticising a method that, while being at at early stage in its development and contribution to the field, may provide substantial insights as well as diagnostic and predictive powers to researchers.

    I have devoted by texbook on consumer neuroscience and neuromarketing to address these issues, and I’d gladly send you a free copy if you PM me. We’re also starting the free Coursera course on neuromarketing and consumer neuroscience in September (https://www.coursera.org/learn/neuromarketing), and I hope that you and likeminded will give it a shot 😉

  2. Steve Needel says:

    August 8th, 2016 at 3:59 pm

    Thomas – first off, only my mother calls me Dr. Needel – please, it’s Steve. I never claimed to be an expert in neuroscience – but I can read and ask smart people questions. The cognitive scientists and statisticians I talked to before posting this don’t find much wrong with Ecklund et. al. except for its inherent unreadability. And to your point in your blog, they did not just use resting states but also task states. My advice – you should be writing a rejoinder to this journal, as you obviously feel strongly about the quality of the research by Ecklund et. al.

    Don’t confuse the Knutson study with real shopping. And if I’m reading the data correctly, the incremental R-squared when we add neuro info is trivial – a 1% improvement. Nor is Ravaja et. al. predicting shopping behavior. We don’t consider one product at a time and decide yes/no to buy.

    And if you read my opening paragraph (so go and re-read it), I want neuroscience to succeed in marketing. I just don’t think we’re there yet and companies that over-promise only hurt that cause, perhaps for longer than the science would mandate.

  3. Kevin Gray says:

    August 8th, 2016 at 4:19 pm

    Neuroimaging, including fMRI, has attracted the attention of statisticians for many years and is a frequent topic in prestigious journals such as the Journal of the American Statistical Association. There are serious measurement issues which are well-known to neuroscientists as well as statisticians, so the study Steve mentions came as no surprise to many of us.

    This does not mean that neuroimaging should be written off, however. I briefly outlined my thoughts in an earlier GB blog on this topic “How to Separate Neuroscience from NeuroHype”: http://www.greenbookblog.org/2015/02/24/how-to-separate-neuroscience-from-neurohype/?utm_content=buffere2ca8&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer .

  4. Andrew Jeavons says:

    August 8th, 2016 at 8:33 pm

    Thomas (we all use our first names here) I think what Steve and I would like to see is some empirical evidence for your assertions about the cognitive states of fMRI subjects. So often we see sweeping statements about neuroscience and marketing based on thin empirical evidence.

    You haven’t presented much in terms of evidence for your refutation of a meticulous paper other than “you don’t understand”. I thought science was a process of conjecture and refutation ? I’d like to see real empirical refutation of Eklund et al rather than telling us we are ignorant. Quoting a lot of theories is a great start to developing a definitive study to refute Eklund et al, when can we expect you to publish ?

    I’d be cautious about the condescension too. I’m willing to bet Steve and I studied neuropsychology a bit before you did. As Steve says you don’t need to be an expert to discuss logical holes in theories, you just have to understand how science works.

    I’ll address you comments on my comments later.

  5. Chris Robinson says:

    August 8th, 2016 at 11:30 pm

    FROM STEVE NEEDEL …
    “And to your point in your blog, they did not just use resting states but also task states. My advice – you should be writing a rejoinder to this journal, as you obviously feel strongly about the quality of the research by Ecklund et. al.”

    This study was serious rebuttal of neuroscience and as Steve has pointed out task state data also failed replicability. Steve is right, you need to address this issue.

    And as Andrew has pointed out this is a professional site and the exchange needs to be of similar tone. I did love your comment on NPS though! Most market researchers (excluding Bain) would agree.

  6. Thomas Zoëga Ramsøy says:

    August 9th, 2016 at 3:40 am

    Hi Steve, Kevin and Andrew — happy to go to first names!

    First things first — I’m terribly sorry if my tone came across as inappropriate and condescending. I never intended that to be the case, and I try to use smileys to avoid exactly this, but without looking like a teenager 🙂

    And thanks for giving me the benefit of a younger age (I might have to update my profile pictures). In any case, its always great to meet fellow neuropsychologists.

    So to the matters:
    On the Eklund study, the statistics is definitely sound, but the interpretation is flawed. While the “resting state” or “default mode” studies can be great for studying brain dynamics, it is a misunderstanding to use it for testing reliability across studies. This criticism has been raised several times, most notably by Morcom and Fletcher in 2007 (http://goo.gl/at95AR). Others, such as Torben Lund and colleagues, have demonstrated that “resting state” fMRI studies are themselves prone to a lot of false positives due to respiration, pulse and other aspects, and that activity related to these physiological factors suspiciously overlap with “resting state” results (http://goo.gl/zjrSG).

    On the cognitive states of fMRI, I was originally referring to decades of research in psychology on daydreaming, task unrelated images and thoughts, and related topics, especially in the 70s to 90s (a collection can be found here https://goo.gl/HGxR1s). Recent studies by people like Smallwood and colleagues has also provided new insights into this aspect of the human mind (https://goo.gl/SeBDbH).

    This is why we should be very careful with over-interpreting results from a meta-analysis on studies that themselves have been criticised on fundamental grounds. If the underlying phenomenon is less reliable than the premises of the statistical analyses assume, the analysis will be corrupted.

    On Knutson, Ravaja and other studies; these are early stage lab based studies, but for Knutson, the “proof of concept” was that the explanatory R-square was as good as self reports (something that later studies have surpassed). Recent studies have now supported and replicated these early findings in actual shopping behaviours. For example, my lab has a manuscript from an in-store EEG (and eye-tracking) study now in the final stages of review, a conference paper can be found here (https://goo.gl/VRpJqf) and I can send the ms if it could be of interest. What we report, is that we can confirm that brain derived measures (EEG) during product viewing can predict purchase with an over 90% accuracy.

    Other studies have related brain activity to cultural effects. Dmochowski and colleagues found that both EEG and fMRI during viewing of TV drama and advertisements could predict population responses (Nielsen ratings and tweets). Others have done similar findings on emotional responses in one cohort that predicts market-level microlending (http://goo.gl/APkiES), public health campaign success (http://goo.gl/5vZsB8) and ad success (http://goo.gl/aLzJbP). I am aware that these findings are recent, and probably needs a review and broader presentation — so perhaps that should be on the to do list 😉

    BTW Kevin, thanks for pointing to your blog post — it’s a great collection of advice.

    Chris: I think I need to read the section of the paper that dealt with task related matters and revert.

    Just a final side note — the Proceedings of the National Academy of Sciences (or PNAS) is considered one of the top science journals, ranking just under journals such as Nature and Science. Actually, the nerdy name for PNAS is Previously Not Accepted in Science 😉
    And you’re right, I should probably try to get a comment into this journal.

    Looking forward to your responses.

  7. Eli Draluk says:

    August 9th, 2016 at 12:17 pm

    They also did a study that essentially indicates that most of the high dollar spend associated with it can be replicated for much less through social media. https://blog.twitter.com/2015/nielsen-twitter-x-tv-activity-levels-indicate-total-audience-engagement

  8. Kevin Gray says:

    August 10th, 2016 at 2:20 am

    The Journal of Marketing Research has dedicated a special issue to neuromarketing (August, 2015) – https://www.ama.org/publications/JournalOfMarketingResearch/Pages/JMR-TOC-2015-4.aspx

    It may still be paywalled for non-AMA members but I believe anyone can access the abstracts. Definitely worth a read, IMO.

  9. Thomas Zoëga Ramsøy says:

    August 10th, 2016 at 2:54 am

    I’d also like to unashamedly point you to my textbook on this: https://goo.gl/NzPnlJ
    and a freely available compendium of recommended articles: https://dl.dropboxusercontent.com/content_link/aSjBMvSt6lkJpDwjuQxGatrpFeMnx62he8Ge4rkOIjzNdEnvI8DXDBlk6ICgoyUO/file

  10. Ellen Woods says:

    August 11th, 2016 at 12:07 pm

    Hi All:

    It seems that what is at issue here is not the validity of the testing but instead the ability to apply it to an actual event. The problem with any predictive model has always been that it can only predict under specific conditions, not dynamic interactions.

    In a perfect world, people would always do what they intend do in the time frame they intended for completion. We know this doesn’t happen consistently. When you test variables that have limitations you can do a pretty good job of interpreting the range of response. When you have an un-contained set of variables, it becomes patently more difficult.

    Neuroimaging seems to do a good job of defining clear winners and losers but not so much when it comes to things that fall in between. What that would seem to imply is that the technique may be useful in trending (things like car colors, materials testing, dashboard layouts, etc.) but not so much in areas where refinement is necessary. Basically, it has to be evaluated on the effort, cost and delivery vs. existing techniques in those areas along with the efficacy of the testing vs. existing methods. My guess is that it would be better in some areas than others and that there will still be a struggle with sampling. Who really wants to take a medical test for marketing? What happens if there is an anomaly discovered? HIPPA invocations?

    Aside of all that, even with this technique we are still stuck on quantifying behaviors in a world where we encourage individuality in choices. Doesn’t that seem counter intuitive? Instead of trying to predict, what if we instead involved consumers at an earlier stage? Many people think they know what they want until they get it and then it isn’t what they want. Have you ever been excited to test drive a car you researched and then hated it? Consumers are very good at decisions where hard choices are made. Maybe with VR included, NI gets more practical and predictive in a measurable way?

  11. Kevin Gray says:

    August 11th, 2016 at 5:07 pm

    This, from the article’s abstract, is what first caught my eye:

    “Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data.”

    That aside, here is a discussion that may be of interest:

    http://blogs.discovermagazine.com/neuroskeptic/2016/07/07/false-positive-fmri-mainstream/#.V6zk-aIcbMK

    Thomas (or anyone) can join the fray is they wish as far as I know.

    FYI, I have no commercial interests in neuroimaging whatsoever. I work with all sorts of data and the way neuroscience most affects my business is through potentially better quality ad pre-testing that I can incorporate into Market Response Modeling (aka Marketing Mix Modeling).

  12. Thomas Zoëga Ramsøy says:

    August 12th, 2016 at 8:39 am

    Thanks for the pointer, Kevin. I have followed the blog post, but not contributed. Seems they have debunked the media hype on this pretty quickly, and some of thee more leading scientists have dug into the data and found that this at best relates to 11% of all studies (http://blogs.warwick.ac.uk/nichols/entry/bibliometrics_of_cluster). It’s still a sizeable chunk, but nowhere near the 40k or so papers being invalid, as originally claimed.

    To be perfectly honest, I am constantly impressed how researchers within the neuroimaging domain are extremely perfectionist when it comes to their statistics, and turn every single stone in making sure that their methods are sound. For example, if you want to do proper big data analysis, hire a person who has dealt with neuroimaging like fMRI.

    My own contribution so far has been to show other problems with fMRI, such as that the at the time standard spatial normalisation algorithm led to errors in where activity was shown and leading to erroneous conclusions), and that certain inhomogeneities in the magnetic field in certain brain regions can also lead to signal errors (and also wrong conclusions about brain regions). Neuroimaging is hampered with problems, this is nothing new.

    What continues to surprise me is the dual standards that is used in business: applied neuroscience is criticized on the basis of its methods, premises, statistics and conclusions, at the same time as we see a complete agnosticism towards the validity and reliability of more traditional research methods. I briefly mentioned the NPS score in a previous comment, and other similar scores suffer the same type of external validity, especially when companies and researchers have so strong vested interests in these metrics and KPIs.

    Also, I should note that fMRI is many things, although we mostly talk about one method; the Blood Oxygen-Level Dependent (BOLD) signal, and not other measures such as perfusion based fMRI (there are also spectography based methods now). Also, it seems that the claim is mostly relevant to only one statistical software package (AFNI) and not the even more predominant ones such as SPM and FSL.

    @Ellen, you write that neurosience “may be useful in trending (…) but not so much in areas where refinement is necessary” to which I must object. When we do studies and measure particular emotional and cognitive responses, these quantified responses not only show differences between ads/products/signs etc, but the scores provide information about a direction in which improvements can be made, such as whether to increase or reduce the amount of information, to make something more or less visually salient, to increase or decrease emotional contents. We see this being used every day in improving anything from webpages, ads and communications, to apps, stores and human-robot interactions.

    You also ask: “What happens if there is an anomaly discovered?” The answer is pretty straightforward: we do not look for or see any anomalies, and it would be comparable to finding an interviewee behaving in an erratic manner. The most often used neuroscience method used in industry is EEG, and you will need to employ a very specific approach to detect anomalies. In part, this is also avoided with specific recruitment criteria.

    The combination of VR (and AR) and NI is indeed one of the promising paths — and for those interested, we did a little funny piece on brain responses to Pokémon GO here: http://neuronsinc.com/neurons-brainsplash-how-your-brain-responds-to-pokemon-go/

  13. Mark Westaby says:

    August 14th, 2016 at 6:14 am

    This is a classic example of that wonderful phrase: “All computer models are wrong but some are useful”.

    I can’t comment on the rights and wrongs of the argument because I haven’t looked in any depth at either side. I do recognise, however, the classic symptoms of what we’re now seeing more and more in today’s data-driven world, whether the subject is marketing, climate science or some other area where “patterns” in data are supposed to emerge.

    With a background in science and engineering I love those multi-coloured diagrams where clusters emerge from the data — they look great and we all dream of producing them in our work. Unfortunately, in the majority of day-to-day cases such examples are rare. What we tend to see are much fuzzier pictures that require a great deal of interpretation.

    As tends to be the case in cases such as this I suspect the truth lies somewhere in the middle of the argument. The neuroscientists are almost certainly making claims based on case studies that have worked — strangely enough, we never hear of the ones that don’t; and they tend to be the vast majority.

    As Steve says, however, that doesn’t make their work irrelevant. As with the big data it just means that anybody using neuroscience in marketing needs to be very cautious about the claims being made.

  14. Kevin Gray says:

    August 15th, 2016 at 4:41 pm

    The paper itself hasn’t been debunked – though Nichols (one of the authors) has expressed regret over the 40,000 figure, which is what some of the media keyed in on. Always fact check the media! I see no evidence that the review process at the Proceedings of the National Academy of Sciences has broken down though, fortunately, which would have had serious implications far beyond marketing.
    On GB and in much of the mr blogosphere, we very often see criticisms of established methods, and NPS has been a particular target. Buyers of mr, unfortunately, seem to pay scant attention to these sources and seldom attend mr conferences, thus bad habits are carried on, from generation to generation in some cases.

  15. Aaron Reid says:

    August 18th, 2016 at 12:45 pm

    Hi Steve,

    Thomas has covered off on a lot of the important content questioning the conclusions of the PNAS study you referenced. But I’ll add a few key points here.

    First, Michael Smith of Nielsen Consumer Neuroscience had a good point in a LinkedIn discussion of this article that is worth repeating here. The vast majority of applied neuroscience studies within market research are using EEG, not fMRI. So the results of this meta-analysis do not call into question the results for most market research studies using neuro-methods.

    Second, I’m surprised that you would object so strongly to the ad stating that a research company’s methods can make ads better. (Note that the ad is probably from one of our competitors, but I don’t object to it, and I’m not averse to objecting to competitor claims in this field (http://bit.ly/2aZzEMZ, http://bit.ly/2b71tjz)). There is a lot of evidence that the use of these techniques (and other market research techniques) does provide insight on how to improve the effectiveness of ads, both on how to optimally trim your content down from 60 to 30 to 15 second spots, and how to increase the impact of the ad on in-market behavior (e.g. social sharing of an ad, increased call-to-action behaviors, sales etc.). I don’t think its an over-promise to suggest that a research company can improve the effectiveness of an ad, and believe me, there have been plenty of over-promises by neuro-based firms in the past.

    Third, we should start thinking about the measurement of non-conscious processes more broadly and include true implicit association techniques within the mix of methods that can add value to traditional conscious research methods. Numerous studies from the past 10 years have shown that adding implicit association data to the best conscious measures in our industry (e.g. choice based conjoint, MaxDiff trade-offs and other forms of derived preference) significantly increases the accuracy of predictions of in-market behavior (e.g. new product sales, the virality of an ad, etc.). The increases in r-square in those studies is in the 25% range – clearly non-trivial.

    And lastly, it is important to note that not all scientists using non-conscious techniques are applying and analyzing the methods appropriately. This is true of both academic and applied scientists – you can read a recent critique of an article published in collaboration with the ARF here (http://bit.ly/implicitadtesting). Just as a “questionnaire” can be poorly designed and inappropriately analyzed, so too can non-conscious measurement techniques. Upon reading the results of a meta analysis of poorly designed and analyzed questionnaire studies, we wouldn’t conclude that the method of “questionnaire” was faulty. We should be careful to not disparage a discipline because of examples of poor design or inappropriate analysis. This is particularly important in the early stages of consumer non-conscious research methods, which hold great promise for us in more accurately predicting what people will do and providing deeper insight on why they will do it.

  16. Steve Needel says:

    August 19th, 2016 at 9:15 am

    Aaron – I’ll partly agree and partly disagree with your comment, mostly the former. First, so you know where I’m coming from, I’m not bashing neuroscience – I want it to help MR do our job better, and I say that very clearly in my opening paragraph. Recall that much of the interest in neuroscience originated with Lindstrom’s “Buyology” and his Pepsi/Coke fMRI examples. EEG has its own issues, but apparently not as much as fMRI may have.

    Advertising is a creative process and, perhaps, a psychological process (I like to think so, as I’m a psychologist). Neuroscience can tell us if people like the ad or not via facial recognition, whether it’s igniting an emotional response or not via EEG, fMRI, GSR, etc., what parts people are attending to and not attending to via eye-tracking. Knowing all of that does not mean you can make a good ad better. It’s a scorecard, nothing more. If there’s a scene that fails to garner attention or drive the intended emotion, then the creatives might need to re-work the scene. Neuroscience hasn’t made the ad better, creatives have. And that said, there’s no guarantee that anything they do will make the ad better.

    As to implicit measures, great, but let’s realize that you’ve only seen the positive outcomes, not the cases where implicit didn’t add anything to predictability, because who’s going to tout negative results. If they routinely added 25% to the r-squared, they would be the standard. The fact that they are not suggests that sometimes they may, but not often enough.

  17. Thomas Zoëga Ramsøy says:

    August 19th, 2016 at 10:34 am

    Thanks Aron and Steve, this is becoming a very important discussion.

    The scorecard analogy, while interesting, is too simple. Indeed, neuroscience done right can go beyond a mere scorecard and be prescriptive. Some examples:
    – if cognitive load scores are too high, customers lose focus and interest and it has a detrimental effect on their product/service preference and choice. The solution is to reduce cognitive load byt for example taking out some of the less important information in an ad, sometimes to simplify cutting of scenes, and overall improving the flow of an ad narrative
    – if emotional responses to specific events do not work as they were intended, then the combination of arousal and valence/motivation points to whether you need to boost the overall emotional engagement or make something more or less positive (or negative, depending on the intended effect)
    – even boosting attention is more than a scorecard: by combining eye-tracking data with other measures (e.g. predictive bottom-up attention tools like NeuroVision) it is possible to understand why certain things are not sufficiently attended, and therefore whether something needs to be made more visually salient, or whether one needs to work on boosting emotional responses to ensure attention in this way

    These are merely three suggestions, and Aron mentioned that neuroscience has the granularity to help clipping and cutting ads in an optimal way.

    These are not merely assertion, they are happening on a daily basis in advertising, and we see that many creatives are embracing these new insights. First, many of these tools allow early vetting of ad effects (e.g. whether a brand logo is sufficiently salient), while full test results typically helps structure the roundtable discussion on which ad version to go with. Instead of the so-often discussion about subjective ad preferences, we see that neuro results help sort the wheat from the chaff — as in “this ad looks great, but when we tested it, we didn’t get the desired responses on X, Y or Z, so we need to rethink it”.

    I realise that this has not been written up anywhere, but that would be an obvious next thing: a pamphlet, book or suitable forum to present this “neuro thinking” in ad design.

  18. Steve Needel says:

    August 19th, 2016 at 10:49 am

    Thomas – all that said, it doesn’t mean you’ll improve the ad. The research can point out good and bad parts, or strengths and weaknesses. Changing it does not necessarily make it better.

  19. Kevin says:

    August 19th, 2016 at 3:33 pm

    Regardless of the methodology or research area, when a new method is claimed to be superior to a traditional method we, as marketing researchers, should closely examine how the new method is new and if the enhancements are actually relevant.

    More importantly, we also should consider the benchmark the new method is being compared against. It might turn out to be substandard practice, not traditional practice as claimed. Often survey data, for example, is not analyzed beyond the level of looking at totals and some simple cross tabs.

    These discussions bring back memories of intensive training some former colleagues and I had with Paul Heylen on his “Implicit” (IMPSYS) method. This was 20 years ago and IMPSYS was already well-established – Paul developed it in the 80s, if memory serves, and interest in implicit measurement dates back much further in MR. Though surely progress has been made since then, many of the current discussions of implicit measurement of various kinds are virtually identical to what I’ve heard for decades.

  20. Kevin says:

    August 19th, 2016 at 5:15 pm

    I won’t be following this discussion any more – time to move on, as the saying goes. Though I have no commercial interests in neuroscience, this discussion has been very stimulating and I’d like to thank Steve for initiating it.

    The PNAS paper has not invalidated neuroscience, something its authors never have maintained, not has the paper itself been invalidated. The authors – and surely the reviewers – were well aware that the brain is always active in various ways. This is neuroscience 101, and the design of the research and the methods used to analyze the data took this into account. Anyone is free to challenge the authors directly regarding this criticism of the paper.

    When evaluating new methodologies, we need to consider their practical implications as well. Say, for instance, one method has a substantially higher model fit (judged by widely-accepted criteria) than another. We need to ask if the “better” method would actually lead to different decisions, and better decisions. It may, but we shouldn’t take this for granted.

    Just one final comment, regarding claims that a model has been validated on the basis of a high correlation with external data. Actually, it is generally not difficult to tune models post hoc to fit external data, especially when the latter have been cherry-picked. This unethical practice, sadly, is not uncommon in science and I would be surprised if it has never happened in marketing research.

  21. Tom Palmer says:

    August 19th, 2016 at 6:57 pm

    Dear all, this is a wonderful discussion and I am grateful for the academic perspective. One critical component we are missing in this discussion stems from what Daniel Kahneman refers to as the Experiencing Self vs. the Remembering Self. For the most part, without parsing the pro’s and con’s of specific tools, neuro-research illuminates the presence and nature of “real time” response within the Experiencing Self. However, the Remembering Self does not always see things the same way as the Experiencing Self, and as Kahneman puts it simply, “The Remembering Self is sometimes wrong, but it is the one that keeps score…and it is the one that makes decisions”. Some of those decisions are made with a set of biases and heuristics described as System 1 while others leverage the more effortful processing of System 2 or some combination of the two (as an aside, there is a common misperception that System 1/2 is a binary model of behavior, which it is not). Regardless of cognitive effort associated with the decision, those who claim that neuro-research of the Experiencing Self is the only “reality” in marketing are sadly misleading the practitioners in our industry.

    The truth is that human behavior is largely driven by memories (implicit and/or explicit) of prior events including advertising exposure. When we buy brands we are not actually buying future experiences; instead we are buying future memories. When advertising drives our behavior as consumers we we are not responding to the experience of watching the ad; instead we are responding to our memory of the ad. To fully understand how an advertisement is working – and how we might make it better – we need to understand how it is stored and retrieved by the Remembering Self and how that process changes the memory structures associated with the brand and the desired behavior.

    To build on the prior example of cutting a :60 to a :30, this could be quite dangerous if one deletes frames based on real-time response of the Experiencer Self. In our R&D we have seen that the remembered content and meaning of specific commercial frames does not always align with neuro-measures. The best way to ensure success here is to remove the bits of the ad that are not relevant to the Remembering Self and to keep only what is most relevant to the Remembering Self.

    Agency-side creative teams instinctively know this as the art of story-telling in which a scene that does not generate an immediate “pop” in neuro-measurement may in fact be critical to the emotional structure of the ad. Conversely, a scene that “pops” in neuro-measurement may actually be superfluous, or its meaning may hinge entirely on the scenes that come before and after.. In our experience, the best way to cut :60 to :30 (or :30 to :15) is to understand how the Remembering Self selects and retains just a few critical frames that, when linked together in the consumer’s mind, tell the intended story complete with emotion and meaning and a clear role for the brand. Those are the scenes you keep when cutting :60 to :30.

  22. Aaron Reid says:

    August 21st, 2016 at 11:29 am

    Hi all,

    A few comments here to clarify and keep the discussion going…

    Tom, an important benefit of including implicit association measures in ad testing paradigms is that you have a memory impact variable. Coupling implicit memory impact with the moment by moment granularity of neuro, bio and eye-tracking measures provides the insight needed for improving both the experience of the ad and the impact of reflecting on it.

    Kevin, don’t check out yet – I’ll bet there is more that you can contribute here. To wit, your comment on validation is critical for the buyers in the industry to understand. In my experience, many providers dangle .9 correlations in pitch decks without specifying the details necessary for evaluation. It is, in fact, fairly easy to produce .9 correlations with post-hoc analyses that allow parameters to freely vary in order to maximize fit. In contrast, the improved r-squared values of 25% that I mentioned above are derived from a theoretically based, fully specified a-priori mathematical model that combines emotion and reason to predict behavior. Further, the improvements are not above and beyond poorly designed conscious measures. Rather the improvements are being made over the best derived preference models in practice in our industry. We don’t take validation lightly, it is critical for the long-term success of consumer neuroscience and the advancement of our industry.

    Steve, thorough insights work in the ad-testing space should provide more than a scorecard. It should also provide diagnostics and prescriptive direction for the success of an ad. The process of improving an ad involves valid research design, analysis to unearth insights, recommendations based on the new knowledge, and then creative execution. To argue that providing this insight and direction does not improve the ad, is not only splitting semantic hairs, it it veers dangerously close to arguing that research cannot improve marketing in general. I find that argument an inaccurate, unnecessarily cynical and damaging for perceptions of the industry. If we’re not able to improve marketing through research, why then are we in the business?

    Lastly, on your assertion that “if implicit were that good, it would be the standard” – I would ask you: what do you think a reasonable amount of time is for a method to become a standard approach in our industry? Remember that this is an industry that is still relying heavily on explicit questions of “importance” to uncover drivers, and whose advanced consumer choice models still rely on the rational expected utility maximization model of decision-making (which has been thoroughly debunked in the behavioral science literature for more than 30 years).

    From my perspective, these methods will become standard in the coming years, and the speed with which they do will depend on our collective scientific integrity with the methods, dissemination of market validation case studies, increased awareness of the applications (beyond ad-testing), and the scalability of the measurement tools.

  23. Steve Needel says:

    August 21st, 2016 at 2:45 pm

    Aaron – my point regarding advertising is a simple one – the neurological or physiological measures currently employed in advertising tend to be of the yes/no variety – yes/no there was an emotional response, yes/no there was attention, yes/no there was blood flow in the brain, and so forth. Even granting that there is a degree of response, allowing one to say too much or too little, changing that has no direct implication for making a good ad a great ad. That’s because an ad is a gestalt, not a series of isolated frames. Changing an element to heighten a low emotional response is no guarantee that it will make the ad better. Indeed, Ehrenber, in J. Adv. Rsch, 1974, suggests that advertising’s role is often a reinforcer of a purchase decision already made, reducing post-decisional dissonance. Such a reduction would actually imply a lowering of neurological responses.

    In its current form, Implicit Association testing has been around for 18 years or so (from Greenwald’s 1998 paper), more if you figure it’s a takeoff on other techniques in use much longer. I’m thinking if it’s SO useful, 18 years is enough time to show itself. If it routinely generated 25% better predictions, then it should be standard by now, and at least one of the big ad testing companies would be pushing it hard and often.

  24. Thomas Zoëga Ramsøy says:

    August 21st, 2016 at 3:52 pm

    Hi Steve, thanks for more fodder for the discussion 😉

    I think I need to understand better how “ad as a Gestalt” works here. This would imply that — following the Gestalt thinking — the content of the ad is more than the sum of its frames. I am not sure I agree at all, and besides noting that it does smack very much like the “mad men era” of advertising, I have some immediate notes:

    * success criteria/KPIs of an ad include 1) leading to change in behaviour, and 2) leading to change in (brand) perception. We may call an ad a Gestalt or not, but if it does not succeed in this, nomenclature is not relevant (and the model is probably wrong)

    * the success criteria are both tangible and possible to measure, and the claim neuro assert is that it can 1) predict these KPIs from responses during ad viewing (the aforementioned Dmochowsky article in Scientific Report being a good example), and 2) lead to ways to improve the ad for these KPIs

    * As far as I can see, the ad as Gestalt idea relates to the way in which only one system of the brain — the Kahneman’s System 2 — perceives the ad. It is the It is the conscious narrative, the explicit memory, and related ad memory and ad liking. Curiously, what drives these effects are also predictable from particular brain responses, such as responses in the orbitofrontal cortex, which are highly related to subjective hedonic experience. Is consciousness “reducable” to the brain? A fair chunk of my own academic research has provided good indication for this link (such as https://goo.gl/AHbykH and http://goo.gl/k8ahXq).

    * the claim from neuromarketing (I prefer the briader term “applied neuroscience”) is that the measures being used can pinpoint specific events in ads that produce emotional and cognitive responses that ultimately lead to the KPIs. This goes to your point on ad as Gestalt — my claim is that the ad Gestalt (if it exists) still is composed of crtiical elements that, if they are altered, will still change the Gestalt. Understanding these elements, being able to measure them, and knowing how to use this knowledge to alter the elements in one’s direction will still be critical. After all, take any Gestalt perception and change it’s components. At some point the Gestalt seizes to be the same. The same should apply for elements of an ad, right?

    * Another claim from applied neuroscience is that it provides an additional understanding of human (and consumer) behaviour, which increases our understanding of human/consumer perception, thinking and choice. This does NOT take anything away from existing approaches — a common misconception in these tyoes of discussions. For example, psychology initially had an extreme degree of resistance towards neuroscience and neurobiology, and both psychiatric and neurological problems were attributed to “purely psychological” phenomena. Today, we have fortunately seen that there is no dichotomy between the mind and the brain, and that psychiatry, neurology and other disciplines embrace biology as part of their language, models and treatments.

    * To your point on post-purchase dissonance. Indeed, what neuroscience studies consistently finds is that our choices are first and foremost driven by unconscious processes, and that our conscious thinking only comes “online” much later (and too late to actually influence most decisions).

    * Implicit association testing has indeed existed for a very long time, and I’d suggest that the post-war era was probaly even earlier, especially in Continental Europe such as Germany and France in proposing that these measures could predict ad effects and other effects. These measures are pretty crude, however, and need multiple repeated measures, which are not suitable for most ad testing, unless we’re trying to test if an ad changes people’s emotional responses to ads.

    * In the same vein, neuroscience tools have been around for about 15 years, a mere fraction of what more traditional measures have been. Still, I would be as crazy to claim that many traditional measures are horrible at predicting ad/communication success, and that neuroscience can do much better with only a subset of the same. Moreover, that traditional measures are not sufficiently good to diagnose or improve the ad, but that neuro does all that.

    * A brief note on nomenclature: the brain does not show any yes/no responses any more than humans are binary in their behaviours. Moreover, I have tried to go beyond the discussion of whether the brain “shows more or less response” — the brain is a massively parallel system that does a gazillion things at any one time, and merely reducing it to “more or less response” is way too simplistic and possibly a huge straw man (although not intended as such).

    The discussion of neurobiology in understanding (and predicting, shaping) human thought and behaviour has a long history of being subject of antagonism , mockery and ignorance, but I believe that just as we have seen in these other disciplines, neurobiology has a very valuable contribution to advertising and beyond.

  25. Aaron Reid says:

    August 21st, 2016 at 10:27 pm

    Hi Steve – it is starting to appear through your comments that you may not understand how implicit association testing is used in combination with moment-by-moment applied neuroscience tools within ad-testing.

    *With implicit association testing, you have a non-conscious impact variable. That is, you have an outcome variable that is dependent on the quality of the ad that is observed. Further, we know that this non-conscious outcome variable is related to measures of ad success (e.g. online views, social shares, digital SOV). So yes, when changes are made to an ad based on insights from applied neuroscience techniques, and those changes produce improvements in the implicit impact of exposure to an ad, then it is fair to say that the research did improve the ad.

    *Thomas, most ad-testing is perfectly suited for the application of implicit association testing. We should talk about this in greater detail offline if you’re interested. If you’re not already using it, I think you might find the addition of true implicit technology could be of benefit. I’m assuming there is a typo in your comment above “unless we’re trying to test if an ad changes people’s emotional responses to ads” – did you mean “brand” instead of “ads” at the end? I ask, because that is exactly what we are trying to test with implicit associations within ad testing – we are measuring change in /reinforcement of key emotional associations with the brand or product featured in the ad. We find that understanding whether exposure to an ad influences emotional associations with the target brand or product is highly valuable to clients (and as I mentioned above, is predictive of ad success).

  26. Thomas Zoëga Ramsøy says:

    August 22nd, 2016 at 6:58 am

    Aaron — I think that I simplified the sentences too much, so what I meant was “implicit association tasks”, which are not suitable for testing second by second responses in ads. The broader view on implicit testing in general (which would include EEG) I agree completely, but would still love to talk directly, there’s always new stuff to learn – I’ll reach out to you on this.

    Steve — one thing I am wondering about is whether there is anything in consumers’ ad responses — ad events or Gestalts alike — that are not the result of brain activity? If the Gestalt is a crucial element of ad effects, would it still not be possible to measure with brain scanning technology?

  27. Chris Robinson says:

    August 24th, 2016 at 6:12 am

    Aaron, I am a bit lost here. Are you saying that the proof of neuroscience’s effectiveness is in the change in the implicit responses after viewing an ad that has been modified through some neurosciences input? I cannot clearly see how it would be the reverse, that implicit tools would enhance the advertising effects? I am open to proof of that. The JMR article from the Aug 2015 issue (Predicting Advertising Success Beyond Traditional Measures: New Insights from Neurophysiological Methods and Market Response Modeling) found no relationship between implicit techniques and the predictive ability to deliver ad effectiveness. I know this has been criticized by some key players in the industry.

  28. Christopher Robinson says:

    August 25th, 2016 at 2:37 am

    This article in ScienceNews provides further criticism of the applications of neuroscience in understanding how the brain works. They took, instead of a brain, a microchip that had been used in games like Donkey Kong. Since they knew how the chip worked they wanted to find out if Neuroscience could predict how the chip functioned. It apparently couldn’t. There are some useful rejoinders in the article. The main point of the article is in this quote.

    “”The paper “does a great job of articulating something that most thoughtful people believe but haven’t said out loud,” says neuroscientist Anthony Zador of Cold Spring Harbor Laboratory in New York. “Their point is that it’s not clear that the current methods would ever allow us to understand how the brain computes in [a] fundamental way,” he says. “And I don’t necessarily disagree.”

    https://www.sciencenews.org/article/what-donkey-kong-can-tell-us-about-how-study-brain

  29. Thomas Zoëga Ramsøy says:

    August 25th, 2016 at 5:01 am

    I’m siding on the people who think this is drawing the analogy too far. By the same token, this would invalidate psychology because it does not fully understand the “language of thought”; it would invalidate marketing because it does not fully understand consumer motivations; physics would definitely be thrown out because it cannot explain the majority of what the universe is made of. Should we criticize the science because it still hasn’t understood some of he fundamental properties? I’m sure you’ll agree that is a bad decision.

    IMO, this is like not allowing ourselves to bake a cake unless we completely understand how every ingredient mix at the molecular or subatomic level.

    Fortunately, neuroscience, like so many disciplines, rely on multiple sources. Also, it not only tries to explain the brain, but also is successful in manipulating specific regions of the brain to produce desired outcomes. It makes good predictions about behavior. Even from the days of Roger Sperry’s split brain studies, or earlier observations from Broca and Wernicke, we have been on the right track in understanding the function-structure relationship between the mind-brain.

    Sure, we don’t understand the language of the brain in full, but we have a fair inkling about it, and this is more than enough to make valid and successful predictions about the real world that can be used in treating a patient or improving our understanding and influence on consumer choice.

  30. Aaron Reid says:

    August 26th, 2016 at 9:53 am

    Hi Chris – thanks for your comment and questions.

    *Please see my critique of the Venkatraman et al. (2015) article here: http://bit.ly/implicitadtesting

    *There are some important contributions of that paper, but also many fundamental design and analysis flaws. I think you’ll find in the critique that their use of implicit associations and their analysis of EEG raises significant questions about the conclusions that they draw.

    *To your inquiry on whether the use of implicit tools would enhance the effects of advertising – I’m not sure I understand your question – implicit association techniques are used to measure the impact of advertising. Can you clarify your question?

Leave a Reply

*

%d bloggers like this: