1. Clear Seas Research
  2. SIS International Research
  3. Research Live
  4. RN_In_App_Effectiveness_GBook_480_60

Honesty of Responses: The 7 Factors at Play

We have witnessed questions where up to 30% or respondents show signs of untruthfulness based upon the context of how you ask the question and the type of questions being answered and who you are asking. The reasons for this, though, are complicated. We have identified a range of driving factors that determine the level of authenticity to an answer we may give.

By Jon Puleston

I read this very interesting post by Edward Appleton about the authenticity of peoples online behavior.

The authenticity of online respondents is a very interesting philosophical question and is an area we have been looking at recently too - albeit in perhaps in a more literal way...

We have been examining ‘honesty’ within the online survey environment, by comparing answers to questions where we hold known norm figures about behavioural activity and building up a picture of levels of honest answering to different types of questions.

We have found that in the main most people in western* markets who respond to surveys are, on the whole, very honest in the answers they give.

Rolled up into averages, about 95% give "honest" answers to the average question - so it is not something you need to worry too much about for a typical survey.

But that figure of 95% is for the average question. This figure can vary dramatically and we have witnessed questions where up to 30% or respondents show signs of untruthfulness based upon the context of how you ask the question and the type of questions being answered and who you are asking.

The reasons for this, though, are complicated. We have identified a range of driving factors that determine the level of authenticity to an answer we may give, these are:

1. The Peacock syndrome: We like to show off and make others think we are smarter, wealthier, happier, better than other people. You can see this issue in effect on magazine readership studies where, for example, the claimed number of readers per copy of Vogue magazine is 12 - about four times higher than the average magazine. The number of people who claim to shop at Harrods the premium London on the TGI survey is about 2 times higher than the estimated footfall*. So you can never really trust stated figures about any claimed behavioral activity if it is socially desirable.

You must also remember that this process works in reverse. So as an example, people admitting to watching soap operas, smoking or noticing advertising, all things that we don't like to admit to doing, are all downgraded by us naturally.

*These figures themselves may not be completely truthful, they are plucked from my memory which is a well known source of untruthfulness if I am to be brutally honest!

2. Pleasers: This is perhaps more of a cultural issue. , Some people like to answer in certain ways to please you. Now, this is different to the peacock syndrome and it effects different types of questions, those that are less commercial questions and more social questions such as political opinions as an example. We are often very reticent to give our real views on certain topics for fear of perhaps offending. Or we may worry that our views are not socially acceptable. Or even in the extreme case of say, the fear of persecution if you were conducting research in an oppressive political regime.

3. Gamers: Now these are a set of people who adapt their answers to play the system, which is a big problem particularly in certain countries and for certain types of topic. For example, if I am trying to sample top end business people and I am offering a high incentive to take part then there will be those who will try to pretend to be business people so that they can qualify to participate in the surveys. But this is not just limited to a small group of fraudulent respondents. A lot of respondents start gaming the system when they simply get bored. They stop saying they do things or buy things to avoid having to answer extra follow-on questions.

4 Disengagement: This leads onto the last group which is the disengaged respondents who don't answer truthfully because they can't be bothered to think about it. It manifests itself on lots of different levels. Ad recall can vary by upwards of 100% between engaged and disengage respondent groups.

5. We lie to ourselves: We are all naturally quite self delusional. The image we have of ourselves photographically, for example, is often nothing like the image that other people see. So if you asked me to pick a picture which is a good likeness of yourself nine times out of ten it will be a different picture to what someone else will pick. It results in peacock answering but also a lot of delusional responses to social behavior questions e.g. are you the sort of person who would recycle? We all like to think we do and a lot of us believe we do, but it is something we don't actually do in practice. Here is a link to a wonderful paper exploring this issue http://www.tandfonline.com/doi/pdf/10.1080/15298868.2011.617886.

6. The power of the unconscious: So much of what we do is controlled by unconscious decision making paths. The conscious v unconscious mind are often in conflict. We, as a result, often make irrational decisions and so we are not very good observers of aspects of our behaviour which is controlled by the unconscious. As a result we are hopeless predictors of certain things we do. Obviously this issue sits at the heart of behavioural economics.

7. General Ignorance: We simply don't know or don't remember answers to certain often very clearly obvious questions: do I have brown eyes? I don't know. Where do I live in the US - 10% of people don't know the answer to this, particularly if you show them a map of where they live (to the extent we had to stop using maps for this question in our surveys!). We also are unable to predict our future behaviour because we don't really know what we will do. , We are, in fact, particularly bad at this. You may as well pluck a random number out of your head instead of asking somebody if they are likely to buy a new product in the future or not as it is likely to be just as accurate.

Cross cultural differences in truthfulness

I have starred *Western markets earlier on because in non western markets the level of "honest" answering can drop significantly down to, in some cases, around 70%. In fact in some countries you can forget any other worry about panel quality and demographic sample biases. The level of honesty is the Number One issue that you need to contend with. The variance in levels of truthfulness is compounded by factors such as income. If the incentive to take part in a survey is relatively higher in one country v another you will see higher numbers of people motivated to be gamers for example.

The big questions

The big questions are to what extent can these relative factor be quantified? What impact do these things have relatively?, and what tools and techniques do we have to counter them?

This is something our Chief Scientist at GMI, Mitch Eggers has been extensively exploring over the last year and I hope we will be able to publish a paper on the topic at some stage in the near future.

Share
You can leave a response, or trackback from your own site.

3 Responses to “Honesty of Responses: The 7 Factors at Play”

  1. edward04 says:

    March 4th, 2012 at 3:20 pm

    Nice post. I tend to believe that estimating 7 factors is perhaps underplaying it – check out the list of cognitive biases here – http://en.wikipedia.org/wiki/List_of_cognitive_biases. Humbling.

  2. jon puleston says:

    March 5th, 2012 at 3:22 am

    Hey this Wikipedia list is great! And really useful.

  3. jon puleston says:

    March 5th, 2012 at 4:31 am

    This is the full list for those interested:

    Ambiguity effect – the tendency to avoid options for which missing information makes the probability seem “unknown.”[6]
    Anchoring – the tendency to rely too heavily, or “anchor,” on a past reference or on one trait or piece of information when making decisions (also called “insufficient adjustment”).
    Attentional Bias – the tendency of emotionally dominant stimuli in one’s environment to preferentially draw and hold attention and to neglect relevant data when making judgments of a correlation or association.
    Availability heuristic – estimating what is more likely by what is more available in memory, which is biased toward vivid, unusual, or emotionally charged examples.
    Availability cascade – a self-reinforcing process in which a collective belief gains more and more plausibility through its increasing repetition in public discourse (or “repeat something long enough and it will become true”).
    Backfire effect – when people react to disconfirming evidence by strengthening their beliefs[7]
    Bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same. Related to groupthink and herd behavior.
    Base rate neglect or Base rate fallacy – the tendency to base judgments on specifics, ignoring general statistical information.[8]
    Belief bias – an effect where someone’s evaluation of the logical strength of an argument is biased by the believability of the conclusion.[9]
    Bias blind spot – the tendency to see oneself as less biased than other people, or to be able to identify more cognitive biases in others than in oneself.[10]
    Choice-supportive bias – the tendency to remember one’s choices as better than they actually were.[11]
    Clustering illusion – the tendency to under-expect runs, streaks or clusters in small samples of random data
    Confirmation bias – the tendency to search for or interpret information in a way that confirms one’s preconceptions.[12]
    Congruence bias – the tendency to test hypotheses exclusively through direct testing, in contrast to tests of possible alternative hypotheses.
    Conjunction fallacy – the tendency to assume that specific conditions are more probable than general ones.[13]
    Conservatism or Regressive Bias – tendency to underestimate high values and high likelihoods/probabilities/frequencies and overestimate low ones. Based on the observed evidence, estimates are not extreme enough[14][15][5]
    Contrast effect – the enhancement or diminishing of a weight or other measurement when compared with a recently observed contrasting object.[16]
    Denomination effect – the tendency to spend more money when it is denominated in small amounts (e.g. coins) rather than large amounts (e.g. bills).[17]
    Distinction bias – the tendency to view two options as more dissimilar when evaluating them simultaneously than when evaluating them separately.[18]
    Empathy gap – the tendency to underestimate the influence or strength of feelings, in either oneself or others.
    Endowment effect – the fact that people often demand much more to give up an object than they would be willing to pay to acquire it.[19]
    Essentialism – categorizing people and things according to their essential nature, in spite of variations.[20]
    Exaggerated expectation – based on the estimates, real-world evidence turns out to be less extreme than our expectations (conditionally inverse of the conservatism bias).[21][5]
    Experimenter’s or Expectation bias – the tendency for experimenters to believe, certify, and publish data that agree with their expectations for the outcome of an experiment, and to disbelieve, discard, or downgrade the corresponding weightings for data that appear to conflict with those expectations.[22]
    Focusing effect – the tendency to place too much importance on one aspect of an event; causes error in accurately predicting the utility of a future outcome.[23]
    Forward Bias – the tendency to create models based on past data which are validated only against that past data.[citation needed]
    Framing effect – drawing different conclusions from the same information, depending on how that information is presented.
    Frequency illusion – the illusion in which a word, a name or other thing that has recently come to one’s attention suddenly appears “everywhere” with improbable frequency (see also recency illusion). Sometimes called “The Baader-Meinhof phenomenon”.
    Gambler’s fallacy – the tendency to think that future probabilities are altered by past events, when in reality they are unchanged. Results from an erroneous conceptualization of the Law of large numbers. For example, “I’ve flipped heads with this coin five times consecutively, so the chance of tails coming out on the sixth flip is much greater than heads.”
    Hard-easy effect – Based on a specific level of task difficulty, the confidence in judgments is too conservative and not extreme enough[24][25][26][5]
    Hindsight bias – sometimes called the “I-knew-it-all-along” effect, the tendency to see past events as being predictable[27] at the time those events happened.(sometimes phrased as “Hindsight is 20/20″)
    Hostile media effect – the tendency to see a media report as being biased due to one’s own strong partisan views.
    Hyperbolic discounting – the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, where the tendency increases the closer to the present both payoffs are.[28]
    Illusion of control – the tendency to overestimate one’s degree of influence over other external events.[29]
    Illusion of validity – when consistent but predictively weak data leads to confident predictions
    Illusory correlation – inaccurately perceiving a relationship between two unrelated events.[30][31]
    Impact bias – the tendency to overestimate the length or the intensity of the impact of future feeling states.[32]
    Information bias – the tendency to seek information even when it cannot affect action.[33]
    Insensitivity to sample size – the tendency to under-expect variation in small samples
    Irrational escalation – the phenomenon where people justify increased investment in a decision, based on the cumulative prior investment, despite new evidence suggesting that the decision was probably wrong.
    Just-world hypothesis – the tendency for people to want to believe that the world is fundamentally just, causing them to rationalize an otherwise inexplicable injustice as deserved by the victim(s).
    Knowledge bias – the tendency of people to choose the option they know best rather than the best option.[citation needed]
    Loss aversion – “the disutility of giving up an object is greater than the utility associated with acquiring it”.[34] (see also Sunk cost effects and endowment effect).
    Mere exposure effect – the tendency to express undue liking for things merely because of familiarity with them.[35]
    Money illusion – the tendency to concentrate on the nominal (face value) of money rather than its value in terms of purchasing power.[36]
    Moral credential effect – the tendency of a track record of non-prejudice to increase subsequent prejudice.
    Negativity bias – the tendency to pay more attention and give more weight to negative than positive experiences or other kinds of information.
    Neglect of probability – the tendency to completely disregard probability when making a decision under uncertainty.[37]
    Normalcy bias – the refusal to plan for, or react to, a disaster which has never happened before.
    Observer-expectancy effect – when a researcher expects a given result and therefore unconsciously manipulates an experiment or misinterprets data in order to find it (see also subject-expectancy effect).
    Omission bias – the tendency to judge harmful actions as worse, or less moral, than equally harmful omissions (inactions).[38]
    Optimism bias – the tendency to be over-optimistic, overestimating favorable and pleasing outcomes (see also wishful thinking, valence effect, positive outcome bias).[39][40]
    Ostrich effect – ignoring an obvious (negative) situation.
    Outcome bias – the tendency to judge a decision by its eventual outcome instead of based on the quality of the decision at the time it was made.
    Overconfidence effect – excessive confidence in one’s own answers to questions. For example, for certain types of questions, answers that people rate as “99% certain” turn out to be wrong 40% of the time.[41][42][43][5]
    Pareidolia – a vague and random stimulus (often an image or sound) is perceived as significant, e.g., seeing images of animals or faces in clouds, the man in the moon, and hearing hidden messages on records played in reverse.
    Pessimism bias – the tendency for some people, especially those suffering from depression, to overestimate the likelihood of negative things happening to them.
    Planning fallacy – the tendency to underestimate task-completion times.[32]
    Post-purchase rationalization – the tendency to persuade oneself through rational argument that a purchase was a good value.
    Pro-innovation bias – the tendency to reflect a personal bias towards an invention/innovation, while often failing to identify limitations and weaknesses or address the possibility of failure.
    Pseudocertainty effect – the tendency to make risk-averse choices if the expected outcome is positive, but make risk-seeking choices to avoid negative outcomes.[44]
    Reactance – the urge to do the opposite of what someone wants you to do out of a need to resist a perceived attempt to constrain your freedom of choice.
    Recency bias – a cognitive bias that results from disproportionate salience of recent stimuli or observations – the tendency to weigh recent events more than earlier events (see also peak-end rule).
    Recency illusion – the illusion that a phenomenon, typically a word or language usage, that one has just begun to notice is a recent innovation (see also frequency illusion).
    Regressive Bayesian likelihood – estimates of conditional probabilities are conservative and not extreme enough[45][46][5]
    Restraint bias – the tendency to overestimate one’s ability to show restraint in the face of temptation.
    Selective perception – the tendency for expectations to affect perception.
    Semmelweis reflex – the tendency to reject new evidence that contradicts a paradigm.[47]
    Social comparison bias – the tendency, when making hiring decisions, to favour potential candidates who don’t compete with one’s own particular strengths.[48]
    Status quo bias – the tendency to like things to stay relatively the same (see also loss aversion, endowment effect, and system justification).[49][50]
    Stereotyping – expecting a member of a group to have certain characteristics without having actual information about that individual.
    Subadditivity effect – the tendency to estimate that the likelihood of an event is less than the sum of its (more than two) mutually exclusive components.[51]
    Subjective validation – perception that something is true if a subject’s belief demands it to be true. Also assigns perceived connections between coincidences.
    Unit bias – the tendency to want to finish a given unit of a task or an item. Strong effects on the consumption of food in particular.[52]
    Well travelled road effect – underestimation of the duration taken to traverse oft-traveled routes and over-estimate the duration taken to traverse less familiar routes.
    Zero-risk bias – preference for reducing a small risk to zero over a greater reduction in a larger risk.
    [edit]Social biases

Leave a Reply

*