Where Does the Buck Stop Again?

When the field center tells me they’re comfortable that the questionnaire is 15 minutes long, and in the field it’s actually 23 minutes, who should be responsible for the inaccuracy of their estimate (or the unexpected length of the actual questionnaire)? I would love to hear your take on this, because I honestly don’t know.

The Buck Stops Here

  • By Ron Sellers

    Who is responsible for estimating questionnaire length?

    That’s not a rhetorical question. I really want to know.

    Let’s say I give your field center a questionnaire that I’ve estimated at 15 minutes in length, and you’ve bid at 15 minutes. You review it, and agree with my estimate that its length is about 15 minutes.

    We all seem to be in agreement. The client has her budget and since we’ve matched the bid length, I’m able to stick to it. Everything seems to be fine.

    Next, you put it into the field and after a couple of days, you come back and tell me the actual length is 23 minutes and I owe you 30% more. Budget busted. Client unhappy.

    Who is at fault for this? Who should bear the burden of the added expense?

    The client’s response goes something like this: “We hired you because of your research expertise. After many years in the business, you should be able to know about how long a questionnaire is. I don’t have the budget for a 30% increase in costs, and my internal client is going to blow a gasket if I tell him we have to cut the sample size down. I’m really not happy.”

    My message to the field center goes something like this: “You’ve been in business long enough that you should be pretty good at estimating questionnaire length. There are not a lot of skip patterns that should impact the length. This is why I gave you the questionnaire in advance and asked for your take on the length – so we could cut things out before we went into the field if this was necessary. My client can’t take on a 30% increase and I’m certainly not going to eat the difference and lose money on this project. I’m really not happy.”

    Now, the field center has two options. One is, “You’re right, we messed up the estimate pretty badly. A couple minutes off is to be expected, but 50% longer than our estimate? We have enough experience with questionnaire length that something like that shouldn’t happen. The additional costs are our responsibility.”

    The other is, “That’s why it’s called an estimate. We have no control over how long your questionnaire is. I’m sorry our estimate was way too short, but you need to cut sample size, cut questions (and we’ll charge you a reprogramming fee for that), or pay us the 30% more we require to complete this.”

    When the field center tells me they’re comfortable that the questionnaire is 15 minutes long, and in the field it’s actually 23 minutes, who should be responsible for the inaccuracy of their estimate (or the unexpected length of the actual questionnaire)? I would love to hear your take on this, because I honestly don’t know.

    Share
    You can leave a response, or trackback from your own site.

    10 Responses to “Where Does the Buck Stop Again?”

    1. Lance Hoffman says:

      March 29th, 2011 at 2:06 pm

      Well, you’re certainly starting along a path, aren’t you Ron? A good topic, though, and one that is certainly known to all. What’s next . . . ‘Incidence . . .fact or fiction?” From a field standpoint, I can tell you that the same conversations happen about incidence also. But for now, let’s focus on time tests . . . and for that matter, let’s break that down into its parts . . .TIME and TEST. What’s that second part? TEST. If this were an actual emergency . . . .

      Clearly the ones who truly incur the cost of labor for additional minutes for which a project was not priced is the field house – and that is neither fair NOR in the spirit of partnership, which everyone says they want, but few actually walk the walk. Most clients say that the have to give fixed-cost pricing on length and incidence – often a price they came up with before a vendor has seen anything that resembles a draft . . .so in that instance, should the vendor bare any of the additional cost? Our pricing is always subject to change based on “actual length and incidence once a project is in the field.” Any gross differences between what was estimated and what we actually experience are brought to the client’s attention immediately after the first full shift of dialing – which is early-enough to make changes to the instrument at that time, without having to incur any additional costs.

      I totally believe that SOME accountability might lie with the field house if the length is totally off AND they cannot tell you why. I know that we have certainly stepped up to the plate on the rare occasion that something had been amiss, and then worked diligently to correct the process so we can more accurately test the length of surveys. However, our clients cannot adopt a Teflon philosophy either. After all, they wrote the questionnaire. And if they hadn’t written the questionnaire, but allowed their client to, then the same argument applies to them: you should have known better. We rely on clients to realize the actual costs we incur, and help protect us from these occurrences, rather than have us act as their bank when something goes wrong. And subject matter can have a lot to do with adding length, for which no field house should be held responsible – especially if open ends are involved. Try time-testing a survey on abortion, with one person talking to another, as is how things are usually tested, and it comes in at 15 minutes. Now field that same survey in the Bible Belt – think it’s coming in at 15-minutes? Try 20+, having nothing to do with the survey or the testing procedure, per se, but the environment and the subject matter – neither of which is on the field house.

      In this business environment, it’s important that everyone be at the top of their game in estimating costs. We’re all being asked to be so competitively, that while previously there may have been some cushion to help in stances such as the one you describe, there is now no cushion to incur additional costs, and rest assured, a 15-minute questionnaire running at 23 minutes HAS ADDITIONAL COSTS, not just in the longer length of a complete, but also in all the people who can’t believe they aren’t finished after 18 minutes and decide to hang up – something for which no one gets paid.

    2. Laurie Koenig says:

      March 29th, 2011 at 2:21 pm

      I tend to have my studies fielded by smaller “mom and pop” shops to aviod the issue you are talking about. Having a personal relationship with your provider – which is not possible w/ the big data collectors – means that we can agree on the appropriate solution.

      In my experience they cut me some slack when the time estimate is a bit off and I know that from time to time and they make it up when its a bit shorter than we expected. Plus I’m willing to pay a little more CPI for that relationship. At the end of the day I think it works out better for my client and I to have a partnership with in the data collection firm.

    3. Jens Erickson says:

      March 29th, 2011 at 2:24 pm

      I don’t have too much experience on the responsibility of the LOI. However from the sample providers prospective, the responsibility is on our client. That is normally the researcher or the writer of the survey. By the time they come to us, the survey is usually written so the LOI is usually established or can be estimated pretty accurately based on the number of questions and the types of questions. We usually estimate 2-3 closed end questions per minute and 1-2 minutes per open end.

      I would imagine the researcher or question writer would be responsible for the LOI and the negotiations with the end client. The end client would ultimately pay the costs for the added length since it is usually based on additional information that they are looking for. Again this would be negotiated between the end client and the survey writer.

    4. Charles Shillingburg says:

      March 29th, 2011 at 2:43 pm

      In the scenario you lay out, I believe it is the responsibility of the Field Center to accurately determine the length of the questionnaire. If there is doubt, then a pilot test should be run to determine the actual length.

      When it comes to cost though, there are other factors that often impact the cost more than the actual length of the questionnaire, such as the time it takes to find proper respondents (incidence estimations), especially as incidence goes down. Cooperation is another element that is often a problem that outweighs length. In other words, it is total time for the job that one is trying to estimate.

    5. Sharron Silva says:

      March 30th, 2011 at 7:44 am

      I see it as the vendor’s responsibility to clarify any issues that could impact cost as part of the price negotiation. So, if there is some uncertainty, for example, about how many screening calls will be required to find eligible respondents, then the vendor should talk with the client about contingency planning, including some early-warning system so there is time to move to Plan B if necessary.

      Sometimes, a vendor’s estimate depends on information that we give them.
      For example, they may develop their estimate based on our telling them that several questions in the middle of the survey will only apply to 20% of the sample. If it turns out that those questions apply to 40%, making the interview that much longer, then we would either pay the overage or cut some questions. Again, I would expect the vendor to raise this issue before fielding and monitor fielding to give us the most flexibility in deciding if we want to pay more for the complete interview or if we want to cut some questions.

    6. Sharron Silva says:

      March 30th, 2011 at 7:44 am

      I see it as the vendor’s responsibility to clarify any issues that could impact cost as part of the price negotiation. So, if there is some uncertainty, for example, about how many screening calls will be required to find eligible respondents, then the vendor should talk with the client about contingency planning, including some early-warning system so there is time to move to Plan B if necessary.

      Sometimes, a vendor’s estimate depends on information that we give them.
      For example, they may develop their estimate based on our telling them that several questions in the middle of the survey will only apply to 20% of the sample. If it turns out that those questions apply to 40%, making the interview that much longer, then we would either pay the overage or cut some questions. Again, I would expect the vendor to raise this issue before fielding and monitor fielding to give us the most flexibility in deciding if we want to pay more for the complete interview or if we want to cut some questions.

    7. Gloria Barotta says:

      March 30th, 2011 at 10:04 am

      I’m old school. We always pretested questionnaire we thought had the potential to run long, especially when the budget was tight. I always gave estimates for a few minutes under and over as well, that way there were few surprises. I’m in agreement with Charles. Incidence often has a bigger impact and THAT is the client’s responsibility. Cooperation rates are the field service’s responsibility. In the end, I believe the consultant is responsible pricing/estimating studies to cover slight increases in questionnaire length.

    8. Kelle Rupp says:

      March 30th, 2011 at 12:12 pm

      Here at Pacific we believe that responsibility lies with both the client and us to communicate effectively to address specification issues, such as the length of survey, early. However, most often the length of a questionnaire cannot be truly determined until the study goes into field. We can provide input as to what length the document appears to be given the number and type of question and logic or even time the survey internally prior to fielding, but until we interact with respondents, it can be difficult to determine the survey’s actual length. That is why a pre-test proves valuable prior to actual fielding to determine the true length. At that point, the client is equipped to decide if there is a need to re-word or possibly cut questions so that there are no cost implications. Additionally, we also find it useful to estimate costs on a couple of lengths at and around the stated length to give a bigger picture to the client of potential cost implications at different survey lengths.

    9. Becky Quarles says:

      March 30th, 2011 at 4:37 pm

      Often, there are too many variables – including skip patterns, awareness levels, mix of languages, topic (e.g., Lance’s example of abortion in the South) to estimate length accurately. So we have a policy of conducting a pretest with a limited number of interviews and, then,cutting – if necessary – the next day. If we suspect that the interview will run long, we often ask clients to identify low priority questions that we can cut if necessary, or we make recommendations to the client about what to cut. This works most of the time, but not always. We have had a few clients doubt the reports from the field company, saying “the interview can’t take that long. I administered it to someone in my office and it didn’t take nearly that long.” But luckily, this is the exception not the rule.

    10. Tim Grainey says:

      April 1st, 2011 at 10:59 am

      I got into this business as a telephone interviewer while in college. That experience helps me some in estimating the interview length but there still can be variability. I use the field service’s estimate on length in my proposal to the client and use other qualifications (percentage of respondents that we expect to respond to open-ends and other mentions, along with incidence, cooperation and even productivity estimates) so that if the fieldwork costs do increase, I have some basis to explain why the costs have risen and why. I will let them know as early as possible that the average completion length is skewing high. Sometimes clients accept that and pay the difference and other times not. If they don’t, I usually have to cover the excess costs.

    Leave a Reply

    *

    %d bloggers like this: