Cheap, Fast, And Good?
By Ron Sellers
There’s an old adage in research: Cheap, Fast, or Good…you can have any two of those three, but not all three.
A recent comment from a client-side researcher posted on Greenbook’s Insight Innovation Challenge made me think of this. The problem as posted was this:
“A quick survey should take no longer than 2 weeks to plan, field and report if it is a simple concept test, or is limited to an area of questions. However, if you have more than one or two questions and want insightful results, most vendors need at least 4 to 6 weeks or longer to get through their process. My counterparts in the Analytics area can deliver the results of their analysis within hours. I’m at a disadvantage with my set of tools and resources. Who can speed up the survey process?”
The comparison is Analytics and survey research isn’t entirely fair, because Analytics already has the data, whereas primary research needs to gather the data from scratch. Of course this will take longer.
But at the same time, this challenge made me think of a project I handled about three years ago. A client needed a full quantitative study done, and they had less than two weeks. That’s it. Not even a full two weeks. And although it’s not an experience I want to relive on a consistent basis, I made it happen for them. Over 600 completed interviews, including three open-ended questions, at 19% incidence, and it went from initial proposal to data and report in under two weeks. So yes, it is possible.
But it’s not common; in fact, judging from this question, from other questions I’ve seen, and from the reaction of my client (including the comment, “None of our other vendors could have made this happen”), it seems it’s pretty rare. Is it truly reasonable for clients to get cheap, fast, and good? And if so, how?
First, hold up your end. I’m amazed how often clients want something fast, but then they can’t do what’s necessary on their own end to make it happen. They want to sit down with their own customers in focus groups next week – but then tell me it will take two weeks for them to pull the list. They want the research to start right away – but then can’t get Purchasing to approve the contract for six weeks. They want to see the questionnaire in three days – but then it has to be reviewed by eight different people over the course of a month. And yes, I’ve had all of these things happen recently on projects.
I even had one client that would not pay for travel unless I booked it through their corporate travel department. Their travel department would not even discuss the travel unless I had an active purchase order number. Purchase order numbers were usually issued after the project was completed. Do you see how this might play just a tiny bit of havoc with the schedule on a rush job?
Second, cut way back on the bureaucracy. If you want me to complete a project for you in a couple of weeks, that means I might have one day to design the questionnaire. Or half a day. But it also means you need to be prepared to review it and approve it right away. If I can design a questionnaire from scratch in five hours, you can give me feedback in two. One would be better.
This probably also means you are the only one in your organization who gets to review it (or the methodology, the sampling plan, the tab plan, or anything else that needs approval). If you want research projects completed fast, be prepared to have one decision maker who is readily available and will make those decisions quickly and confidently.
Next, know what you want and communicate it clearly. A rush project means we don’t have the time to go back and forth discussing research objectives. It means you come to me with your objectives ready to go, and that we stick with those objectives. Unless you have a clear view of what you want, I cannot have a clear view of how to get it to you.
Let me immediately create a plan of action. Everyone needs to know what is expected of them, and when. While the client is reviewing the questionnaire, I can be arranging the programming with the field vendor. While the survey is in the field, I can be starting on the weighting plan and the tab plan. For this two-week project, I created a detailed timeline for the field vendor, so they knew exactly how much time they had to get me the initial programming to check, exactly when to expect my comments, exactly when it needed to go into the field and be back out, exactly when I needed the SPSS file, etc.
Once we’ve agreed on a plan of action, stick to it. If you want a project in the field in three days, you can’t review the questionnaire draft and then decide you want to add some objectives. We can’t go into the field only to have you change the sampling requirements. Any changes will add to the time it takes to do the project. As the old quote goes, if you don’t have time to do it right, where will you find the time to do it over?
Simplify, simplify, simplify. If you need the research really fast, be very aware that we don’t have time for unnecessary complexity. I’m probably not going to be able to get from Pensacola to Sacramento without a full travel day, but I can get from Charlotte to Boston in the morning and then moderate that night. I probably can’t design, field, tab, and analyze a twenty-minute questionnaire with five open ends in a week, but I may be able to do one that’s ten minutes long with one open end. We might have to pare down the sample size from 2,000 to 600, or from ten respondents per focus group to six.
When people try to set land speed records on the Bonneville Salt Flats, they don’t drive vehicles loaded down with luggage racks, chrome rims, and a 200-watt stereo. They strip it down to what’s absolutely necessary, because all the other stuff will slow them down. In research, all the “other stuff” might be nice if we had nine weeks, but not if we have nine days.
Weigh the trade-offs between speed and quality. The need for speed means we may have to sacrifice some things. Yes, you may prefer a phone survey for this work, but I guarantee you an online survey will be faster. Yes, we’d like the requisite three attempted contacts per record in the field, but we may be in the field for one or two nights and that’s it. Yes, multiple regression would provide great feedback on this, but we only have time for bivariate analysis. Yes, you’d like three open-ended questions but that doesn’t mean 1,800 responses are going to get coded in a day.
You never want to do things wrong in the name of speed, but you may be forced into trade-offs. If you want to do the study by phone rather than online, are you willing to spend four more days in the field? If you want three attempts per record, are you willing to spend another two or three days dialing? If you want a bunch of fancy programming in the online survey, such as drag-and-drops and whatnot, are you willing to give the programmers an extra day to accomplish that?
Be prepared to pay a bit more. On a real rush project, I’m going to be working nights and weekends in order to get you what you need, and asking others to do the same thing. I may have to put off other work, and after your project is done spend more nights and weekends catching up on that work.
On the two-week project I mentioned, I spent one entire weekend putting together the final report, and that came on top of several late nights handling other aspects of the project. The field vendor had to postpone a couple of other projects in order to get that one programmed and into the field immediately. Yes, I do think it is reasonable that the vendors be compensated for that.
In addition, we may have to take some steps that cost more. For example, on any survey using an online panel I know we’re going to toss out some responses for speeding, straightlining, or giving responses such as “kjkjkjkjkjk” to open-ended questions. Normally, responses are screened as they come in, and we toss the bad ones out along the way. On this project, we were in the field for 24 hours. That meant I had to arrange for 675 completes, figuring I might have to toss out about 10%. The client had to pay for a large sample size because we needed speed. I also had to pay a rush charge for the panel vendor to program the questionnaire in less than a day.
I’m willing to go the extra mile for you. Don’t then turn around and try to get me to shave a thousand dollars off the final bill so you can impress your boss with your fiscal parsimony. And don’t pay my invoice in ten weeks when I just moved heaven and earth to get you a report in ten days.
Work to create a partnership with your vendor. Why did I take on this difficult request? Because it came from a client with whom I have built a culture of trust. They don’t make a habit of asking for unreasonable things. They don’t constantly complain about pricing, or send projects to another vendor because my bid was 1% higher. They understand I don’t want to moderate four focus groups in a day with no break. They don’t tell me something is a rush and then take two weeks to get back to me. They actually pay my invoices on time. They are effusive with their praise and recommendations of my work.
In short, they trusted me to get this done, and I trusted them to make it possible for me to get this done. Not only that, I wanted to make this happen for them because of the relationship we’ve built. A good vendor-client relationship is a partnership with mutual respect and trust. This client got something they desperately needed because we have a good partnership.
Know your vendors well enough to know who can pull this off. If you don’t have a vendor that can do something like this, start looking for one that can – well in advance of your next rush project – and work to build a partnership with them so that everything will be ready the next time you have a research emergency.
There’s nothing wrong with a vendor who can’t do something like this, or who doesn’t want to, but it’s certainly nice to have one who can if this sort of need comes up for you at least occasionally. In addition, it will probably depend on the vendor’s schedule when your need arises; had I been on a cross-country moderating tour when this need arose, there’s no way I could have pulled this one off.
Finally, have a real need, not one you created. I’ll make things happen for you if you have a real need. If you have known you need research for a month and you waited until the last minute to call me because you were “just so darned busy,” my desire to turn my world upside down in order to serve your needs declines by a significant amount. It’s only human nature. It’s also part of the trust factor I noted earlier. And yes, I have been asked “Can we do focus groups tomorrow?” by someone who had been working on his project for a couple of months.
So let’s take a quick look at the actual timeline on the project I mentioned earlier, and how the things I’ve written above worked in the real world:
· Thursday, September 10: client approaches me with the project and the timeline; I agree to make it happen (work to create a partner with your vendor).
· Friday, September 11: I bid the fieldwork out to field vendors and note I need their bid in a couple of hours (my vendors will get this to me because I’ve worked hard to create a partnership of mutual trust with them).
· Friday, September 11: the client provides a clear set of objectives to me and we move forward with the project (know what you want and communicate it clearly).
· Saturday, September 12: I design a draft of the questionnaire.
· Sunday, September 13: I finalize a draft of the questionnaire and provide it to the client so they can have it first thing Monday morning. The draft excludes a couple of complex issues we simply couldn’t address because of the time frame for the study (simplify, simplify, simplify).
· Wednesday, September 16: the client provides feedback on the questionnaire – two days later than agreed upon because too many people had to review it, which meant the entire timeline had to be recreated to account for the delay (cut way back on the bureaucracy).
· Thursday, September 17: the questionnaire is finalized first thing in the morning and sent to programming. The field vendor programs it, I double check all the programming, and it enters the field (immediately create a plan of action).
· Friday, September 18: I get a partial SPSS file with about 100 completes from Thursday, so I can start to develop the coding frames, the weighting scheme, and the programming for tabs. Once the coding frames are finished, coding is completed as completes roll in (there’s that plan of action again).
· Saturday and Sunday, September 19 and 20: The field is completed, even though I normally spend a week in the field even with an online study (weigh the trade-offs between speed and quality), and we got 675 completed interviews so we could clean out bad interviews for a final sample size of around 600 (be prepared to pay a bit more). Also, I build a framework for the report.
· Monday, September 21: The final SPSS file is loaded into the programming that was previously completed. The data is cleaned, weighting finalized, tabs run, and a topline draft of the final report (20 key PowerPoint slides) is completed and sent to the client Monday night so they have it first thing Tuesday, seven hours before their 3:00 p.m. deadline (know your vendors well enough to know who can pull this off).
A full, comprehensive report of 80 slides was completed in the next few days, but the client had the tabs, the key findings, and the full data set they could explore themselves eight business days after the first call – in time to make the critical decisions.
By the way, the client never argued with the pricing. The initial invoice was paid in three days; the final invoice in seven days. And the client came back a few months later with a similar project (but this time with about three weeks to make it happen). We’ve worked on a number of other projects since then.
So that’s it – initial request September 10, data and comprehensive topline report September 21 – eleven calendar days later, including only eight “work” days (and taking into account an unfortunate two-day delay by the client). So yes, it most definitely can happen. As the client, the key is to find a vendor who can make this sort of thing happen when you really need it, and create a situation in which that vendor has what they need from you to succeed for you.