Timing Is Everything: Specific Time Frames in Question Design
By Ron Sellers
Say you want to gauge how often people wash their cars. Or drink orange juice, or go to a movie theater, or whatever. There are two primary ways you can ask this if you’re trying to measure times per month (for instance):
- “In a typical month, how many times (if any) do you wash your car or have it washed?”
- “In the last 30 days, how many times (if any) did you wash your car or have it washed?”
Same difference, right? Pretty interchangeable? Not so fast. Are these two questions really asking the same thing?
For one thing, researchers too often are not realistic when asking people to frame their activities within specific periods of time. If the survey is being completed on August 18, will the respondent really remember whether she had her car washed on July 17 or July 19? Will she really remember exactly how many times that car has been washed in exactly the last 30 days?
The problem grows as the time period grows. I can tell you exactly how many glasses of milk I drank in the last 24 hours. Maybe in the last 72 hours. In the last week? I can give you an estimate. In the last month? Now I’m down to pure guesswork.
I frequently answer questionnaires demanding that I provide information I have no hope of remembering. How many times did I rent a car over the last 12 months? What did I pay when I stayed at the Hilton three months ago (and I’ve stayed at eight other hotels since)?
But aside from that issue, there are some activities that tend to vary from one time period to the next. People may watch more TV news as elections near, or during football season. Church attendance is particularly heavy near Christmas and Easter. Residents of Phoenix may bicycle frequently during our nine months of nice weather, but few are insane enough to break out the ten-speed when it’s 116° in August. I’m guessing car washing patterns in Minneapolis are not the same in July and January.
Not only are some activities seasonal, but events may mean a respondent who normally does an activity frequently was not able to do that activity recently. I was reminded of this last year when I completed a questionnaire about my media use. I normally read the local newspaper every morning, but I had just spent three weeks in Portugal, and my Portuguese isn’t good enough to read the local paper. So when asked how many times I read a local newspaper in the past 30 days, I had to answer that it was nearly zero – not at all an accurate reflection of my normal media habits.
These disruptions of normal activities may be more common than we realize: people have to deal with new births, divorces, family members getting sick or dying, travel, health problems, job losses, moving, family visiting from out of town, severe weather, and any number of other issues that may disrupt their typical activities for a period of time. I watch a lot less TV, and spend a lot more time reading, when I’m on the road than when I’m at home, and I’m on the road a lot.
Asking about a typical 30 days (or week, or year) means the respondent is telling you what his life is usually like, regardless of whether it was like that in the last 30 days. But it’s also probably more of an estimate. If I wash my car once a week in the summer, and once a month in the winter, there is no “typical” for me, so now I’m being forced to do math and average it out (which may or may not be accurate).
Asking about the last 30 days may bring a more realistic number, but that number may vary by when the questionnaire was completed, and you may be getting a very unrealistic number because of this. On socially desirable activities, someone may even falsify the number, under the reasoning that “I usually give to charity every month, even if I didn’t last month, so I should say I gave last month ‘cause that’s what I usually do.”
Neither of these approaches is necessarily the “right” answer. But as with everything else in research, it’s really critical to understand what you are getting by asking your question in a specific way. Every word of every question requires thought, and there are no “standard” questions that can just be used over and over in every situation. Understanding that, and the impact it may have on the resulting data, is simply part of doing the research properly.