How It's Done: Constructing a Successful Survey
Sample Population
If you don't talk to the right people, your findings will be misleading--no matter how beautifully you analyze the data.
Composition
The composition of a project has to reflect your overall strategy. If you want to extend the reach of your product, you need to sample outside of your customer base. If you want to monitor customer satisfaction with your offering, make sure that you only include members of your franchise in your sample.
In addition, it helps if your sampling strategy stays constant from one study to another. For example, you will want to insure that tracking study sample descriptions are comparable from wave to wave, and you will want your attitude and usage (A&U) samples to match the samples of your copy tests. In this way, you can compare and combine findings from several studies because they are directed at the same sample. Things get trickier, of course, when you change your strategy. For example, suppose you go from an "increase usage occasions" strategy to a "competitive advantage" strategy. In such a case, you will need to increase sample size to include sample quotas relevant to both the old and new strategies.
Sample Size
In theory, the size of project sample is a function of what you plan to do with the information. In practice, the sample size for a survey will have more to do with the potential cost of the study then with any of the kinds of statistical procedures that you learned in grad school.
One way of deciding on the appropriate sample size for a project is to predict the accuracy with which the findings will predict the "true" value in the relevant population - the "plus or minus X percent" that defines our confidence in the results of a study. You can make this determination by using the equation e = +/- (0.5)2/N. Of course, the actual degree to which any reported information will be projectable to a relevant universe will be a function of the variance present in the data. And to determine this, you will need to have some data.
No matter how many people you include in the total sample, try to insure that there are sufficient numbers to evaluate any sub group that your want to examine. This is a decision that you need to make up front in designing the study. If you want to focus on the opinions of, say, heavy users, make sure that you sample with that in mind.
As a rule of thumb, include at least 100 people of any subgroup that you may want to examine. For example: if you want to understand the perceptions of both men and women, you probably want to include at least 200 people in your sample.
Questionnaire Items
One of the most frequently made mistakes involves the wording of the items on the questionnaire. In order to craft questions that will elicit meaningful responses, one must ensure that:
-
Questions are clearly understandable by the respondents
Questions phrased in marketing lingo have little or no reference to consumers. The answers to such questions are likely to be misleading. Only questions clearly phrased can elicit clear judgments - and thus meaningful responses. -
You ask only one thing with each of your questions
Sloppy question writing will not only confuse respondents, but it will also make it impossible to effectively analyze your findings. You won't know which part of the question respondents were answering.
Length
Keep it short. The people you want to talk to rarely have much time to spend on your questionnaire. By this, I mean that while the occasional lonely individual may be willing to spend 45 minutes exploring the minutiae of long-distance telephone subscription, average Americans - the kind of sample you need to support a mainstream brand - can only spare a little time for your questions.
Listening in to a few telephone interviews illustrates that people have limited time to offer feedback on surveys. The respondents start off willingly enough, but as the interview continues on, they start to try to beg off - asking how much longer the interview will take. Finally some of them grow desperate enough that they begin to give the same response for each question (e.g., "Give them all eights") in an attempt to get off the phone.
Such a situation does two things, both of them bad:
- First, it gives you meaningless data - negating the whole point of the study and providing useless or even misleading information.
- Second, it alienates respondents and makes them less likely to consent to take part in a future survey. This hurts everyone in the consumer research community.
So why do people field long unwieldy surveys, you ask? There are a number of reasons.
All too often, questionnaire writing becomes an exercise in satisfying the curiosity of every marketing group and senior executive in a marketing organization. You send out a short, sensible questionnaire for everyone to review and you get back a tome.
In addition, members of your product group will inevitably decide that this is their chance to conduct a single piece of research that will answer all of their questions. And so they insist that you ask everything. This is harmful both because it perilously increases the length of interviews and it tends to destroy the flow of an interview. These realities will put you in a difficult spot. You will want to satisfy your co-workers or internal clients, but you want to get usable information or the project will be useless. Be polite, but firm.
Flow
The flow of topics in a questionnaire is very important. The best questionnaires "make sense" to respondents - with each question building on the previous questions. As such, when writing a questionnaire, you need to be very cognizant of the order in which you ask questions. You need to know what you have already asked and when you asked it, in order to avoid influencing responses to later questions with earlier items. Simply put: You can't ask a question about a specific brand before you ask which category brands the respondent is aware of.
A good questionnaire will begin with general, unaided questions about the target category (e.g., "What brands come to name when you think of dish washing liquid?") The questionnaire should then move to increasingly specific questions - finishing with very specific aided questions (e.g.; "Rate Sudzo in terms of its cleaning capabilities.").
Source
"How It's Done: A Research and Planning Handbook For Marketing Communications Professionals." Seth Ginsburg. New York: ANA, 2005.