Can This Survey Be Saved?
"What's troubling me is that you can't just pick a new random sample just because somebody didn't like the results of the first survey. Please tell me more about what's been done." Your voice is clear and steady, trying to discover what actually happened and, you hope, to identify some useful information without the additional expense of a new survey.
"It's not that we didn't like the results of the first survey," responded R.L. Steegmans, "it's that only 54% of the membership responded. We hadn't even looked at their planned spending when the decision [to sample again] was made. Since we had (naively) planned on receiving answers from nearly all of the 400 people initially selected, we chose 200 more at random and surveyed them also.
That's the second sample." At this point, sensing that there's more to the story, you simply respond "Uh huh. ..." Sure enough, more follows: "Then E. S. Eldredge had this great idea of following up on those who didn't respond. We sent them another whole questionnaire, together with a crisp dollar and a letter telling them how important their responses are to the planning of the industry. Worked pretty well. Then, of course, we had to follow up the second sample as well."
"Let me see if I understand," you reply. "You have two samples: one of 400 people and one of 200. For each, you have the initial responses and follow-up responses. Is that it?" "Well, yes, but there was also the pilot study-12 people in offices downstairs and across the street. We'd kinda like to include them, average them, with the rest because we worked so hard on that at the start, and it seems a shame to throw them away. But all we really want is to know average spending to within about a hundred dollars."
At this point, you feel that you have enough of the background information to evaluate the situation and to either recommend an estimate or an additional survey. Here are additional details for the survey of the 8,391 overall membership in order to determine planned spending over the next quarter.
Discussion Questions
1. Do you agree that drawing a second sample was a good idea?
2. Do you agree that the follow-up mailings were a good idea?
3. How might you explain differences among averages in the results?
4. Are there useful results here? Which ones are useful? Are they sufficient or is further study needed?