Who Are Those Angry Women?
A well-conducted survey can be very informative, but a poorly conducted one can be a complete disaster. As an extreme example, Moore (1997, p. 11) reports that for her highly publicized book Women and Love, Shere Hite (1987) sent questionnaires to 100,000 women asking about love, sex, and relationships. Only 4.5% of the women responded, and Hite used those responses to write her book. As Moore notes, "The women who responded were fed up with men and eager to fight them.
For example, 91% of those who were divorced said that they had initiated the divorce. The anger of women toward men became the theme of the book." Do you think that women who were angry with men would be likely to answer questions about love relationships in the same way as the general population of women? The Hite sample exemplifies one of the most common problems with surveys: The sample data may not represent the population. Extensive nonparticipation (nonresponse) from a random sample, or the use of a self-selected (i.e., all volunteer) sample, will probably produce biased results. Those who voluntarily respond to surveys tend to care about the issue and therefore have stronger and different opinions than those who do not respond.
Moral of the Story: An unrepresentative sample, even a large one, tells you almost nothing about the population.
Definitions: Nonparticipation bias (also called nonresponse bias) can occur when many people who are selected for the sample either do not respond at all or do not respond to some of the key survey questions. This may occur even when an appropriate random sample is selected and contacted. The survey is then based on a non representative sample, usually those who feel strongly about the issues. Some surveys don't even attempt to contact a random sample but instead ask anyone who wishes to respond to do so. Magazines, television stations, and Internet websites routinely conduct this kind of poll, and those who respond are called a self-selected sample or a volunteer sample. In most cases, this kind of sample tells you nothing about the larger population at all; it tells you only about those who responded.