top of page

Problems with Sample Surveys

Chapter 4 - Day 3 - Lesson 4.1

Learning Targets
  • Explain how undercoverage, nonresponse, question wording, and other aspects of a sample survey can lead to bias.

Activity: What is Wrong with These Surveys? 
Answer Key:

The context for each one of the examples in this Activity are very accessible and students should have some good discussion.  During the Activity, it is not important that students use the correct vocabulary (undercoverage, nonresponse, response bias).  It is more important that they are thinking about each of these ideas and then trying to communicate their understanding.  Remember…”Experience First, Formalize Later”.  We took care of the “formal” language in the debrief. 

Voluntary Response or Nonresponse

Students will often confuse the concepts of nonresponse and voluntary response.  Consider the following two scenarios:

  1. The New York Times asks for people to call in for a survey.  400 people call in with responses.

  2. The New York Times randomly selected 1000 households with landlines in the United States.  They call all of these households and 600 people hang up on them and  400 respond.

Scenario (1) is clearly voluntary response.  Some students will argue that Scenario (2) is also voluntary response because the 400 people volunteered their answer.  It is not about if a person “volunteers” their answer, it is about if the person “volunteers” to be part of the  sample.  In the first scenario, the people decided to become a part of the sample.  In the second scenario New York Times decided who was going to be in the sample.  So the issue in Scenario (2) is nonresponse.

AP Exam Tips

Many students lose credit when describing what can go wrong in sample surveys because they use incorrect terminology. While it is important for students to understand and use the vocabulary of statistics correctly, they are rarely required to use specific vocabulary in their responses on the AP exam. To be safe, tell your students to not worry about naming a specific problem or source of bias. Instead, have them clearly describe the problem and its consequences in the context of the question.

If you’re asked to describe how issues with the collection of survey data lead to bias, you’re expected to address two ideas: (1) describe how the members of the sample might respond differently than the rest of the population, and (2) explain how this difference would lead to an underestimate or overestimate.

Final note: Josh Tabor has a great Response Bias Project (with rubric) that can be done with students.

bottom of page