top of page

Response Bias Project Makeover

Leigh Nataro teaches elementary statistics, math for business, and math for teaching at Moravian University in Bethlehem, PA. Leigh has been an AP Exam Reader and Table Leader and was on the AP Statistics Instructional Design Team, where she helped to tag items for the AP Classroom question bank. In addition to leading AP Statistics workshops, Leigh leads in-person and virtual Desmos workshops with Amplify. Leigh can be reached on Twitter at @mathteacher24.

A picture of a pizza is shown to a student and then the student is randomly assigned to answer one of two questions.

Question 1: Would you eat this pizza?

Question 2: Would you eat this vegan pizza?

Does telling a person that a food is vegan impact their response? This was the experiment created by a pair of my students for one of the most engaging projects done in AP Statistics - The Response Bias Project. Students create an experiment to see if they can purposefully bias the results of a question.

Does inserting the word “vegan” into this question while showing the exact same picture bias the results? Here are the results obtained by the students. What do you think?

In my original iteration of this project, students would create various colorful (and often glitter-laden) graphs to compare the results.

But how likely would it be to get these results by chance if no bias was present?

To answer this question, The Response Bias Project needed a makeover.

The Makeover: Include a Simulation

One of the most challenging AP Stats topics for students is estimating p-values from simulations. This is one of the reasons we do a simulation on the first day of class with playing cards called “Hiring Discrimination: It Just Won’t Fly”. (An online version of the scenario and simulation can be found here: Hiring Discrimination.) Several times during the first semester, we use playing cards to perform simulations and then interpret the results to determine if a given claim is convincing. Although we never formally call the results a p-value, examining the dotplots from simulated results sets the stage for tests of significance in the second half of the course. Making over this project with a simulation leads to more robust conclusions and reinforces the idea of a p-value based on simulations.

Originally my students used the free trial of Fathom to create their simulations. However, in recent years Fathom has not been supported on newer Mac operating systems. This led me to investigate using the Common Online Data Analysis Platform or CODAP. This is a free online tool for data analysis and it is specifically designed for use with students in grades 6-14. Students can save their work in google drive, but an account is not required to access CODAP.

To understand what students need to do to create their simulation, I share an instructional video related to one of the projects. Here is the video that goes with the vegan pizza project: CODAP for Response Bias Project and the CODAP file: Vegan Pizza Simulation.

Dotplot created in CODAP. Is it likely to get 22 Yes results or fewer by chance?

Based on their results from the simulation, the students determined if adding "vegan" created biased results or if it was possible to get results like what they saw in their experiment due to chance alone. Students display their work on posters that are hung around the room. (A sample of posters is included at the end of the blog.) Then, half the class stands by their posters and they present to the few students that are in front of them. This takes about five minutes. Then, the students move on to another poster and pair of presenters. Students get to give their presentation about three or four times to different small groups of their peers. There are no powerpoints, no notecards, less nervousness and stress for students and this gives them practice with more informal presentations they might need to give at some point in the future. Another benefit is students get to see multiple instances of simulated results which helps to lay the foundation of the concept of a p-value for future units in the course.

Why Is This Makeover Helpful?

Although students learn about some common topics from AP Statistics earlier in their math careers, simulation is a topic that is new and often challenging for many students. Creating the simulation in CODAP helps students to understand what each dot in the simulation represents and how the overall distribution shows more likely and less likely outcomes. Students also identify where the value of their statistic (the count of yes answers) falls on the dotplot. They are essentially answering the test of significance question, that is “assuming there was no bias, how many times did the observed outcome or a more extreme outcome occur by chance alone?” Reading about the results on their classmates' posters and hearing about it in multiple presentations solidifies the concept of using probabilities from simulations to draw conclusions.

The Concept of Simulations from the AP Statistics Course and Exam Description

  • Skill 3.A: Determine relative frequencies, proportions, or probabilities using simulation or calculations.

  • UNC.2.A.4 Simulation is a way to model random events, such that simulated outcomes closely match real-world outcomes. All possible outcomes are associated with a value to be determined by chance. Record the counts of simulated outcomes and the count total.

  • UNC.2.A.5 The relative frequency of an outcome or event in simulated or empirical data can be used to estimate the probability of that outcome or event.

Note: On the 2023 AP Statistics Operational Exam, determining a probability based on data from a simulation was part of free-response question 6.

To understand how students are expected to use simulations on the AP Exam, consider Free Response Question 5 from the 2022 AP exam. Students were asked to use results from a simulation to estimate a p-value from a dotplot. Understanding what each dot represented and that they needed to determine where the sample statistic of 5.66 fell relative to the dots on the plot were the two key concepts needed to calculate the correct p-value. On the dotplot only 3 of the 120 simulated differences in means was 5.66 or higher. Students then needed to also compare the p-value to 0.05 and state their conclusion in context.

2022 #5 estimating a P-value from a dotplot

Examples of Student Posters:

3,168 views7 comments

Recent Posts

See All


Nov 14, 2023

Can we do a simulation with stapplet?

Leigh Nataro
Leigh Nataro
Dec 30, 2023
Replying to

Since it is entirely web-based, it should work on Chromebooks.


Unknown member
Oct 26, 2023

Is there an updated version of the project outline and/or rubric that includes the simulation piece? I'm new to the project side of things, so any guidance would be appreciated!


Unknown member
Oct 19, 2023

This post just made it to the top of my favorite's list. Thanks for sharing such a great resource (CODAP). I've been a diehard Fathom user, but it's lack of Mac support has been a barrier to students using it in the classroom. CODAP is a game changer for me (thank you, again).

I've enjoyed doing the Response Bias Project for years but I LOVE the addition of the simulation piece. This is so cool! Now I just have to go and revise the rubric to include the simulation component to the project. I can't wait to see the results this semester.


Unknown member
Oct 18, 2023

My school shut down this project a few weeks ago (after us doing this for nearly 20 years). Apparently, it is against district policy for students to collect data from other students.

Leigh Nataro
Leigh Nataro
Oct 21, 2023
Replying to

Depending on the question, they could collect data from teachers or family members or other people in the community. Would that be a possiblity? Or since this is such a great project, could you get the policy changed to allow this project specifically? What was the rationale for banning this after 20 years?

bottom of page