Name: _________________
AP Statistics Handout: Lesson 4.3
Topics: observation vs. experiment, components of experiments, completely randomized design
Lesson 4.3 Guided Notes
Observational Studies vs. Experiments
Data Sources:
-Huang, J. et al. “Historical comparison of
gender inequality in scientific careers across
countries and disciplines.” Proceedings of
the National Academy of Sciences, Mar 2020,
117 (9) 4609-4616; DOI:
10.1073/pnas.1914221117
-Boston Consulting Group, “What’s Keeping
Women out of Data Science?”
bcg.com/publications/2020/what-keeps-
women-out-data-science.aspx
1. One possible cause of gender gaps in STEM is hiring bias. Provide an explanation for how hiring biases
could cause the trends shown above:
2. Using the observational data displayed above, can we prove that hiring discrimination is the cause of
gender gaps in STEM? Why or why not?
Confounding variables: Provide ___________________ explanations for trends between explanatory
(gender) and response (hiring rates) variables.
3. Name and discuss one confounding variable that could also explain the gender gap in STEM:
Observational study: a study in which data is collected ___________________________ any treatments.
• Retrospective study: examines ________________________ on individuals
• Prospective study: follows individuals to gather ____________________
• Both ______________ show cause and effect because they do not control for confounding!
Experiments: a study in which treatment is _______________ on subjects.
• If well designed, experiments _____________ cause-effect relationships by _________________
for confounding variables.
Material adapted from Skew The Script (skewthescript.org)
2
Components of Experiments
The Jenn/John Study
1
In this study, experimenters printed up copies of fake application
materials for a science lab manager job. All applications were
identical, except for one thing: the name. On half the applications,
the applicant’s name was listed as “John.” The others had “Jennifer.”
Experimenters found 127 science lab faculty members and randomly sent them either a “John” (n = 63)
or “Jennifer” (n = 64) application. Each faculty member was told that the application was for a real
position at their University. The faculty members independently rated the applicant’s “hireability” (1-7
scale) and estimated their starting salary. Experimenters compared these results across the two groups.
Experimental units: the ________________________ (person, animal, plant, virus, particle, etc.) that are
assigned to different treatments.
What were the experimental units in the hiring study?
Explanatory variable: the variable that is purposefully __________________. This is also known as the
factor.
What was the explanatory variable in the hiring study?
Treatments: the different _____________ of the explanatory variable in the experiment.
What were the treatments in the hiring study?
Response variable: the measured experiment ___________________ that is compared between
treatment groups.
What was the response variable in the hiring study?
The Four Principles of Experimental Design
1. Comparison
a) Explain how comparison is implemented in the Jenn/John study:
2. Random Assignment
a) Why would it be a problem if the experimenters assigned Tier 1 University labs to get “John” and
Tier 2 labs to get “Jennifer” application materials?
1Example inspired by Advanced High School Statistics. Original study: Moss-Racusin, C., Dovidio, J., et al. “Science faculty’s subtle gender biases favor male
students.” PNAS October 9, 2012 109 (41) 16474-16479; https://doi.org/10.1073/pnas.1211286109
Material adapted from Skew The Script (skewthescript.org)
3
b) In general, why is it important to randomly assign experimental units to treatment?
c) Sampling Experiments
Random ___________ Random ____________
→ Reduces ________ → Reduces _______________
3. Replication (definition: having many experimental units in each treatment group)
a) Why would it be a problem if I only assigned one faculty member to the “John” group and one
faculty member to the “Jennifer” group?
b) In general, why is it important to include many experimental units in each treatment group?
4. Control
a) Explain how control is implemented in the Jenn/John study:
Describing a Completely Randomized Design
Completely randomized design: An experimental design in which experimental units are assigned to
treatments _______________________________________.
• This is the “SRS” of experiments – the simplest (but still effective) randomized experiment.
Question: Describe how you would implement a completely randomized design of the Jennifer/John
experiment, with 127 science faculty members.
When describing how to perform a completely randomized
design experiment:
1. Assign each experimental unit a number 1 – n (sample size).
2. Write all the numbers on identical slips of paper, put into a
hat, and mix well.
3. Draw out nt (treatment group size) slips of paper, without
replacement. The corresponding units are assigned treatment 1.
Draw out another nt slips of paper, assign to treatment 2, etc.
4. Compare response among treatment groups
Material adapted from Skew The Script (skewthescript.org)
4
Lesson 4.3 Discussion
Here’s a summary of the Jenn/John study results:
Estimates based on
publicly available data
from the study.
The differences in salary estimates and hireability ratings John Jennifer Difference
were found to be statistically significant: so extreme that Hireability 3.78 2.93 -0.85
they were unlikely to happen by chance alone. Salary $30,238 $26,508 -$3,730
Since this was a well-designed experiment, we can infer that these average differences are
_________________ by gender bias.
Discussion Question: Is the proven gender bias among the faculty in this study enough to explain the
large gender gaps in STEM? Or might there be other causes as well? Explain your reasoning.
Lesson 4.3 Practice
Teachers: We recommend providing additional practice exercises from your AP Stats textbook or from
prior AP Stats exams. The following textbook sections and AP exam questions are aligned to the content
covered in this lesson.
• The Practice of Statistics (AP Edition), 4th-6th editions: section 4.2
• Stats: Modeling the World (AP Edition), 4th/5th editions: ch 12, 3rd edition: ch 13
• Statistics: Learning from Data (AP Edition), 2nd edition: section 1.4
• Advanced High School Statistics, section 1.5
• AP Exam Free Response Questions (FRQs): 2019 Q2, 2011 Form B Q2, 2018 Q4 (part a)
Material adapted from Skew The Script (skewthescript.org)
