(Dis) trusting Statistics: A Mini-Guide
1. What is the source of the statistics?
What person or organization is providing this information?
What are its qualifications and area of expertise? What are
likely to be its biases, if any? How readily can you check
the source – and possibly its sources in turn – for their
qualifications, reputation, and consistency with other
statistics offered by respected organizations and journals?
2. What is actually being claimed?
Are words and terms clear, and used in a way consistent
with definitions in the relevant field? Is the knowledge
claim a factual one about the present or past, or is it a
hypothetical prediction about the future? Does it report on
a single study or a meta-study? If it reports a survey, how
large and representative is the sample population?
(Become familiar with the following: different kinds of
averages; difference between correlation and causation;
the terms “statistically significant”, "p-value", “p-hacking”, Vampires don't like sunshine. Can you tell?
“background noise”.)
3. How are the statistics framed in context?
Are the statistics used as supporting evidence for a
knowledge claim or argument? (And how valid is that
argument?) Are the numbers being used – instead or
as well – to impress in a more emotional way? Do
accompanying images or language clarify the
significance of the statistics – and/or possibly
heighten an emotional impact? Does it seem that
other important statistical information has been
omitted?
4. What is your own emotional response to
the statistics?
Do you notice in your own reaction to the statistics
any inclination to accept or reject the statistics even
before you’ve examined them as above? Do you
detect in yourself any signs of confirmation bias – the
inclination to believe whatever harmonizes with what
you already think, or what you wish were true,
regardless of the quality of the information?
Eieen Dombrowski, Theory of Knowledge blog OUP https://educationblog.oup.com/category/theory-of-knowledge and Activating TOK
https://activatingtok.net/ Cartoons by Theo Dombrowski, permission for use in ToK classrooms