Introduction to Bayesian
Data Analysis
Study Guide 1
Prepared by:
Asst. Prof. Sherelyn A. Evangelio
Introduction
• Bayesian data analysis has two foundational ideas. The first idea is that Bayesian
inference is reallocation of credibility across possibilities.
• The second foundational idea is that the possibilities, over which we allocate
credibility, are parameter values in meaningful mathematical models.
Bayesian Inference is Reallocation of Credibility
Across Possibilities
Suppose we step outside one morning and notice that the sidewalk is wet, and
wonder why. We consider all possible causes of the wetness, including possibilities
such as recent rain, recent garden irrigation, a newly erupted underground spring, a
broken sewage pipe, a passerby who spilled a drink, and so on. If all we know until
this point is that some part of the sidewalk is wet, then all those possibilities will have
some prior credibility based on previous knowledge. For example, recent rain may
have greater prior probability than a spilled drink from a passerby. Continuing on our
outside journey, we look around and collect new observations. If we observe that the
sidewalk is wet for as far as we can see, as are the trees and parked cars, then we re-
allocate credibility to the hypothetical cause of recent rain. The other possible causes,
such as a passerby spilling a drink, would not account for the new observations. On
the other hand, if instead we observed that the wetness was localized to a small area,
and there was an empty drink cup a few feet away, then we would re-allocate
credibility to the spilled-drink hypothesis, even though it had relatively low prior
probability.
Bayesian Inference is Reallocation of Credibility
Across Possibilities
• The word “Credibility” is
synonymous with “probability.”
• Another example of Bayesian
inference has been immortalized
in the words of Sherlock Homes to
Doctor Watson:
“How often have I said to you that
when you have eliminated the
impossible, whatever remains,
however improbable, must be the
truth?”
Bayesian Inference is Reallocation of Credibility
Across Possibilities
• The reallocated distribution of
credibility is called the posterior
distribution. It then becomes the
prior beliefs for subsequent
observations.
• This reallocation of credibility is
not only intuitive, it is also what
the exact mathematics of Bayesian
inference prescribe.
Bayesian Inference is Reallocation of Credibility
Across Possibilities
• The complementary form of reasoning is also
Bayesian, and can be called judicial exoneration.
• Suppose there are several possible culprits for a
crime, and that these suspects are mutually
unaffiliated and exhaust all possibilities. If evidence
accrues that one suspect is definitely culpable, then
the other suspects are exonerated.
Data are noisy and inferences are probabilistic
• Previous examples assumed that observed data had definitive, deterministic
relations to the candidate causes. For example, Holmes may have found a footprint
at the scene of the crime and identified the size and type of shoe with complete
certainty, thereby completely ruling out a particular candidate suspect.
• In reality, data have only probabilistic relations to their underlying causes, the
measurements are not perfect, and the footprint is only an imperfect representation
of the shoe that produced it.
• The relation between the cause and the measured effect is full of random variation.
• In scientific research, measurements are replete with randomness. Extraneous
influences contaminate the measurements despite tremendous efforts to limit their
intrusion.
• All scientific data have some degree of “noise” in their values. The techniques of
data analysis are designed to infer underlying trends from noisy data.
Data are noisy and inferences are probabilistic
• We can collect data and only incrementally adjust the
credibility of some possible trends. The beauty of Bayesian
analysis is that the mathematics reveal exactly how much to
reallocate credibility in realistic probabilistic situations.
• Suppose there is a manufacturer of inflated bouncy balls, and
the balls are produced in four discrete sizes, diameters of 1.0,
2.0, 3.0, and 4.0 units. Suppose we submit an order to the
factory for the three balls of size 2. We receive three balls with
diameters of 1.77, 2.23, and 2.70.
From those measurements, can we conclude that the factory
correctly sent us three balls of size 2, or did the factory send
size 3 or size 1 by mistake, or even size 4?
Data are noisy and inferences are probabilistic
• Inferring the underlying manufacturing size of the balls from their “noisy”
individual diameters is analogous to data analysis in real-world scientific research
and applications.
• The data are noisy indicators of the underlying generator. We hypothesize a range
of possible underlying generators, and from the data we infer their relative
credibilities.
• Other examples are testing people for illicit drug use and detection of spam in
email.
• Bayesian analysis is the mathematics of reallocating credibility in a logically
coherent and precise way across possibilities. The distribution of credibility initially
reflects prior knowledge about the possibilities. Then new data are observed.
Possibilities that are consistent with the data garner more credibility, while those
that are not consistent lose credibility.
Possibilities are Parameter Values in Descriptive
Models
• A key step in Bayesian analysis is defining the set of possibilities over which
credibility is allocated. A posterior predictive check is a process of expanding the set of
possibilities when the data seem not to be well described by the chosen set of
possibilities.
• Consider the example of the blood-pressure drug, in which blood pressures are
measured in one group that took the drug and in another group that took placebo.
The magnitude of difference in blood pressure describes the data, and our goal is to
assess which possible descriptions are more or less credible.
• In general, data analysis begins with a family of candidate descriptions for the data.
These are mathematical formulas that characterizes the trends and spreads in data.
The formulas have parameter values that determine the exact shape of mathematical
forms.
Possibilities are Parameter Values in Descriptive
Models
• For example, the normal distribution has two
parameters, the mean which is the location parameter,
and the standard deviation which is the scale parameter.
• The role of the Bayesian inference is to compute the
exact relative credibilities of candidate parameter
values.
• In realistic applications, the candidate parameter values
can form an infinite continuum. For normal
distribution, the range of the location parameter are all
real numbers.
• Bayesian inference operates without trouble on infinite
continuums.
Possibilities are Parameter Values in Descriptive
Models
• There are two main desiderata for a mathematical description of data. First, the
mathematical form should be comprehensible with meaningful parameters.
• Second, it should be descriptively adequate, the mathematical form should “look
like” the data.
• It is important to understand that mathematical descriptions of the data are no
necessarily causal explanations of the data.
• The parameters are “meaningful” only in the context of the familiar mathematical
form defined by the distribution; the parameter values have no necessary meaning
with respect to causes in the world.
The Steps of Bayesian Data Analysis
In general, Bayesian analysis of data follows these steps:
1. Identify the data relevant to the research questions. What are the measurement
scales of the data? Which data variables are to be predicted, and which data
variables are supposed to act as predictors?
2. Define a descriptive model for the relevant data. The mathematical form and its
parameters should be meaningful and appropriate to the theoretical purposes of
the analysis.
3. Specify a prior distribution on the parameters. The prior must pass muster with
the audience of the analysis, such as skeptical scientists.
4. Use Bayesian inference to re-allocate credibility across parameter values. Interpret
the posterior distribution with respect to theoretically meaningful issues
(assuming that the model is a reasonable description of the data; see next step).
The Steps of Bayesian Data Analysis
5. Check that the posterior predictions mimic the
data with reasonable accuracy (i.e., conduct a
“posterior predictive check”). If not, then
consider a different descriptive model.
Suppose we are interested in the relationship
between weight and height of people. In particular,
we might be interested in predicting a person’s
weight based on their height.
Step 1: Identify the relevant data. Suppose we have
been able to collect heights (in) and weights (lb)
from 57 mature adults sampled at random from a
population of interest.
The Steps of Bayesian Data Analysis
Step 2: Define a descriptive model of the data that is meaningful for our research of
interest. We will describe predicted weight as a multiplier times height plus a
baseline, denoted mathematically as
𝑦ො = 𝛽1 𝑥 + 𝛽0
where 𝑦ො is the predicted weight, 𝛽1 indicates how much the predicted weight
increases when the height 𝑥 goes up by 1 inch, and the baseline 𝛽0 represents the
weight of a person who is 0 inches tall. The above equation is the model of trend and
is often called linear regression.
We have to describe the random variation of actual weights around the predicted
weights. We assume that the actual weights 𝑦 are distributed randomly according to a
normal distribution around 𝑦ො and with standard deviation 𝜎, symbolically denoted as
𝑦~normal(𝑦,ො 𝜎)
The full model has three parameters, the slope 𝛽1 , the intercept 𝛽0 and standard
deviation of the “noise,” 𝜎.
The Steps of Bayesian Data Analysis
Step 3: Specify a prior distribution on the parameters. We might inform the prior with
previously conducted research. For this example, we will use equal prior credibility
across a vast range of possible values for the slope and intercept, both centered at 0.
For the noise parameter, we will use uniform distribution. This choice of prior
distribution implies that it has virtually no biasing influence on the resulting posterior
distribution.
Step 4: Interpret the posterior distribution. Bayesian inference has reallocated
credibility across parameter values. The posterior distribution indicates combinations
of 𝛽1 , 𝛽0 , and 𝜎 that together are credible, given the data.
The Steps of Bayesian Data Analysis
Step 4 (cont.):
The posterior distribution of 𝛽1 indicates that
the most credible value of the slope is about
4.1. It also shows the uncertainty in the
estimated slope. One way to summarize the
uncertainty is by marking the span of values
that are most credible and cover 95% of the
distribution, called the highest density interval
(HDI).
It can be observed that the slope of 0 falls far
outside any credible value for the slope. We
could decide to “reject” 0 slope.
The Steps of Bayesian Data Analysis
Step 5: Check that the model actually mimics the
data reasonably well. This is called the
“posterior predictive check.”
By visual inspection of the graph, we can see
that the actual data appear to be well described
by the predicted data.
If the actual data appear to deviate
systematically from the predicted form, then we
could contemplate alternative descriptive
models.
Data analysis without parametric models?
• One situation in which it might appear that parametized models are not used is
with so called nonparametric models. But they do actually have parameters.
• Suppose we want to describe the weights of dogs, sampled at random from the
entire spectrum of dog breeds. There are probably clusters of weights, each has its
own parameters, for different breeds of dogs. The number of parameter in the
model is inferred and can grow to infinity with infinite data.
• There are variety of situations which it might seem at first that no parametized
model would apply. In case of disease diagnosis, the parameter refer to discrete
states instead of continuous distributions.
• Finally, there might be some situation in which the analyst is loathe to commit to
any parametized model of the data, even tremendously flexible infinitely
parametized models. One case is trying to make inferences from data without using
a model is resampling or bootstrapping.
