Statistics & Probability
Probability vs. Statistics Both deal with uncertainty, randomness
Probability: It deals with the prediction of likelihood of future events
Logically self-contained
Follows few rules
Has one correct answer
Statistics: It involves the analysis of the frequency of past events
Works on experimental data
No single correct answer
Helps to understand patterns, relationships, and trends within data
Statistics – Classification
Descriptive Statistics: It involves methods for summarizing and presenting data in a
meaningful and concise manner. Provides a snapshot of the data’s characteristics.
e.g.: Mean, Median, Mode, Range, Variance, Standard Deviation
Inferential Statistics: It involves using sample data to draw inferences, predictions, or
decisions about a larger population.
e.g.: Hypothesis Testing, Confidence Intervals
Terminology
A population is the entire group that you want to draw conclusions about.
A sample is the specific group that you will collect data from using random
sampling. The size of the sample is always less than the total size of the population.
Sampling
Inference
Link: Quora
I.I.D
Independent and identically distributed (or IID) random variables are mutually
independent of each other and are identically distributed in the sense that they are drawn
from the same probability distribution.
Example: Coin flips, Rolling a fair six-sided die, survey responses are assumed to be IID
Who is best? X vs. Y
Exams 1 2 3 4 5 6 7 8 9 10
Student Scores (50)
X 28 32 44 33 43 30 41 36 42 27
Y 48 19 22 45 26 50 31 28 38 49
Student X Student Y
SUM 356 356
MEAN 35.6 35.6
MEDIAN 34.5 34.5
STANDARD 6.4 11.9
DEVIATION
Probability
In a random experiment, probability of an event is a number indicating how likely that
event will occur.
The sample space associated with a random experiment is the set of all
possible outcomes. An event is a subset of the sample space.
e.g., Ω = {1,2,3,4,5,6}
A real number P(A) is assigned to every event A, called the probability of A.
This number P(A) is always between 0 and 1, where 0 indicates impossibility and
1 indicates certainty.
Probability
To qualify as a probability, P must satisfy three axioms:
Axiom 1: P(A) ≥ 0 for any event A
Axiom 2: Probability of the sample space, P(Ω) = 1
Axiom 3: If A1,A2, . . . are disjoint (mutually exclusive) events, then
P(A1 A2 A3..... )=P(A1) + P(A2) + P(A3) + ....
Probability Models – Discrete Probability
Discrete Probability Model: In discrete probability models we can compute the
probability of events by adding all the corresponding outcomes
Bernoulli Model: Simple Model that represents binary outcomes
with a single parameter
Binomial Model: Describes the number of successes in a fixed Source: Wikipedia
number of independent Bernoulli trial
Poisson Model: Models the number of events occurring in a fixed
interval of time or space
Probability Models – Continuous Probability
Continuous Probability Models: In continuous probability models, a random variable X
can take on any value (is continuous).
Uniform Distribution: Represents a situation where all values in an interval are equally
likely. e.g.: Rolling of a fair six-sided die, generating random numbers between 0 and 1
Normal (Gaussian) Distribution: Describes continuous variables that are symmetrically
distributed around a mean, often referred to as the bell curve. e.g.: Height of adult
individuals in a population
Probability Models – Joint Probability
Joint Probability: It is the likelihood of more than one event occurring at the same time
Conditions:
One is that events X and Y must happen at the same time. Example: Throwing two dice
simultaneously.
The other is that events X and Y must be independent of each other. That means the
outcome of event X does not influence the outcome of event Y.
Example: Rolling two Dice.
If the following conditions met, then P(A∩B) = P(A) * P(B).
Probability Models – Joint Probability
Tossing a fair coin and Rolling a Die
Find the joint probability of getting heads on the coin toss and rolling a 5 on the die?
the probability of getting heads (event A): P(A) = 0.5
the probability of rolling a 5 (event B): P(B) = 1/6
P(A ∩ B) = P(A) * P(B) = 0.5 * (1/6) = 1/12 ≈ 0.0833
What will happen to the joint probability of two dependent events?
Probability Models – Conditional Probability
Conditional probability: It is the probability of occurrence of an event B given the
knowledge that an event A has already occurred. It is denoted by P(B|A).
The joint probability of two dependent events then becomes P(A and B) = P(A)P(B|A)
Note:
Two events A and B are independent if P(AB) = P(A) P(B)
Two events A and B are conditionally independent given C if they are independent after
conditioning on C
P(AB|C) = P(B|AC)P(A|C) = P(B|C)P(A|C)
Conditional Probability - Example
60% of students pass the Final and 45% of students pass both the Final and the
Midterm. What percent of students who passed the Final also passed the Midterm?
What percent of students passed the Midterm given they passed the Final?
P(F) =0.6 , P(M and F) = 0.45, P(M|F)=?
P(M and F) = P(F) P(M|F)
P(M|F) = P(M and F) / P(F) = .45 / .60 = .75
Bayes Theorem
Joint Probability of two dependent events:
P(A and B) = P(A)P(B|A) P(B and A) = P(B)P(A|B)
P(A)P(B|A) = P(B)P(A|B) P(A|B) = P(A) P(B|A) / P(B)
This is the Bayes theorem
It tells us: how often A happens given that B happens: P(A|B) – Posterior Probability
When we know:
how often B happens given that A happens, P(B|A) – Likelihood
how likely A is on its own, P(A) – Prior Probability
how likely B is on its own, P(B) – Evidence
Bayes Theorem [Example]
A rare disease affect 0.1% of the population. A test proposed for this disease is 99% accurate
for people who have the disease. What is the probability that a person actually has the disease
if they test positive?
Only 9% chance that the person is having the disease!
Probability Distribution
A probability distribution is a mathematical function that describes the likelihood of
various outcomes in a random experiment or process.
Probability Mass Function (PMF):
The PMF gives the probability that a specific value of the random variable occurs in
case of discrete distributions
Probability Density Function (PDF):
The PDF in case of continuous distributions, gives the relative likelihood of the random
variable taking on a particular value.
The area under the PDF curve over an interval represents the probability of the variable
falling within that interval
Probability Mass Function
A PMF describes the probabilities of
discrete random variables taking on specific
values.
It provides a complete distribution of
probabilities for all possible outcomes.
𝟏 𝟏 𝟏 𝟏 𝟏 𝟏
𝑬 𝑿 =𝟏 +𝟐 +𝟑 +𝟒 +𝟓 +𝟔
𝟔 𝟔 𝟔 𝟔 𝟔 𝟔
= 𝟑. 𝟓
PMF-Example
Consider a biased coin with P(H) = 0.7 and P(T) = 0.3
X: The number of times the coin lands heads up in two consecutive flips.
Possible Outcomes: X={0,1,2}
P(X=0) = P(TT) = 0.3*0.3 = 0.09
P(X=1) = P(HT) + P(TH) = 0.7*0.3+0.3*0.7 = 0.42 PMF
P(X=2) = P(HH) = 0.7*0.7 = 0.49
Expected Value (Mean):
PMF-Example
Consider a biased coin with P(H) = 0.7 and P(T) = 0.3
X: The number of times the coin lands heads up in two consecutive flips.
Possible Outcomes: X={0,1,2}
P(X=0) = P(TT) = 0.3*0.3 = 0.09
P(X=1) = P(HT) + P(TH) = 0.7*0.3+0.3*0.7 = 0.42 PMF, Total Probability = 1.0
P(X=2) = P(HH) = 0.7*0.7 = 0.49
Variance, Var(X)
+
Types of PMFs – Bernoulli Distribution
The Bernoulli distribution models a single binary trial with two possible outcomes:
success (S) with probability "p" and failure (F) with probability "q" (where q = 1 - p).
Each trial in the Bernoulli distribution has only two possible outcomes, often labeled as 1
(success) and 0 (failure).
Types of PMFs – Binomial Distribution
It is a discrete probability distribution that models the number of successes in a fixed
number of independent Bernoulli trials
It summarizes the probability that a value will take one of two independent values
under a given set of parameters or assumptions
N = 20 trials with p=q=1/2
The binomial distribution of obtaining exactly n
successes out of N Bernoulli trials is given by:
Types of PMFs – Poisson Distribution (
It is a discrete probability distribution that models the number of events occurring in a
fixed interval of time or space, given the average rate of occurrence.
The distribution is used to model rare events that occur randomly and independently
over a specified time or space interval
λ: the average rate of occurrence
k: number of events
Probability Density Function
It is used to describe the probability distribution of a continuous random variable, where
the set of possible outcomes is an uncountably infinite range, such as real numbers within
an interval.
It defines the likelihood of the variable falling within a particular range of values.
Uniform Distribution
It describes a situation where all values within a specific interval [a, b] are equally
likely to occur, i.e., have the same probability of occurring
a b
PDF - Normal/Gaussian Distribution
It is a continuous probability distribution that is
characterized by its bell-shaped curve. The curve
tails off towards the extremes.
It is symmetric with the highest point at the
mean, and the spread of the distribution
determined by the standard deviation.
( )
( )
Normal Distribution - Empirical (68-95-99.7)Rule
It is a statistical guideline that describes the approximate
distribution of data in a normal distribution
About 68.26% of the data falls within 1σ of µ.
About 95.44% of the data falls within 2σ of µ.
About 99.72% of the data falls within 3σ of µ.
The rest 0.28% of the whole data lies outside http://www.cs.uni.edu/~campbell/stat/normfact.html
3σ of µ
Example
Consider a dataset of exam scores that follows a normal distribution with a mean of 80 and
a standard deviation of 10.
About 68% of the scores fall within the range: 70-90
About 95% of the scores fall within the range: 60-100
About 99.7% of the scores fall within the range: 50-110
Central Limit Theorem
It describes the behavior of the sample means from a population, regardless of the
population's underlying distribution
It states that as the sample size increases, the distribution of sample means
approaches a normal distribution, regardless of the original population's distribution.
Provided we have a population with μ and σ and take large random samples (n ≥ 30)
from the population with replacement, the distribution of the sample means will be
approximately normally distributed with:
