Probability Theory: Class Notes

1. Introduction to Probability Theory

Probability theory is the branch of mathematics that deals with the analysis of random phenomena. It
provides a framework for quantifying uncertainty, predicting outcomes, and modeling random events.
The key concepts in probability theory are:

Random Experiment: An experiment or process with uncertain outcomes.

Sample Space: The set of all possible outcomes of a random experiment.

Event: A specific outcome or a collection of outcomes from the sample space.

Probability: A number between 0 and 1 that quantifies the likelihood of an event occurring.

2. Basic Definitions

a. Sample Space and Events

Sample Space (S): The set of all possible outcomes of an experiment. For example, when rolling a die, the
sample space is

,
6

S={1,2,3,4,5,6}.

Event (A): A subset of the sample space. For example, getting an even number when rolling a die is the
event

A={2,4,6}.

b. Probability

The probability of an event

A, denoted as

P(A), is the measure of the likelihood that event

A will occur. It satisfies the following properties:

0≤P(A)≤1

P(S)=1, where

S is the sample space.

For mutually exclusive events

A and

B,

𝐴
∪

P(A∪B)=P(A)+P(B).

3. Probability Rules

a. Addition Rule (for Mutually Exclusive Events)

If two events,

A and

B, are mutually exclusive (i.e., they cannot happen simultaneously), the probability that either event
occurs is:

∪
𝐵

P(A∪B)=P(A)+P(B)

b. General Addition Rule

For events that may not be mutually exclusive, the probability of the union is given by:

)
+

P(A∪B)=P(A)+P(B)−P(A∩B)

Where

P(A∩B) is the probability of both events occurring.

c. Multiplication Rule (for Independent Events)

If events

A and

𝐵
B are independent (the occurrence of one does not affect the probability of the other), the probability
that both events occur is:

P(A∩B)=P(A)⋅P(B)

d. Conditional Probability

The probability of event

A given that event

B has occurred is called the conditional probability of

𝐴
A given

B, denoted as

P(A∣B):

𝐵
)

P(A∣B)=

P(B)

P(A∩B)

This formula is only valid if

>

P(B)>0.

4. Types of Probability

a. Theoretical Probability

Theoretical probability is based on the assumption that all outcomes are equally likely. For example,
when rolling a fair six-sided die, the probability of rolling a 3 is:

rolling a 3

P(rolling a 3)=
6

b. Empirical (Experimental) Probability

Empirical probability is determined by conducting an experiment or observing data. It is estimated as:

Number of times event A occurs

Total number of trials

P(A)=

Total number of trials

Number of times event A occurs

c. Subjective Probability

Subjective probability is based on personal judgment, intuition, or experience rather than statistical
analysis or theoretical models.

5. Random Variables and Probability Distributions

a. Random Variables

A random variable is a function that assigns a real number to each outcome of a random experiment.
There are two types:

Discrete Random Variable: Takes on a finite or countably infinite number of values (e.g., the number of
heads in 3 coin flips).
Continuous Random Variable: Takes on an uncountably infinite number of values within a range (e.g., the
height of a person).

b. Probability Distribution

The probability distribution of a random variable gives the probability of each possible outcome. It can
be:

Discrete Probability Distribution: For a discrete random variable, the probability distribution is given by a
probability mass function (PMF).

Continuous Probability Distribution: For a continuous random variable, the probability distribution is
given by a probability density function (PDF).

6. Common Probability Distributions

a. Binomial Distribution

The binomial distribution models the number of successes in a fixed number of independent Bernoulli
trials (each with two possible outcomes, success or failure). The probability mass function is given by:

(
1

P(X=k)=(

)p

(1−p)

n−k

Where:

n is the number of trials.

k is the number of successes.

p is the probability of success on each trial.

b. Normal Distribution

The normal distribution (or Gaussian distribution) is a continuous probability distribution that is
symmetric around its mean. The probability density function is:
𝑓

f(x)=

2πσ

2
1

2σ

(x−μ)

Where

μ is the mean and

σ is the standard deviation. The normal distribution is important in many fields due to the Central Limit
Theorem.

c. Poisson Distribution

The Poisson distribution models the number of events occurring in a fixed interval of time or space,
given a known average rate. Its probability mass function is:

𝑋
=

P(X=k)=

k!

−λ

Where

λ is the average rate of events.

7. Expected Value and Variance

a. Expected Value (Mean)

The expected value of a random variable

X, denoted as

E(X), is the long-term average or the "center" of the distribution. For a discrete random variable:

E(X)=

∑
x⋅P(X=x)

For a continuous random variable:

E(X)=∫

−∞

xf(x)dx

b. Variance

The variance of a random variable

𝑋
X, denoted as

Var

Var(X), measures the spread or dispersion of the distribution. For a discrete random variable:

Var

Var(X)=E(X

2
)−(E(X))

8. Law of Large Numbers

Law of Large Numbers (LLN) states that as the number of trials in an experiment increases, the empirical
probability (relative frequency) of an event will converge to the theoretical probability. In other words,
with a large enough sample size, the average of observed outcomes will get closer to the expected value.

9. Central Limit Theorem

The Central Limit Theorem (CLT) states that for a sufficiently large sample size, the distribution of the
sum (or average) of independent and identically distributed random variables will approach a normal
distribution, regardless of the original distribution.

10. Applications of Probability Theory

Statistics: Probability theory underlies statistical methods such as hypothesis testing, confidence
intervals, and regression analysis.

Risk Analysis: Used in finance, insurance, and decision-making under uncertainty.

Machine Learning: Many algorithms in machine learning, like Naive Bayes and Bayesian inference, are
based on probability theory.

11. Conclusion

Probability theory is essential for modeling uncertainty, making predictions, and understanding random
processes. It forms the foundation for various fields, from statistics to machine learning and decision
theory. Mastery of probability theory helps in interpreting data, analyzing risks, and solving real-world
problems.