Statistics for Data Science - 2
Week 10 Notes
1. Parameter estimation: Let X1 , . . . , Xn ∼ iid X, parameter Θ
Prior distribution of Θ: Θ ∼ fΘ (θ)
Samples: x1 , . . . , xn , notation S = (X1 = x1 , . . . Xn = xn )
Bayes’ rule: posterior ∝ likelihood × prior
P (Θ = θ | S) = P (S | Θ = θ)fΘ (θ)/P (S)
P
In case of discrete: P (S) = P (S | Θ = θ)fΘ (θ)
θ R
In case of continuous: P (S) = P (S | Θ = θ)fΘ (θ) dθ
θ
Posterior mode: θ̂ = arg maxθ P (S | Θ = θ)fΘ (θ)
Posterior mean: E[Θ | S], mean of posterior distribution.
2. Bernoulli(p) samples with uniform prior: X1 , . . . , Xn ∼ iid Bernoulli(p)
Prior p ∼ Uniform[0, 1]
Samples: x1 , . . . , xn
Posterior: p| (X1 = x1 , . . . Xn = xn )
Posterior density ∝ P (X1 = x1 , . . . Xn = xn | p = p) × fp (p)
Posterior density ∝ pw (1 − p)n−w
⇒ Posterior density: Beta(w + 1, n − w + 1)
X1 + X 2 + . . . + Xn + 1
Posterior mean: p̂ =
n+2
3. Bernoulli(p) samples with beta prior: X1 , . . . , Xn ∼ iid Bernoulli(p)
Prior p ∼ Beta(α, β)
⇒ fp (p) ∝ pα−1 (1 − p)β−1
Samples: x1 , . . . , xn
Posterior: p| (X1 = x1 , . . . Xn = xn )
Posterior density ∝ P (X1 = x1 , . . . Xn = xn | p = p) × fp (p)
Posterior density ∝ pw+α−1 (1 − p)n−w+β−1
⇒ Posterior density: Beta(w + α, n − w + β)
X1 + X 2 + . . . + Xn + α
Posterior mean: p̂ =
n+α+β
4. Normal samples with unknown mean and known variance: X1 , . . . , Xn ∼ iid
Normal(M, σ 2 )
1
Prior M ∼ Normal(µ0 , σ02 )
2
⇒ fM (µ) = √2πσ 1
exp(− (µ−µ2
2σ0
0)
)
0
Samples: x1 , . . . , xn , Sample mean: x = (x1 + . . . + xn )/n
Posterior: M| (X1 = x1 , . . . Xn = xn )
Posterior density ∝ f (X1 = x1 , . . . Xn = xn | M = µ) × fM (µ)
2 2 2
Posterior density ∝ exp(− (x1 −µ) +...+(x 2σ02
n −µ)
)exp(− (µ−µ 0)
2σ02
)
⇒ Posterior density: Normal
X1 + X2 + . . . + Xn nσ02 σ2
Posterior mean: µ̂ = + µ 0
n nσ02 + σ 2 nσ02 + σ 2
5. Geometric(p) samples with Uniform[0, 1] prior: X1 , . . . , Xn ∼ iid Geometric(p)
Prior p ∼ Uniform[0, 1]
Samples: x1 , . . . , xn
Posterior: p| (X1 = x1 , . . . Xn = xn )
Posterior density ∝ P (X1 = x1 , . . . Xn = xn | p = p) × fp (p)
Posterior density ∝ pn (1 − p)x1 +...+xn −n
⇒ Posterior density: Beta(n + 1, x1 + . . . + xn − n + 1)
n+1
Posterior mean: p̂ =
X1 + . . . + Xn + 2
6. Poisson(λ) samples with gamma prior: X1 , . . . , Xn ∼ iid Poisson(Λ)
Prior Λ ∼ Gamma(α, β)
⇒ fΛ (λ) ∝ λα−1 e−βλ
Samples: x1 , . . . , xn
Posterior: Λ | (X1 = x1 , . . . Xn = xn )
Posterior density ∝ P (X1 = x1 , . . . Xn = xn | Λ = λ) × fΛ (λ)
Posterior density ∝ e−nλ λx1 +...+xn λα−1 e−βλ
⇒ Posterior density: Gamma(x1 + . . . + xn + α, β + n)
X1 + X2 + . . . + Xn + α
Posterior mean: λ̂ =
n+β
