Theory of Statistics II

SA302
Lecture 25

Course Instructor: Dr Reham Elshaer

Email: [email protected]
1
 Bayesian Estimation under the absolute error loss function

The Bayes’ estimator under the absolute error loss function.

Theorem 2:
መ 𝜃 the loss resulting from estimating 𝜃 by 𝜃መ be given by |𝜃መ − 𝜃|, the
Let 𝐿 𝜃,
absolute error loss function.
Then the Bayes’ estimator is given by the posterior median.
Proof:
መ መ 𝜃መ − 𝜃 𝑖𝑓 𝜃መ > 𝜃
𝐿 𝜃, 𝜃 = |𝜃 − 𝜃|= ൝
−(𝜃መ − 𝜃) 𝑖𝑓 𝜃መ < 𝜃
መ 𝜃 = E𝜃|𝑥 𝐿 𝜃,
R 𝜃, መ 𝜃

መ 𝜃 𝑔
= න 𝐿 𝜃, 𝛩|𝑥 𝜃 𝑥 𝑑𝜃
෡
𝜃 ∞
= ‫׬‬−∞(𝜃መ − 𝜃) 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 + ‫𝜃׬‬෡ −(𝜃መ − 𝜃) 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃
መ 𝜃 with
To minimize the risk function we have to obtain the first derivative or 𝑅 𝜃,
respect to 𝜃መ and equate it to 0.

2
 Bayesian Estimation under the absolute error loss function

Using the relationship

∂ 𝑔2(θ ) ∂𝑔2 𝜃 ∂𝑔1 𝜃 𝑔2 (θ ) ∂𝑓 𝑥; 𝜃
𝑓(𝑥; 𝜃)dx 𝑓(𝑔 𝜃 ; 𝜃) 𝑓(𝑔 𝜃 ; 𝜃) ∂𝜃 𝑑𝑥
‫׬‬ = 2 - 1 + ‫׬‬
∂𝜃 𝑔1(θ ) ∂𝜃 ∂𝜃 𝑔1 (θ)
We have
∂R(𝜃,𝜃)
෡ ∂𝜃෡ መ መ ෡
𝜃 ∂𝜃෡ መ መ
= ෡ 𝜃 − 𝜃 𝑔𝛩|𝑥 𝜃 𝑥 + ‫׬‬−∞ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 + ෡ 𝜃 − 𝜃 𝑔𝛩|𝑥 𝜃መ 𝑥 −
መ
∂𝜃෡ ∂𝜃 ∂𝜃
∞
‫𝜃׬‬෡ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃
෡
𝜃 ∞
= ‫׬‬−∞ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 − ‫𝜃׬‬෡ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃
෡
𝜃 ∞
Equating the above equation to zero we get ‫׬‬−∞ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 = ‫𝜃׬‬෡ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃
Note that if m is the median of 𝑔𝛩|𝑥 𝜃 𝑥 then
𝑚 ∞
1
න 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 = න 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 =
−∞ 𝑚 2
It follows that 𝜃መ is the posterior median.

3
Bayesian Estimation under the absolute error loss function

To check that 𝜃෠ is a minimum point, we obtain the

second derivative of the posterior risk.
∂R(𝜃,𝜃)
෡ ෡
𝜃 ∞
= ‫׬‬−∞ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 − ‫𝜃׬‬෡ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃
∂𝜃෡
෠𝜃
∂2 𝑅 𝜃, ∂𝜃෠ ∂𝜃෠
∴ = 𝑔 ෠𝑥 +
𝜃 𝑔 ෠𝑥
𝜃
∂𝜃෡ 2 ∂𝜃෠ 𝛩|𝑥 ∂𝜃෠ 𝛩|𝑥
= 2𝑔𝛩|𝑥 𝜃෠ 𝑥 > 0 since 𝑔𝛩|𝑥 𝜃෠ 𝑥 is a p.d.f
∴ 𝜃෠ minimizes the posterior risk

4
 Bayesian Estimation

Example
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be a random sample from a Gamma(3, 𝜃)
distribution. The parameter 𝜃 is a value of a random variable 𝜣
having Exponential distribution with parameter a.
Find the posterior distribution of 𝜣 and then find the Bayes’
estimator of 𝜃
1) Assuming a squared error loss function.
2) Assuming an absolute error loss function.

5
 Bayesian Estimation

Solution:
𝜃 3 −𝜃𝑥 2
𝑓 𝑥𝜃 = 𝑒 𝑥 𝑥 > 0, 𝜃 > 0
Γ(3)
−𝑎𝜃
𝑔𝛩 𝜃 = 𝑎𝑒 𝑎 > 0,𝜃 > 0
𝐿(𝑥|𝜃)𝑔 (𝜃)
𝑔𝛩 (𝜃| 𝑥) = 𝛩
‫𝜃𝑑 𝜃 𝛩𝑔 𝜃|𝑥 𝐿 ׬‬
𝜃3𝑛 −𝜃 σ 𝑥𝑖 (ς𝑛 𝑥 2 )𝑎𝑒 −𝑎𝜃
Γ 3 𝑛 𝑒 𝑖=1 𝑖
𝑔𝛩 (𝜃| 𝑥) = ∞ 𝜃3𝑛
‫=𝜃׬‬0 Γ 3 𝑛 𝑒 −𝜃 σ 𝑥𝑖 (ς𝑛 2
𝑖=1 𝑥𝑖 )𝑎𝑒
−𝑎𝜃 𝑑𝜃

𝜃 3𝑛 −𝜃(σ 𝑥𝑖 +𝑎)
2𝑛 𝑒
=
∞ 𝜃 3𝑛 −𝜃(σ 𝑥 +𝑎)
‫=𝜃׬‬0 2𝑛 𝑒 𝑖 𝑑𝜃

6
 Bayesian Estimation

𝜃 3𝑛 𝑒 −𝜃(σ 𝑥𝑖 +𝑎)
=
Γ(3𝑛 + 1)
σ 𝑥𝑖 + 𝑎 3𝑛+1
𝜃 3𝑛 𝑒 −𝜃(σ 𝑥𝑖 +𝑎) σ 𝑥𝑖 + 𝑎 3𝑛+1
=
Γ(3𝑛 + 1)
∴ 𝛩| 𝑥 ~ Gamma(3n+1, σ 𝑥𝑖 + 𝑎)
i) The posterior mean is given by
3n + 1
E𝜃|𝑥 𝜃 𝑥 =
σ 𝑥𝑖 + 𝑎
7
 Bayesian Estimation

ii) To obtain the posterior median we have

2𝜃|𝑥 (𝑎 + σ 𝑥𝑖 ) ~ 𝜒22 3𝑛+1
𝑃(𝜃|𝑥 < 𝜃) ෠ =1
2
1
෠
𝑃[2𝜃|𝑥(𝑎 + σ 𝑥𝑖 ) < 2𝜃(𝑎+ σ 𝑥𝑖 )] = (1)
2
Let 𝑄 = 2𝜃|𝑥(𝑎 + σ 𝑥𝑖 )
We have that
1
𝑃 𝑄 < 𝜒22 3𝑛+1 ,0.5 = (2)
2
From (1) and (2)
෠
2𝜃(𝑎+ σ 𝑥𝑖 ) = 𝜒22 3𝑛+1 ,0.5
𝜒22 3𝑛+1 ,0.5
∴ 𝜃෠ =
2(σ 𝑥𝑖 +𝑎)

8
 Bayesian Interval Estimation

In this case, we obtain the posterior distribution

then we find the lower limit L and the upper limit U
such that
𝑃(𝐿 < 𝜃| 𝑋 < U) = 1 − 𝛼
In the case of an equal tail interval, we have
𝐿 𝛼
‫׬‬−∞ 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 = 𝐺𝛩|𝑥 𝐿|𝑥 = 2
∞
𝛼
න 𝑔𝛩|𝑥 𝜃 𝑥 𝑑𝜃 = 1 − 𝐺𝛩|𝑥 𝑈|𝑥 =
𝑈 2
𝛼
∴ 𝐺𝛩|𝑥 𝑈|𝑥 = 1 −
2
9
 Bayesian Interval Estimation

Example 1:
Consider the previous example (Slide 7) where it was shown
that 𝛩| 𝑥~ 𝐺𝑎𝑚𝑚𝑎(3𝑛 + 1, 𝑎 + σ 𝑥𝑖 )
Find a 1 − 𝛼 100% Bayesian interval for 𝜃.
Solution:
𝑃(𝜒22 3𝑛+1 𝛼
,2
< 2(𝑎+ σ 𝑥𝑖 ) 𝜃| 𝑥 < 𝜒22 3𝑛+1 𝛼
,1− 2
)=1−𝛼

𝜒2 𝛼 𝜒2 𝛼
2 3𝑛+1 , 2 2 3𝑛+1 ,1− 2
𝑃( < 𝜃| 𝑥 < )=1−𝛼
2(𝑎+ σ 𝑥𝑖 ) 2(𝑎+ σ 𝑥𝑖 )

10
 Bayesian Interval Estimation

Example 2:
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be a random sample from a N(𝜃, 1) distribution. The parameter
𝜃 is a value of a random variable 𝜣 having N(𝑎, 1) distribution.
Find a 1 − 𝛼 100% Bayesian interval for 𝜃.
Solution:
1 1
− 𝜃−𝑎 2
𝑔𝛩 𝜃 = 𝑒 2 −∞<𝜃 <∞
2𝜋
1
1 −2 𝑥−𝜃 2
𝑓(𝑥|𝜃) = 𝑒 −∞ < 𝑥 < ∞
2𝜋

1 2
1 −2 σ 𝑥𝑖 −𝜃
𝐿 𝑥𝜃 = 𝑛 𝑒
2𝜋 2
𝐿(𝑥|𝜃)𝑔 (𝜃)
𝑔𝛩 (𝜃| 𝑥) = 𝛩
‫𝜃𝑑 𝜃 𝛩𝑔 𝜃|𝑥 𝐿 ׬‬

11
 Bayesian Interval Estimation

1 1
−2[σ𝑛 (𝑥𝑖2 −2𝑥𝑖 𝜃+𝜃 2 )+(𝜃 2 −2𝜃𝑎+𝑎2 )]
𝑛+1 𝑒 𝑖=1

2𝜋 2
= 1
∞ 1 −2[σ𝑛 (𝑥 2 −2𝑥 𝜃+𝜃 2 )+(𝜃 2 −2𝜃𝑎+𝑎2 )]
𝑖
‫׬‬−∞ 𝑛+1 𝑒 𝑖=1 𝑖 𝑑𝜃
2𝜋 2

1
−2[σ𝑛 𝑥
𝑖=1 𝑖
2 −2𝜃 σ𝑛 𝑥 +𝜃2 (𝑛+1)−2𝜃𝑎+𝑎2 ]
𝑖=1 𝑖
𝑒
= 1
∞ −2[σ𝑛𝑖=1 𝑥2
𝑖 −2𝜃 σ𝑛𝑖=1 𝑥𝑖 +𝜃2 (𝑛+1)−2𝜃𝑎+𝑎2 ]
‫׬‬−∞ 𝑒 𝑑𝜃

12
 Bayesian Interval Estimation

Consider
𝑛 𝑛

෍ 𝑥𝑖2 − 2𝜃 ෍ 𝑥𝑖 +𝜃 2 𝑛 + 1 − 2𝜃𝑎 + 𝑎2
𝑖=1 𝑖=1
𝑛 𝑛

= ෍ 𝑥𝑖2 − 2𝜃[෍ 𝑥𝑖 + 𝑎] +𝜃 2 (𝑛 + 1) + 𝑎2
𝑖=1 𝑖=1

2 2𝜃(σ𝑛
𝑖=1 𝑥𝑖 +𝑎) 𝑛 2
= 𝑛+1 𝜃 − + σ𝑖=1 𝑥𝑖 + 𝑎2
𝑛+1
2
σ𝑛
𝑖=1 𝑥𝑖 +𝑎 2 σ𝑛
𝑖=1 𝑥𝑖 +𝑎 2
= 𝑛 + 1 [(𝜃 − ) − ] + σ𝑛𝑖=1 𝑥𝑖 + 𝑎2
𝑛+1 𝑛+1

13
 Bayesian Interval Estimation

2
𝑛+1 σ𝑛
𝑖=1 𝑥𝑖 +𝑎
− 𝜃− 𝐾
𝑒 2 𝑛+1
∴ 𝑔𝛩 (𝜃| 𝑥) = 2
𝑛+1 σ𝑛
𝑖=1 𝑥𝑖 +𝑎
2𝜋 ∞ 𝑛+1 − 𝜃−
𝐾 ‫=𝜃׬‬−∞ 𝑒 2 𝑛+1 𝑑𝜃
𝑛+1 2𝜋
𝑛 2
𝑛+1 σ𝑖=1 𝑥𝑖 +𝑎 1 2
−2 (σ𝑛 𝑥 𝑖 +𝑎 2)
Where 𝐾 = 𝑒 2 𝑛+1 𝑒 𝑖=1

Note that the integral in the denominator is that of a normal

σ𝑛
𝑖=1 𝑥𝑖 +𝑎 1
distribution 𝑁 , , hence it is equal to 1
𝑛+1 n+1

14
 Bayesian Interval Estimation

σ𝑛 𝑥 +𝑎 2
𝑛+1 𝑖
− 2 𝜃− 𝑖=1
𝑒 𝑛+1
𝑔𝛩 (𝜃| 𝑥) = 2𝜋
−∞ < 𝜃 < ∞
𝑛+1

2
𝑛+1 σ𝑛 𝑥𝑖 +𝑎
𝑛+1 − 2 𝜃− 𝑖=1
= 𝑒 𝑛+1
2𝜋
σ𝑛
𝑖=1 𝑥𝑖 +𝑎 1
∴ 𝜃| 𝑥 ~𝑁( , )
𝑛+1 𝑛+1
σ𝑛
𝑖=1 𝑥𝑖 +𝑎
𝜃| 𝑥 − 𝑛+1
∴ 1 ~𝑁(0,1)
𝑛+1
15
 Bayesian Interval Estimation

σ𝑛 𝑥 +𝑎
𝜃| 𝑥 − 𝑖=1 𝑖
𝑛+1
𝑃(−𝑧1−𝛼 < 1 < 𝑧1−𝛼 ) = 1 − 𝛼
2 𝑛+1 2

1 σ𝑛
𝑖=1 𝑥𝑖 +𝑎 1
𝑃(− 𝑧 𝛼 < 𝜃| 𝑥 − < 𝑧 𝛼) =1−𝛼
𝑛+1 1− 2 𝑛+1 𝑛+1 1− 2

∴ The 1 − 𝛼 100% Bayesian interval for 𝜃| 𝑥 is given by

σ𝑛
𝑖=1 𝑥𝑖 +𝑎 1 σ𝑛
𝑖=1 𝑥𝑖 +𝑎 1
𝑃( − 𝑧 𝛼 < 𝜃| 𝑥 < + 𝑧 𝛼) =1−𝛼
𝑛+1 𝑛+1 1− 2 𝑛+1 𝑛+1 1− 2

16