Analysis of Variance and Design g of ExperimentExperiment p -I

MODULE II
LECTURE - 5

GENERAL LINEAR HYPOTHESIS AND ANALYSIS OF VARIANCE

Dr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur

Analysis of variance
Analysis of variance is a body of statistical methods of analyzing the measurements assumed to be structured as

yi = 1 xi1 + 2 xi 2 + ... + p xip + i , i = 1, 2,..., n

where

xij

are integers, generally 0 or 1 indicating usually the absence or presence of effects

j ; and i s are assumed to

be identically and independently distributed with mean 0 and variance 2 . It may be noted that the i s can be assumed additionally to follow a normal distribution N (0, 2 ). It is needed for the maximum likelihood estimation of parameters from the beginning of analysis but in the least squares estimation, it is needed only when conducting the tests of hypothesis and the confidence interval estimation of parameters. parameters The least squares method does not require any knowledge of distribution like normal upto the stage of estimation of parameters. We need some basic concepts to develop the tools.

Least squares estimate of

of for which Let y1 , y2 ,..., yn be a sample of observations on Y1 , Y2 ,..., Yn . The least squares estimate of is the values

the sum of squares due to errors, i.e.,

S 2 = i2 = ' = ( y X )( y X )
i =1

= yy 2 X ' y + X X

y = ( y1 , y2 ,..., yn ) . Differentiating i minimum is i i where h Diff i i S2 with i h respect to and d substituting b i i i it to b be zero, the h normal l
equations are obtained as

dS 2 = 2 X X 2 X y = 0 d
or X X = X y.

If X has full rank then ( X X ) has a unique inverse and the unique least squares estimate of is

= ( X X ) 1 X y
which is the best linear unbiased estimator of in the sense of having minimum variance in the class of linear and unbiased

) estimator If rank of X is not full, estimator. full then generalized inverse is used for finding the inverse of ( X X ).
If L is a linear parametric function where L = ( 1 , 2 ,..., p ) is a non-null vector, then the least squares estimate of L
. is L

L admits A question ti arises i th t what that h t are the th conditions diti under d which hi h a linear li parametric t i function f ti d it a unique i l least t
squares estimate in the general case.

The concept of estimable function is needed to find such conditions.

Estimable functions
A linear function of the parameters with known is said to be an estimable parametric function (or estimable) if there exists a linear function L Y of Y such that

E ( LY ) = for all Rb .

Note that not all parametric functions are estimable.

Following results will be useful in understanding the further topics.

Theorem 1
A linear parametric function L admits a unique least squares estimate if and only if L is estimable.

Th Theorem 2 (Gauss (G M k ff theorem) Markoff h )

where is a solution of X X = X Y If the linear parametric function L is estimable then the linear estimator L

g minimum variance in the class of all linear and unbiased is the best linear unbiased estimater of L in the sense of having estimators of L .

Theorem 3
If the linear parametric function is also estimable.

1 = l1' , 2 = l2' ,..., k = lk'

are estimable, then any linear combination of

1 , 2 ,..., k

Theorem 4
All linear parametric functions in are estimable if and only if X has full rank rank.

If X is not of full rank, then some linear parametric functions do not admit the unbiased linear estimators and nothing can be inferred about them. The linear parametric functions which are not estimable are said to be confounded. A possible solution to this problem is to add linear restrictions on so as to reduce the linear model to a full rank.

Theorem 5
' ' be their least squares estimators. Then and L'2 be two estimable parametric functions and let L1 and L'2 Let L1

' ' Var ( L1 ( X X ) 1 L1 ) = 2 L1 , L' ) = 2 L' ( X X ) 1 L Cov ( L' 1 2 1

X can be used in place of unique inverse. assuming that X is a full rank matrix. If not, the generalized inverse of X

6 q estimation 2 based on least squares

Estimator of

Consider an estimator of 2 as 1 )( y X ) 2 = (y X n p

1 [ y X ( X X ) 1 X ' y ][ y X ( X X ) 1 X y ] n p 1 y [ I X ( X X ) 1 X ][ I X ( X X ) 1 X ] y = n p 1 = y [ I X ( X X ) 1 X ] y n p =

X ) X ] is an idempotent matrix with its trace as where the hat matrix [ I X ( X

1

tr [ I X ( X X ) 1 X '] = trI trX ( X X ) 1 X = n tr t ( X X ) 1 X X (using i th the result lt tr t ( AB ) = tr t ( BA)) = n tr I p = n p.

Note that, using E ( y Ay ) = ' A + tr ( A ), we have

2) = E (

2
n p

tr[ I X ( X X ) 1 X ]

= 2
and so

2 is an unbiased estimator of 2 .

Maximum likelihood estimation

The least square method does not uses any distribution of the random variables in the estimation of parameters. We need the distributional assumption in case of least squares only while constructing the tests for hypothesis and the confidence

intervals. For maximum likelihood estimation, we need the distributional assumption from the beginning. Suppose y1 , y2 ,..., yn are independently and identically distributed following a normal distribution with mean E ( yi ) = j xij and variance Var ( yi ) = 2 (i = 1, 2,, n). Then the likelihood function of
p

y1 , y2 ,..., yn is

j =1

L( y | , 2 ) =

1 (2 ) ( )
n 2 n 2 2

1 exp 2 ( y X )( y X ) 2

where

y = ( y1 , y2 ,..., .yn. ). Then

n n 1 L = ln L ( y | , 2 ) = log 2 log 2 ( y X )( y X ). 2 2 2 2

Differentiating the log likelihood with respect to and 2 , we have

L = X y, = 0 X X 1 L 2 )( y X ). = 0 = (y X ) 2 n

8
Assuming the full rank of X, the normal equations are solved and the maximum likelihood estimators are obtained as

= ( X X ) 1 X y )( y X ) 2 = (y X 1 n 1 = y I X ( X X ) 1 X y. n

and The second order differentiation conditions can be checked and they are satisfied for 2 to be the maximum
likelihood estimators.
and is same as the least squares estimator Note that in the maximum likelihood estimator

is an unbiased estimator of , i.e., E ( ) = like the least squares estimator but n p 2 2) = 2 is not an unbiased estimator of 2 , i.e., E ( 2 unlike the least squares estimator. n Now we use the following theorems for developing the test of hypothesis.

Theorem 6
Let

Y = (Y1 , Y2 ,..., Yn )

follow a multivariate normal distribution N ( , ) with mean vector and positive definite covariance

matrix . Then Y AY follows a noncentral chi chi-square square distribution with p degrees of freedom and noncentrality parameter

A , i.e., 2 ( p, A ) if and only if A is an idempotent matrix of rank p.

Theorem 7
Let Y = (Y1 , Y2 ,..., Yn ) follows a multivariate normal distribution N (, ) with mean vector and positive definite covariance 2 2 matrix . Let Y AY follows ( p1 , A1 ) and Y A2Y follows ( p2 , A2 ). Then Y AY and Y A2Y are independently distributed if 1 1

A1 A2 = 0.

Theorem 8
Let Y = (Y1 , Y2 ,..., Yn ) follow a multivariate normal distribution N ( , 2 I ), then the maximum likelihood (or least squares)

follow N L , L( X X )1 L of estimable linear parametric function is independently distributed of 2 ; L estimator L

and

2 n

2 follows (n p) where rank(X) = p.

= ( X X ) 1 X Y , then Proof: Consider

) = L( X X ) 1 X E (Y ) E ( L = L( X X ) 1 X X = L ) = LVar ( )L Var ( L )( )L = LE ( = 2 L( X X ) 1 L.
is a linear function of , so L follows a normal distribution N L , 2 L( X X ) 1 L . is a linear function of y and L Since 1 1 1 Let A = I X ( X X ) X and B = L '( X X ) X , then L = L ( X X ) X Y = BY

and

2 = (Y X ) ' n X )1 X I X ( X (Y X ) = Y ' AY .

2 follows a 2 (n p) . Also So, using Theorem 6 with rank(A) = n p, n

BA = L( X X )1 X L( X X )1 X X ( X X )1 X = 0. 0 So using Theorem 7, Y ' AY and Y ' BY are independently distributed.