Page 1
2018, Study Session # 3, Reading # 10
“MULTIPLE REGRESSION AND ISSUES IN REGRESSION ANALYSIS”
MSR = Mean Regression Sum of Squares = Critical F taken from F
MSE = Mean Squared Error 1. INTRODUCTION Distribute Table
RSS = Regression Sum of Squares = Null Hypothesis
SSE = Sum of Squared Errors/Residuals ∝ = Alternative Hypothesis
α = Level of Significance Multiple linear regression models are more X = Independent Variable
sophisticated. Y = Dependent Variable
They incorporate more than one independent F = F Statistic (calculated)
variable.
2. MULTIPLE LINEAR REGRESSIONS
Allows determining effects of more than one independent variable on a
particular dependent variable
= + + + ⋯ +
Tells the impact on Y by changing X1 by 1 unit keeping other independent
variables same.
Individual slope coefficients (e.g. b1) in multiple regressions known as partial
regression/slope coefficients.
2.1 Assumption of the Multiple Linear Regression Model
Relationship b/w Y and , , , … is linear.
Independent variables are not random and no exact linear relationship exists
b/w 2 or more independent variables.
Expected value of error terms is 0.
Variance of error term is same for all observations.
Error term is uncorrelated across observations.
Error term is normally distributed.
2.2 Predicting the Dependent Variable in a Multiple Regression Model
Obtain estimates of regression parameters.
= ^ , ^ , ^ , … ^
= , , , …
Determine assumed values of , …
Compute predicted value of using = + + + ⋯ +
To predict dependent variable:
Be confident that assumptions of the regression are met.
Predictions regarding X must be within reliable range of data used to estimate the model.
2.3 Testing Whether All Population Regression Coefficients Equals Zero
⇒ All slope coefficients are simultaneously = 0, none of the X
variable helps explain Y.
To test F-test is used.
T-test cannot be used.
=
/
=
/(
())
Copyright © FinQuiz.com. All rights reserved.
Page 2
2018, Study Session # 3, Reading # 10
2.3 Testing Whether All Population Regression Coefficients Equals Zero
Where
= −
= −
n = no. of observation
k = no. of slope coefficients
Decision rule ⇒ reject if F > FC (for given α).
It is a one-tailed test.
df numerator =k
df denominator =n-(k+1).
For k and n the test statistic representing H0, all slope coefficients are
equal to 0, is ,
()
In F-distribution table ,
() where K represents column and n-
(k+1) represents row.
Significance of F in ANOVA table represents ‘p value’.
F-statistic chances of Type I error.
2.4 Adjusted R2
R2 with addition of independent variables (X) in regression
= 1 − 1 − !.
When k ≥ 1 ⇒ >
can be –ve but R2 is always +ve.
If is used for comparing regression models.
Sample size must be the same
Dependent variable is defined in the same way.
Does not necessarily indicate regression is well specified.
3. USING DUMMY VARIABLES IN REGRESSION
Dummy variable ⇒ takes 1 if particular condition is
true & 0 when it is false.
Diligence is required in choosing no. of dummy
variables.
Usually n-1 dummy variables are used
where n= no. of categories.
Copyright © FinQuiz.com. All rights reserved.
Page 3
2018, Study Session # 3, Reading # 10
4. VIOLATIONS OF REGRESSION ASSUMPTIONS
4.1 Heteroskedasticity
Variance of errors differs across observations ⇒ heteroskedastic
Variance of errors is similar across observations ⇒ homoskedastic
Usually no systematic relationship exists b/w X & regression residuals.
If systematic relationship is present ⇒ heteroskedasticity can exist.
4.1.1 The Consequence of Heteroskedasticity
It can lead to mistake in inference.
Does not affect consistency.
F-test becomes unreliable.
Due to biased estimators of standard errors, t-test also becomes unreliable.
Most likely result of heteroskedasticity is that the:
estimated standard errors will be underestimated.
t-statistic will be inflated.
Ignoring heteroskedasticity leads to significant relationship that does not exist actually.
It becomes more serious while developing investment strategy using regression analysis.
Unconditional heteroskedasticity ⇒ when heteroskedasticity of error variance is not correlated with
independent variables in the multiple regression.
Create major problems for statistical inference.
Conditional heteroskedasticity ⇒ when heteroskedasticity of error variance is correlated with the
independent variables.
It causes most problems.
Can be tested & corrected easily through many statistically software packages.
4.1.2 Testing for Heteroskedasticity
Breush-Pagan test is widely used.
Regression squared residuals of regression on independent variables.
Independent variables explain much of the variation of errors ⇒
conditional heteroskedasticity exists.
= no conditional heteroskedasticity exists.
= conditional heteroskedasticity exist
Under Breush-pagan test statistic = nR2
R2: from regression of squared residuals on X
Critical value ⇒ calculated χ2 distribution.
df = no. of independent variables
Reject if test-static > critical value.
Copyright © FinQuiz.com. All rights reserved.
Page 4
2018, Study Session # 3, Reading # 10
4.1.3 Correcting for Heteroskedasticity
Robust Standard Errors Generalized Least Squares
Corrects standard error of estimated Modify original equation.
coefficients. Requires economic expertise to
Also known as heteroskedasticity implement correctly on financial data.
consistent standards errors or white-
corrected standards errors.
4.2 Serial Correlation
Regression errors correlated across observations.
Usually arises in time-series regression.
4.2.1 The Consequences of Serial Correlation
Incorrect estimate of regression coefficient standard errors
Parameter estimates become inconsistent & invalid when Y is lagged onto X under serial
correlation.
Positive serial correlation ⇒ positive (negative) errors chance of positive (negative) errors
Negative serial correlation ⇒ positive (negative) errors chance of negative (positive) errors
It leads to wrong inferences
If positive serial correlation:
Standard errors underestimated
T-statistic & F-statistics inflated
Type-I error
If negative serial correlation
Standard errors overestimated
T-statistics & F-statistics understated
Type-II error
4.2.2 Testing for Serial Correlation
Variety of tests, most common → Durbin-Watson test
∑
"# =
షభమ
సమ
∑ మ
సభ
Where = regression residual for period t.
For large sample size Durbin-Watson statistic (d) is approximately
→DW ≈ 2(1-r)
→where r = sample correlation b/w regression residuals of t and t-1
Values of DW can range from 0 to 4.
DW = 2 ⇒ r=0 ⇒ no serial correlation.
DW = 0 ⇒ r=1 ⇒ perfectly positively serially correlated.
DW = 4 ⇒ r = -1 ⇒ perfectly negatively serially correlated.
For positive serial correlation:
⇒ No positive serial correlation
⇒ Positive serial correlation
"# < $ ⇒ reject
"# > ⇒ do not reject
dl ≤ "# ≤ ⇒ inconclusive.
Copyright © FinQuiz.com. All rights reserved.
Page 5
2018, Study Session # 3, Reading # 10
4.2.2 Testing for Serial Correlation
For negative serial correlation:
⇒ No negative serial correlation.
⇒ Negative serial correlation.
"# > 4 − $ ⇒ Reject .
"# < 4 − ⇒ do not reject
4 − ≤ "# ≤ 4 − $ ⇒ inconclusive.
4.2.3 Correcting for Serial Correlation
Adjust the coefficient standard errors. Modify regression equation.
→ Recommended method Extreme care is required.
Hansen’s method ⇒ most prevalent one. May lead to inconsistent parameters
estimates.
4.3 Multicollinearity
Occurs when two or more independent variables (X) are highly
correlated with each other.
Regression can be estimated but result becomes problematic.
Serious practical concern due to commonly found approximate linear
relation among financial variables.
4.3.1 The Consequences of Multicollinearity
Difficulty in detecting significant relationships.
Estimates become extremely imprecise & unreliable though consistency is unaffected.
F-statistic is unaffected.
Standard errors of regression can .
Causing insignificant t-tests
Wide confidence interval
Type II error
4.3.2 Detecting Multicollinearity
Multicollinearity is a matter of degree rather than the presence / absence.
Pair wise correlation does not necessarily indicate presence of Multicollinearity
Pair wise correlation does not necessarily indicate absence of Multicollinearity
With 2 independent variables ⇒ correlation is a useful indicator.
R2 significant, F-statistic significant, insignificant t-statistic on slope coefficients ⇒
classic symptom of Multicollinearity
4.3.3 Correcting Multicollinearity
Exclude one or more regression variables.
In many cases, experimentation is done to determine
variable causing Multicollinearity
Copyright © FinQuiz.com. All rights reserved.
Page 6
2018, Study Session # 3, Reading # 10
5. MODEL SPECIFICATION AND ERRORS IN SPECIFICATION
Model specification ⇒ set of variables included in
regression.
Incorrect specification leads to biased & inconsistent
parameters
5.1 Principles of Model Specification
Model grounded on economic reasoning.
Functional form of variables compatible with nature of variables
Parsimonious ⇒ each included variable should play an essential role
Model is examined for the violation of regression assumptions.
Model is tested for the validity & usefulness of the out of sample data.
5.2 Misspecified Functional Form
One or more variables are omitted. If omitted variable is correlated with
remaining variable, error term will also be correlated with the latter and
the:
result can be biased & inconsistent.
estimated standard errors of the coefficients will be inconsistent.
One or more variables may require transformation.
Pooling of data from different samples that should not be pooled.
Can lead to spurious results.
5.3 Times-Series Misspecification (Independent Variables Correlated with Errors)
Including lagged variables (dependent) as independent
with serial correlation.
Including a function of the dependent variable as an
independent variable.
Independent variables measured with error
5.4 Other Types of Time-Series Misspecification
Nonstationarity: variable properties, e.g. mean, are
not constant through time.
In practice nonstationarity is a serious problem.
Copyright © FinQuiz.com. All rights reserved.
Page 7
2018, Study Session # 3, Reading # 10
6. MODELS WITH QUALITATIVE DEPENDENT VARIABLES
Qualitative dependent variables ⇒ dummy variables used as dependent instead of
independent.
Probit model⇒ based on normal distribution estimates the probability:
of discrete outcome, given values of independent variables used to explain that
outcome.
that Y=1, implying a condition is met.
Logit model:
Identical to Probit model.
Based on logistic distribution.
Both Logit and Probit models must be estimated using maximum likelihood methods.
Discriminate analysis ⇒ can be used to create an overall score that is used for classification.
Qualitative dependent variable models can be used for portfolio management and business
management.
Copyright © FinQuiz.com. All rights reserved.
