community project
encouraging academics to share statistics support resources
All stcp resources are released under a Creative Commons licence
Statistical Methods
12. Multiple Linear
Regression and Analysis
of Covariance
Based
on
materials
provided
by
Coventry
University
and
Loughborough
University
under
a
Na9onal
HE
STEM
Programme
Prac9ce
Transfer
Adopters
grant
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Workshop outline
q Multiple Linear Regression:
Ø Two independent variables
Ø Multicollinearity: VIF and tolerance
Ø More than two independent variables:
o Direct variable entry method
o Backwards regression method
Ø Robustness
q Analysis of Covariance
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Please note
q This workshop assumes knowledge of simple
linear regression – see Workshop 11
q Some disciplines have a different culture in
applying multiple linear regression without
assumption checking – please seek guidance
from your faculty
q Most people want to look for significance for
deciding which variables to include in the
model, not for the purpose of prediction
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Multiple Linear Regression with
two independent variables
Model:
y
=
b0
+
b1x
+
b2z
Where:
q y is the dependent variable
q x and z are the independent variables
q b0 is the intercept coefficient
q b1 and b2 are the slope coefficients
Goal: To minimise the sum of the squares of the
errors
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
http://cast.massey.ac.nz/core/index.html?book=general, Section 6.3.7
Least squares estimation of y against x and z
yi
=
b0
+
b1xi
+
b2zi
+
ei
Choose
b0,
b1
and
b2
such
that
∑𝐢=𝟏↑𝐧▒
𝐞↓𝐢 ↓↑𝟐
is
minimised
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Example 1: Monthly sales
figures for women’s clothing
q 120 monthly sales figures for a catalogue-based mail
ordering company from January 1989 to December 1998
q Independent variables:
Ø Number of phone lines open for ordering
Ø Amount spent on print advertising
q Open Sheet1 of the Excel file CatalogueData.xlsx
associated with this workshop
q Turn it into an SPSS
data file – for the Date
field, use the data type
“Date” and the format
“dd-mmm-yy”
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
The regression analysis
process
q Step 1: Get to know your data
q Step 2: Formulate a model and check the
assumptions
q Step 3: Fit the model to the data
q Step 4: Report, interpret, and use the model
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 1A: Scatter plot of Sales
against NoPhoneLines
q Relationship Decimal places
appears to be removed
linear
q Variance in Font size
Sales appears increased to 10
to be constant
for different Font size
increased
values of
to 12
NoPhoneLines
q ‘Cigar shaped’
Minimum value changed to 0
data set
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 1B: Scatter plot of Sales
against PrintAdvertising
q Relationship
appears to be
linear
q Variance in
Sales appears
to be constant
for different
values of
PrintAdvertising
q ‘Cigar shaped’
data set
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 2: Formulate a model
Sales = b0 + b1×NoPhoneLines + b2×PrintAdvertising
Note: A model is always an approximation to the
data
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 2: Assumptions of
Multiple Linear Regression
q The observations of the dependent variable are
independent, e.g. they are not time or sequence dependent
q The independent variables are normally distributed or
binary
q The dependent variable is normally distributed for each
value of each predictor (independent) variable
q The variability of the outcome variable is the same for each
value of the predictor variable
q The dependent variable varies linearly as the independent
variables vary
q All seem to be OK for our data set (as indicated by the
‘cigar shaped’ scatter plots)
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 3: Fit the model to the data
q Analyze > Regression > Linear
q Add Sales as the dependent variable
q Add NoPhoneLines and PrintAdvertising as the
independent variables
Select
Statistics…
and choose
Collinearity
diagnostics
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Adjusted R
Square = 0.383 ⇒
model explains
38.3% of the
variation in Sales b0 = -21366 b1 = 653.55 b2 = 1.374
b0, b1 and b2 all significantly Tolerance coefficients slightly
different from 0 less than 1, VIF slightly more
than 1 (see later)
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 4: Fitted model
Sales = -21366 + 653.55×NoPhoneLines
+ 1.374×PrintAdvertising
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Multicollinearity
q Multicollinearity means there are high correlations
between the predictor (independent) variables
q Thus two or more predictor variables carry similar
information about the outcome variable
q With multicollinearity there is very high uncertainty for
the regression coefficients
q Multicollinearity can be assessed informally by looking
at the bivariate correlations between the independent
variables (any r > 0.8 indicates a possible problem)
q There are two formal measures of multicollinearity:
Ø Variance Inflation Factor (VIF)
Ø Tolerance
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Variance Inflation Factor
q Based on fitting predictor variables to other
predictor variables (i.e. NoPhoneLines to
PrintAdvertising for our example) and
calculating R2
q Values of VIF (Variance Inflation Factor):
Ø VIF < 5: don’t worry
Ø 5 < VIF < 10: multicollinearity may be a problem, be
cautious
Ø VIF > 10: multicollinearity is definitely a problem and
will adversely affect results
Source: Myers, R. (1990) Classical and modern
regression with applications. 2nd ed. Boston,
MA: Duxbury
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Tolerance
q The percentage of the variance in a predictor
variable that cannot be explained by the
other predictors
q In our example the tolerance for each
variable was 0.992, meaning 99.2% of the
variance in both predictor variables cannot
be explained by the other variables
q Tolerance is the reciprocal of VIF – so you
only need to look at VIF
q For our example, VIF was 1.009 for both
variables so it was not a problem
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Multiple Linear Regression with
>2 independent variables
Model:
y
=
b0
+
b1x1
+
b2x2
+
…
+
bmxm
Where:
q y
is
the
dependent
variable
q x1,
x2,
…,
xm
are
the
independent
variables
q b0
is
the
intercept
coefficient
q b1,
b2,
…,
bm
are
the
slope
coefficients
Goal:
To
minimise
the
sum
of
the
squares
of
the
errors
Rule
of
thumb:
For
a
sample
size
of
n,
use
no
more
than
√n
independent
variables
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Issues with >2 variables
q Much more likely to have multicollinearity, or
variables with non-significant coefficients
q Impossible to know in advance which variables are
best removed
⇒ Use a systematic variable selection method in
SPSS to determine an adequate model
⇒ Justify any decision you make
q We recommend backwards removal of predictor
variables:
Ø All predictor variables initially included
Ø Least significant variable removed
Ø Repeat process until least significant variable is below a
threshold (default is 0.1)
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Example 2: Monthly sales
figures for women’s clothing
q Independent variables:
Ø Number of phone lines open for ordering
Ø Amount spent on print advertising
Ø Number of catalogues mailed
Ø Number of pages in catalogue
Ø Number of customer service representatives
q Open Sheet2 of the Excel file
Catalogue2Data.xlsx associated with this
workshop
q Turn it into an SPSS data file
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 1A: Sales v. NoMailed
q Clear linear
relationship
q May be
heteroscedastic
– variance in
errors seems to
depend on
NoMailed
q This looks like
an outlier –
reason?
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Activity
q Create a new variable called NoMailedGroup by
recoding NoMailed into a new variable
q Choose a suitable cut-off value
q NoMailedGroup = 1 below the cut-off value
q NoMailedGroup = 2 above the cut-off value
q Run a linear regression of Sales against NoMailed
and choose Unstandardised residuals under
Save…
q Run an independent samples t-test of the
unstandardised residuals against NoMailedGroup
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
NoMailed cut-off = 12,500
q Levene’s test for equality of variances returns a non-
significant result
q Only 8 data values in NoMailedGroup 2
q Probably OK to assume homoscedasticity
q There is no specific test for heteroscedasticity in SPSS
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 1B: Sales v. NoPages
q Seems to
be a weak
linear
relationship
q Variance
seems to be
OK
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 1C: Sales v.
NoServiceReps
q Stronger
linear
relationship
q Data seems
to be in two
distinct
groups –
month
dependent?
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Activity
q Create a new scale variable called Month
q Enter 1 for a date in January, 2 for a date in
February, etc.
q Create a scatter
plot of Sales
against Month
q Clearly lower in
January and
February
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
q Recode Month into a
new variable called
MonthGroup with 1 =
Jan/Feb and 2 =
otherwise
q Add values to
MonthGroup to
represent these
groups
q Create a grouped
scatter plot of Sales v.
NoServiceReps with
MonthGroup as the
grouping variable
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
q Most of the
lower Sales
were from
Jan/Feb
q We could
add an
additional
variable to
the model
which is 1 for
Jan/Feb and
0 for other
months
q For the moment the months Jan/Feb are excluded
q Also see ANCOVA analysis later
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 2: Formulate a model
For the months March – December:
Sales = b0 + b1×NoPhoneLines + b2×PrintAdvertising
+ b3×NoMailed + b4×NoPages + b5×NoServiceReps
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Step 4: Fit the model
q Select only the cases with MonthGroup = 2
q Analyze > Regression > Linear
q Select Sales as the dependent variable and the other 5
variables and the independent variables
Under Statistics…
select collinearity
diagnostics
Select the
method as
Backward
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
q Only one model
required
q R2adj = 0.795 ⇒ model
accounts for 79.5% of
the variation in Sales
q All variables included in initial model (backwards method)
q No variable removed because all the probability values < 0.1
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Activity
q Repeat the analysis with all the months included
q What affect does this have on the models?
q Analysis now has 2
models
q R2adj slightly higher
in second model
q Both markedly
lower than previous
analysis
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
q NoPhoneLines was removed from Model 2 because its
probability value in Model 1 > 0.1 (the absolute value of its
standardised coefficient was low)
q Model 2 has very low probability values for all the variables
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Robustness exceptions
q Homoscedasticity is mandatory – otherwise use a
nonparametric technique
q Linearity is mandatory – otherwise transform the
independent variable or use another model, e.g.
quadratic or polynomial
q Normality is “not necessary for the least-squares
fitting of the regression model, but it is required in
general for inference making.” (e.g. calculating the p-
values and confidence intervals of the coefficients)
“…only extreme departures of the distribution of Y
from normality yield spurious results.”
Source: (Kleinbaum et al., 2008: 120)
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Application of robustness
exceptions to activity
q Sales v. NoServiceReps was not normally distributed
⇒ The probability values of the coefficients may not be reliable
q However, the probability value was not borderline so the
model can still be used
q Including January & February data just increases the ‘noise’
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Example 3: Monthly catalogue
sales figures for jewellery
q Independent variables:
Ø Number of phone lines open for ordering
Ø Amount spent on print advertising
Ø Number of catalogues mailed
Ø Number of pages in catalogue
Ø Number of customer service representatives
q Open Sheet3 of the Excel file
CatalogueData.xlsx associated with this
workshop
q Turn it into an SPSS data file
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Activity
q Use JewellerySales as the dependent variable
q Fit the model with all 5 independent or predictor
variables (direct or ‘enter’ method)
q Is multicollinearity important?
q Just re-fitting the model with only the variables
with significant coefficients may not be sufficient
q Try also the backward variable selection
method
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Five independent variable direct model
Three independent variable direct model
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Results of backwards
regression method
R2adj does not
necessarily
reduce when
variables are
removed from
the model
Final model removes NoServiceReps
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Analysis of Covariance
(ANCOVA)
q Combines ANOVA and Linear Regression:
Ø ANOVA: Explains outcome for different groups of the
data
Ø Linear Regression: Explains outcome with
explanatory variables
Ø ANCOVA: Does both simultaneously
q Increases the precision of the analysis
q Compares the means at the average values of
the predictor variables
q The predictor (independent) variables must be
correlated to the outcome (dependent) variable
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Difference Average Difference of means adjusted
of means Covariate to average covariate
Outcome variable
Group 1
Group 2
Covariate variable
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Assumptions for ANCOVA
q Same as ANOVA and Linear Regression:
Ø Independence of observations
Ø Equality of variances
Ø Normality of distribution
q In addition:
Ø Equal regression slopes – if this assumption is in
doubt, add an extra binary variable to the model to
represent the two groups
q ANCOVA models can be very complex
q Seek advice if you feel ANCOVA is appropriate
for your research
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Example of ANCOVA
q Open the SPSS file Catalogue2 you created
earlier
q We want to compare Sales in January and
February against the rest of the year
q The independent variable is NoServiceReps
q Use Analyze > General Linear Model >
Univariate:
Ø Dependent variable: Sales
Ø Fixed Factor: MonthGroup
Ø Covariate: NoServiceReps
Ø Under Options… choose Parameter estimates
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Model
accounts for
71.4% of the
total variance
The
MonthGroup
=1 coefficient
should be
added to the
intercept
coefficient for
this group
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Add a fitted line to a grouped
scatter plot
q Open the chart
editor
q Select the Add a
reference line from
Equation tool
q Enter the Custom
Equation from the
ANCOVA output
q Change the line
colour using the
Lines tab
q Repeat for the other month group by subtracting the
ANCOVA coefficient from the constant
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Scatter plot with fitted lines
Clearly an
improvement
on Simple
Linear
Regression
but the best
fit slope for
the two
groups is
slightly
different
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Activity
q Open the file AlligatorData.xlsx associated with
this Workshop
q The data file contains pelvic canal width,
snout-vent length and the gender of 35
alligators
q Create a scatterplot for PelvicWidth against
SnoutLength for the different Gender groups
q Run an ACOVA model for PelvicWidth against
SnoutLength for the different Gender groups
q Plot the ANCOVA model on the scatterplot
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Scatterplot of alligator data
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
ANCOVA for alligator data
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Plot of ANCOVA on scatterplot
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Recap
q We have introduced Multiple Linear Regression by
adding one predictor variable then several predictor
variables to Simple Linear Regression
q New diagnostics were required in order to address
the possibility of multicollinearity, namely VIF or
Tolerance
q Better to use a systematic method for introducing or
removing variables, such as Backwards
q Robustness arguments are similar to those in
Simple Linear Regression
q Analysis of Covariance (ANCOVA) is a combination
of ANOVA and Linear Regression
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
Bibliography
Bovas, A. & Ledolter, J. (2006) Introduction to Regression Modelling.
Belmont, CA: Thomson Brooks/Cole.
Field, A. (2013) Discovering Statistics using SPSS: (And sex and drugs
and rock 'n' roll), 4th ed., London: SAGE, Sections 8.5 - 8.7 and
Chapter 12.
Kleinbaum, D., Kupper L., Muller, K. and Nizam, A. (2008) Applied
Regression Analysis and Other Multivariable Methods. 4th ed.
Belmont, CA: Thomson Brooks/Cole.
Kutner, M., Nachtsheim, C. and Neter, J. (2004) Applied Linear
Regression Models. 4th ed., Irwin: McGraw-Hill.
Myers, R. (1990) Classical and Modern Regression with Applications. 2nd
ed. Boston, MA: Duxbury.
statstutor (n. d.) Multiple Regression resources. Available at:
http://www.statstutor.ac.uk/topics/regression-and-model-building/
multiple-regression/ [Accessed 8/01/14].
Stirling, W. D. (2013) Welcome to the General CAST e-book. Available at:
http://cast.massey.ac.nz/core/index.html?book=general [Accessed
8/01/14], Section 6.3.7.
Peter
Samuels
Reviewer:
Ellen
Marshall
www.statstutor.ac.uk
Birmingham
City
University
University
of
Sheffield
