Exercise 4 : Linear Regression
Essential Libraries
Let us begin by importing the essential Python Libraries.
NumPy : Library for Numeric Computations in Python
Pandas : Library for Data Acquisition and Preparation
Matplotlib : Low-level library for Data Visualization
Seaborn : Higher-level library for Data Visualization
# Basic Libraries
import numpy as np
import pandas as pd
import seaborn as sb
import matplotlib.pyplot as plt # we only need pyplot
sb.set() # set the default Seaborn style for graphics
Setup : Import the Dataset
Dataset from Kaggle : The "House Prices" competition
Source: https://www.kaggle.com/c/house-prices-advanced-regression-techniques
The dataset is train.csv; hence we use the read_csv function from Pandas.
Immediately after importing, take a quick look at the data using the head function.
houseData = pd.read_csv('train.csv')
houseData.head()
Id MSSubClass MSZoning LotFrontage LotArea Street Alley LotShape
\
0 1 60 RL 65.0 8450 Pave NaN Reg
1 2 20 RL 80.0 9600 Pave NaN Reg
2 3 60 RL 68.0 11250 Pave NaN IR1
3 4 70 RL 60.0 9550 Pave NaN IR1
4 5 60 RL 84.0 14260 Pave NaN IR1
LandContour Utilities ... PoolArea PoolQC Fence MiscFeature MiscVal
MoSold \
0 Lvl AllPub ... 0 NaN NaN NaN 0
2
1 Lvl AllPub ... 0 NaN NaN NaN 0
5
2 Lvl AllPub ... 0 NaN NaN NaN 0
9
3 Lvl AllPub ... 0 NaN NaN NaN 0
2
4 Lvl AllPub ... 0 NaN NaN NaN 0
12
YrSold SaleType SaleCondition SalePrice
0 2008 WD Normal 208500
1 2007 WD Normal 181500
2 2008 WD Normal 223500
3 2006 WD Abnorml 140000
4 2008 WD Normal 250000
[5 rows x 81 columns]
Problem 1 : Predicting SalePrice using GrLivArea
Extract the required variables from the dataset, as mentioned in the problem.
houseGrLivArea = pd.DataFrame(houseData['GrLivArea'])
houseSalePrice = pd.DataFrame(houseData['SalePrice'])
Plot houseSalePrice against houseGrLivArea using standard JointPlot.
trainDF = pd.concat([houseGrLivArea, houseSalePrice], axis =
1).reindex(houseGrLivArea.index)
sb.jointplot(data=trainDF, x='GrLivArea', y='SalePrice', height = 12)
<seaborn.axisgrid.JointGrid at 0x25b36d1db50>
Import the LinearRegression model from sklearn.linear_model.
# Import essential models and functions from sklearn
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
# Create a Linear Regression object
linreg = LinearRegression()
Prepare both the datasets by splitting in Train and Test sets.
Train Set with 1100 samples and Test Set with 360 samples.
# Split the dataset into Train and Test
#houseGrLivArea_train = pd.DataFrame(houseGrLivArea[:1100])
#houseGrLivArea_test = pd.DataFrame(houseGrLivArea[-360:])
#houseSalePrice_train = pd.DataFrame(houseSalePrice[:1100])
#houseSalePrice_test = pd.DataFrame(houseSalePrice[-360:])
# Split the Dataset into Train and Test
houseGrLivArea_train, houseGrLivArea_test, houseSalePrice_train,
houseSalePrice_test = train_test_split(houseGrLivArea, houseSalePrice,
test_size = 360/1460)
# Check the sample sizes
print("Train Set :", houseGrLivArea_train.shape,
houseSalePrice_train.shape)
print("Test Set :", houseGrLivArea_test.shape,
houseSalePrice_test.shape)
Train Set : (1100, 1) (1100, 1)
Test Set : (360, 1) (360, 1)
Fit Linear Regression model on houseGrLivArea_train and houseSalePrice_train
linreg.fit(houseGrLivArea_train, houseSalePrice_train)
LinearRegression()
Visual Representation of the Linear Regression Model
Check the coefficients of the Linear Regression model you just fit.
print('Intercept \t: b = ', linreg.intercept_)
print('Coefficients \t: a = ', linreg.coef_)
Intercept : b = [16608.46906887]
Coefficients : a = [[108.93750382]]
Plot the regression line based on the coefficients-intercept form.
# Formula for the Regression line
# Alternative code for below: regline_x =
houseGrLivArea_train.to_numpy()
regline_x = houseGrLivArea_train
regline_y = linreg.intercept_ + linreg.coef_ * regline_x
# Plot the Linear Regression line
f = plt.figure(figsize=(16, 8))
plt.scatter(houseGrLivArea_train, houseSalePrice_train)
plt.plot(regline_x.to_numpy(), regline_y.to_numpy(), "r-", linewidth =
3)
plt.show()
Prediction of Response based on the Predictor
Predict SalePrice given GrLivArea in the Test dataset.
# Predict SalePrice values corresponding to GrLivArea
houseSalePrice_test_pred = linreg.predict(houseGrLivArea_test)
# Plot the Predictions
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(houseGrLivArea_test, houseSalePrice_test, color = "green")
plt.scatter(houseGrLivArea_test, houseSalePrice_test_pred, color =
"red")
plt.show()
Goodness of Fit of the Linear Regression Model
Check how good the predictions are on the Train Set.
Metric : Explained Variance or R^2 on the Train Set.
print("Explained Variance (R^2) \t:",
linreg.score(houseGrLivArea_train, houseSalePrice_train))
Explained Variance (R^2) : 0.5083530900052877
Check how good the predictions are on the Test Set.
Metric : Explained Variance or R^2 on the Test Set.
print("Explained Variance (R^2) \t:",
linreg.score(houseGrLivArea_test, houseSalePrice_test))
Explained Variance (R^2) : 0.4778441365777981
You should also try the following
• Convert SalePrice to log(SalePrice) in the beginning and then use it for
Regression
Code : houseSalePrice =
pd.DataFrame(np.log(houseData['SalePrice']))
• Perform a Random Train-Test Split on the dataset before you start with the
Regression
Note : Check the preparation notebook M3 LinearRegression.ipynb for the code
Problem 2 : Predicting SalePrice using LotArea
Extract the required variables from the dataset, as mentioned in the problem.
housePredictor = pd.DataFrame(houseData['LotArea'])
houseSalePrice = pd.DataFrame(houseData['SalePrice'])
trainDF = pd.concat([housePredictor, houseSalePrice], axis =
1).reindex(housePredictor.index)
sb.jointplot(data=trainDF, x='LotArea', y='SalePrice', height = 12)
<seaborn.axisgrid.JointGrid at 0x25b374c2e50>
Linear Regression on SalePrice vs Predictor
# Import LinearRegression model from Scikit-Learn
from sklearn.linear_model import LinearRegression
# Split the dataset into Train and Test
#housePredictor_train = pd.DataFrame(housePredictor[:1100])
#housePredictor_test = pd.DataFrame(housePredictor[-360:])
#houseSalePrice_train = pd.DataFrame(houseSalePrice[:1100])
#houseSalePrice_test = pd.DataFrame(houseSalePrice[-360:])
# Split the Dataset into Train and Test
housePredictor_train, housePredictor_test, houseSalePrice_train,
houseSalePrice_test = train_test_split(housePredictor, houseSalePrice,
test_size = 360/1460)
# Create a Linear Regression object
linreg = LinearRegression()
# Train the Linear Regression model
linreg.fit(housePredictor_train, houseSalePrice_train)
LinearRegression()
Visual Representation of the Linear Regression Model
print('Intercept \t: b = ', linreg.intercept_)
print('Coefficients \t: a = ', linreg.coef_)
# Formula for the Regression line
# Alternative code for below: regline_x =
housePredictor_train.to_numpy()
regline_x = housePredictor_train
regline_y = linreg.intercept_ + linreg.coef_ * regline_x
# Plot the Linear Regression line
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_train, houseSalePrice_train)
plt.plot(regline_x.to_numpy(), regline_y.to_numpy(), 'r-', linewidth =
3)
plt.show()
Intercept : b = [162100.36069809]
Coefficients : a = [[1.83081318]]
Prediction of Response based on the Predictor
# Predict SalePrice values corresponding to GrLivArea
houseSalePrice_test_pred = linreg.predict(housePredictor_test)
# Plot the Predictions
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_test, houseSalePrice_test, color = "green")
plt.scatter(housePredictor_test, houseSalePrice_test_pred, color =
"red")
plt.show()
Goodness of Fit of the Linear Regression Model
print("Explained Variance (R^2) on Train Set \t:",
linreg.score(housePredictor_train, houseSalePrice_train))
print("Explained Variance (R^2) on Test Set \t:",
linreg.score(housePredictor_test, houseSalePrice_test))
Explained Variance (R^2) on Train Set : 0.062378129532616566
Explained Variance (R^2) on Test Set : 0.08979920486099247
Problem 2 : Predicting SalePrice using TotalBsmtSF
Extract the required variables from the dataset, as mentioned in the problem.
housePredictor = pd.DataFrame(houseData['TotalBsmtSF'])
houseSalePrice = pd.DataFrame(houseData['SalePrice'])
trainDF = pd.concat([housePredictor, houseSalePrice], axis =
1).reindex(housePredictor.index)
sb.jointplot(data=trainDF, x='TotalBsmtSF', y='SalePrice', height =
12)
<seaborn.axisgrid.JointGrid at 0x25b384d3a60>
Linear Regression on SalePrice vs Predictor
# Import LinearRegression model from Scikit-Learn
from sklearn.linear_model import LinearRegression
# Split the dataset into Train and Test
#housePredictor_train = pd.DataFrame(housePredictor[:1100])
#housePredictor_test = pd.DataFrame(housePredictor[-360:])
#houseSalePrice_train = pd.DataFrame(houseSalePrice[:1100])
#houseSalePrice_test = pd.DataFrame(houseSalePrice[-360:])
# Split the Dataset into Train and Test
housePredictor_train, housePredictor_test, houseSalePrice_train,
houseSalePrice_test = train_test_split(housePredictor, houseSalePrice,
test_size = 360/1460)
# Create a Linear Regression object
linreg = LinearRegression()
# Train the Linear Regression model
linreg.fit(housePredictor_train, houseSalePrice_train)
LinearRegression()
Visual Representation of the Linear Regression Model
print('Intercept \t: b = ', linreg.intercept_)
print('Coefficients \t: a = ', linreg.coef_)
# Formula for the Regression line
# Alternative code for below: regline_x =
housePredictor_train.to_numpy()
regline_x = housePredictor_train
regline_y = linreg.intercept_ + linreg.coef_ * regline_x
# Plot the Linear Regression line
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_train, houseSalePrice_train)
plt.plot(regline_x.to_numpy(), regline_y.to_numpy(), 'r-', linewidth =
3)
plt.show()
Intercept : b = [66517.88779439]
Coefficients : a = [[107.22623803]]
Prediction of Response based on the Predictor
# Predict SalePrice values corresponding to GrLivArea
houseSalePrice_test_pred = linreg.predict(housePredictor_test)
# Plot the Predictions
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_test, houseSalePrice_test, color = "green")
plt.scatter(housePredictor_test, houseSalePrice_test_pred, color =
"red")
plt.show()
Goodness of Fit of the Linear Regression Model
print("Explained Variance (R^2) on Train Set \t:",
linreg.score(housePredictor_train, houseSalePrice_train))
print("Explained Variance (R^2) on Test Set \t:",
linreg.score(housePredictor_test, houseSalePrice_test))
Explained Variance (R^2) on Train Set : 0.37367019826026027
Explained Variance (R^2) on Test Set : 0.38123042547864716
Problem 2 : Predicting SalePrice using GarageArea
Extract the required variables from the dataset, as mentioned in the problem.
housePredictor = pd.DataFrame(houseData['GarageArea'])
houseSalePrice = pd.DataFrame(houseData['SalePrice'])
trainDF = pd.concat([housePredictor, houseSalePrice], axis =
1).reindex(housePredictor.index)
sb.jointplot(data=trainDF, x='GarageArea', y='SalePrice', height = 12)
<seaborn.axisgrid.JointGrid at 0x25b39697280>
Linear Regression on SalePrice vs Predictor
# Import LinearRegression model from Scikit-Learn
from sklearn.linear_model import LinearRegression
# Split the dataset into Train and Test
#housePredictor_train = pd.DataFrame(housePredictor[:1100])
#housePredictor_test = pd.DataFrame(housePredictor[-360:])
#houseSalePrice_train = pd.DataFrame(houseSalePrice[:1100])
#houseSalePrice_test = pd.DataFrame(houseSalePrice[-360:])
# Split the Dataset into Train and Test
housePredictor_train, housePredictor_test, houseSalePrice_train,
houseSalePrice_test = train_test_split(housePredictor, houseSalePrice,
test_size = 360/1460)
# Create a Linear Regression object
linreg = LinearRegression()
# Train the Linear Regression model
linreg.fit(housePredictor_train, houseSalePrice_train)
LinearRegression()
Visual Representation of the Linear Regression Model
print('Intercept \t: b = ', linreg.intercept_)
print('Coefficients \t: a = ', linreg.coef_)
# Formula for the Regression line
# Alternative code for below: regline_x =
housePredictor_train.to_numpy()
regline_x = housePredictor_train
regline_y = linreg.intercept_ + linreg.coef_ * regline_x
# Plot the Linear Regression line
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_train, houseSalePrice_train)
plt.plot(regline_x.to_numpy(), regline_y.to_numpy(), 'r-', linewidth =
3)
plt.show()
Intercept : b = [73619.02664658]
Coefficients : a = [[227.27199019]]
Prediction of Response based on the Predictor
# Predict SalePrice values corresponding to GrLivArea
houseSalePrice_test_pred = linreg.predict(housePredictor_test)
# Plot the Predictions
f, axes = plt.subplots(1, 1, figsize=(16, 8))
plt.scatter(housePredictor_test, houseSalePrice_test, color = "green")
plt.scatter(housePredictor_test, houseSalePrice_test_pred, color =
"red")
plt.show()
Goodness of Fit of the Linear Regression Model
print("Explained Variance (R^2) on Train Set \t:",
linreg.score(housePredictor_train, houseSalePrice_train))
print("Explained Variance (R^2) on Test Set \t:",
linreg.score(housePredictor_test, houseSalePrice_test))
Explained Variance (R^2) on Train Set : 0.3949854232045428
Explained Variance (R^2) on Test Set : 0.37249611550040873
Extra : Predicting SalePrice using Multiple Variables
Extract the required variables from the dataset, and then perform Multi-Variate Regression.
housePredictor =
pd.DataFrame(houseData[['GrLivArea','LotArea','TotalBsmtSF','GarageAre
a']])
houseSalePrice = pd.DataFrame(houseData['SalePrice'])
Linear Regression on SalePrice vs Predictor
# Import LinearRegression model from Scikit-Learn
from sklearn.linear_model import LinearRegression
# Split the dataset into Train and Test
#housePredictor_train = pd.DataFrame(housePredictor[:1100])
#housePredictor_test = pd.DataFrame(housePredictor[-360:])
#houseSalePrice_train = pd.DataFrame(houseSalePrice[:1100])
#houseSalePrice_test = pd.DataFrame(houseSalePrice[-360:])
# Split the Dataset into Train and Test
housePredictor_train, housePredictor_test, houseSalePrice_train,
houseSalePrice_test = train_test_split(housePredictor, houseSalePrice,
test_size = 360/1460)
# Create a Linear Regression object
linreg = LinearRegression()
# Train the Linear Regression model
linreg.fit(housePredictor_train, houseSalePrice_train)
LinearRegression()
Coefficients of the Linear Regression Model
Note that you CANNOT visualize the model as a line on a 2D plot, as it is a multi-dimensional
surface.
print('Intercept \t: b = ', linreg.intercept_)
print('Coefficients \t: a = ', linreg.coef_)
Intercept : b = [-18973.0369382]
Coefficients : a = [[70.07255406 0.19864354 43.29137855
96.31552172]]
Prediction of Response based on the Predictor
# Predict SalePrice values corresponding to GrLivArea
houseSalePrice_train_pred = linreg.predict(housePredictor_train)
houseSalePrice_test_pred = linreg.predict(housePredictor_test)
# Plot the Predictions vs the True values
f, axes = plt.subplots(1, 2, figsize=(24, 12))
axes[0].scatter(houseSalePrice_train, houseSalePrice_train_pred, color
= "blue")
axes[0].plot(houseSalePrice_train.to_numpy(),
houseSalePrice_train.to_numpy(), 'w-', linewidth = 1)
axes[0].set_xlabel("True values of the Response Variable (Train)")
axes[0].set_ylabel("Predicted values of the Response Variable
(Train)")
axes[1].scatter(houseSalePrice_test, houseSalePrice_test_pred, color =
"green")
axes[1].plot(houseSalePrice_test.to_numpy(),
houseSalePrice_test.to_numpy(), 'w-', linewidth = 1)
axes[1].set_xlabel("True values of the Response Variable (Test)")
axes[1].set_ylabel("Predicted values of the Response Variable (Test)")
plt.show()
Goodness of Fit of the Linear Regression Model
print("Explained Variance (R^2) on Train Set \t:",
linreg.score(housePredictor_train, houseSalePrice_train))
print("Explained Variance (R^2) on Test Set \t:",
linreg.score(housePredictor_test, houseSalePrice_test))
Explained Variance (R^2) on Train Set : 0.6503512298667333
Explained Variance (R^2) on Test Set : 0.6970077531306746
Interpretation and Discussion
Now that you have performed Linear Regression of SalePrice against the four variables
GrLivArea, LotArea, TotalBsmtSF, GarageArea, compare-and-contrast the Exaplained
Variance (R^2) to determine which model is the best in order to predict SalePrice. What do
you think?
