ML practice-Week 6:
Logistic Regression for binary classification
Introduction
Logistic Regression (also called Logit Regression) is commonly used to estimate the probability
that an instance belongs to a particular class.
If the estimated probability is
greater than 50%, then the model predicts that the instance
belongs to that class
(called the positive class, labeled “1”), or else it predicts that it does
not (i.e., it
belongs to the negative class, labeled “0”).
This makes it a binary classifier.
Loading the dataset
Cleveland Heart-disease dataset
Attribute Information:
1. Age (in years)
2. Sex (1 = male; 0 = female)
3. cp -chest pain type
4. trestbps - resting blood pressure (anything above 130-140 is typically cause for concern)
5. chol-serum cholestoral in mg/dl (above 200 is cause for concern)
6. fbs - fasting blood sugar ( > 120 mg/dl) (1 = true; 0 = false)
7. restecg - resting electrocardiographic results (0 = normal;1 = having ST-T wave
abnormality; 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria)
8. thalach-maximum heart rate achieved
9. exang - exercise induced angina (1 = yes; 0 = no)
10. oldpeak - depression induced by exercise relative to rest
11. slope - slope of the peak exercise ST segment (1 = upsloping; 2 = flat Value; 3 =
downsloping)
12. ca - number of major vessels (0-3) colored by flourosopy
13. thal - (3 = normal; 6 = fixed defect; 7 = reversable defect
14. num (target) - diagnosis of heart disease (angiographic disease status)( 0: < 50% diameter
narrowing ; 1: > 50% diameter narrowing)
Reference
Import basic libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from pandas.plotting import scatter_matrix
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
Download Heart disease dataset
#Define the column names
cols = ['age', 'sex', 'cp', 'trestbps', 'chol', 'fbs', 'restecg', 'thalach', 'exang', 'ol
# Load the dataset
heart_data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/heart-
heart_data
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope c
0 63.0 1.0 1.0 145.0 233.0 1.0 2.0 150.0 0.0 2.3 3.0 0.
1 67.0 1.0 4.0 160.0 286.0 0.0 2.0 108.0 1.0 1.5 2.0 3.
2 67.0 1.0 4.0 120.0 229.0 0.0 2.0 129.0 1.0 2.6 2.0 2.
3 37.0 1.0 3.0 130.0 250.0 0.0 0.0 187.0 0.0 3.5 3.0 0.
4 41.0 0.0 2.0 130.0 204.0 0.0 2.0 172.0 0.0 1.4 1.0 0.
... ... ... ... ... ... ... ... ... ... ... ... .
298 45.0 1.0 1.0 110.0 264.0 0.0 0.0 132.0 0.0 1.2 2.0 0.
299 68.0 1.0 4.0 144.0 193.0 1.0 0.0 141.0 0.0 3.4 2.0 2.
300 57.0 1.0 4.0 130.0 131.0 0.0 0.0 115.0 1.0 1.2 2.0 1.
301 57.0 0.0 2.0 130.0 236.0 0.0 2.0 174.0 0.0 0.0 2.0 1.
302 38.0 1.0 3.0 138.0 175.0 0.0 0.0 173.0 0.0 0.0 1.0
303 rows × 14 columns
# to check the type of data variable
type(heart_data)
pandas.core.frame.DataFrame
# Display first five rows of the dataset
heart_data.head() #head is first 5 rows
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca
0 63.0 1.0 1.0 145.0 233.0 1.0 2.0 150.0 0.0 2.3 3.0 0.0
1 67.0 1.0 4.0 160.0 286.0 0.0 2.0 108.0 1.0 1.5 2.0 3.0
2 67.0 1.0 4.0 120.0 229.0 0.0 2.0 129.0 1.0 2.6 2.0 2.0
3 37.0 1.0 3.0 130.0 250.0 0.0 0.0 187.0 0.0 3.5 3.0 0.0
4 41.0 0.0 2.0 130.0 204.0 0.0 2.0 172.0 0.0 1.4 1.0 0.0
# Display last five rows of the dataset
heart_data.tail() #tail is last 5 rows
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope c
298 45.0 1.0 1.0 110.0 264.0 0.0 0.0 132.0 0.0 1.2 2.0 0.
299 68.0 1.0 4.0 144.0 193.0 1.0 0.0 141.0 0.0 3.4 2.0 2.
300 57.0 1.0 4.0 130.0 131.0 0.0 0.0 115.0 1.0 1.2 2.0 1.
301 57.0 0.0 2.0 130.0 236.0 0.0 2.0 174.0 0.0 0.0 2.0 1.
302 38.0 1.0 3.0 138.0 175.0 0.0 0.0 173.0 0.0 0.0 1.0
Visualizing dataset and features
for feature in cols:
plt.hist(heart_data[feature])
plt.title(feature)
# display histogram
plt.show()
Preprocessing : class labels
Experiments with the database have concentrated on simply attempting to distinguish
presence (values 1,2,3,4) from absence (value 0)
Let us change instances with labels 2,3 and to 1.
# converting class labels 2,3, and 4 into label 1
heart_data = heart_data.replace({"num": {2:1,3:1, 4:1}})
# Visualize the label
heart_data
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope c
0 63.0 1.0 1.0 145.0 233.0 1.0 2.0 150.0 0.0 2.3 3.0 0.
1 67.0 1.0 4.0 160.0 286.0 0.0 2.0 108.0 1.0 1.5 2.0 3.
2 67.0 1.0 4.0 120.0 229.0 0.0 2.0 129.0 1.0 2.6 2.0 2.
3 37.0 1.0 3.0 130.0 250.0 0.0 0.0 187.0 0.0 3.5 3.0 0.
4 41.0 0.0 2.0 130.0 204.0 0.0 2.0 172.0 0.0 1.4 1.0 0.
... ... ... ... ... ... ... ... ... ... ... ... .
298 45.0 1.0 1.0 110.0 264.0 0.0 0.0 132.0 0.0 1.2 2.0 0.
299 68.0 1.0 4.0 144.0 193.0 1.0 0.0 141.0 0.0 3.4 2.0 2.
300 57.0 1.0 4.0 130.0 131.0 0.0 0.0 115.0 1.0 1.2 2.0 1.
301 57.0 0.0 2.0 130.0 236.0 0.0 2.0 174.0 0.0 0.0 2.0 1.
302 38.0 1.0 3.0 138.0 175.0 0.0 0.0 173.0 0.0 0.0 1.0
303 rows × 14 columns
Preprocessing : Replacing missing values
The feature 'ca' has missing values that are given as '?'. Let us replace the '?' with nan and then
fill those missing values using 'mean' imputation strategy.
heart_data.replace('?',np.nan, inplace=True)
imputer = SimpleImputer(missing_values = np.nan, strategy ='mean')
imputer = imputer.fit(heart_data)
heart_imputed = imputer.transform(heart_data)
heart_data_imputed = pd.DataFrame(heart_imputed, columns = cols)
heart_data_imputed
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope
0 63.0 1.0 1.0 145.0 233.0 1.0 2.0 150.0 0.0 2.3 3.0 0.
1 67.0 1.0 4.0 160.0 286.0 0.0 2.0 108.0 1.0 1.5 2.0 3.
2 67.0 1.0 4.0 120.0 229.0 0.0 2.0 129.0 1.0 2.6 2.0 2.
3 37.0 1.0 3.0 130.0 250.0 0.0 0.0 187.0 0.0 3.5 3.0 0.
4 41.0 0.0 2.0 130.0 204.0 0.0 2.0 172.0 0.0 1.4 1.0 0.
... ... ... ... ... ... ... ... ... ... ... ...
298 45.0 1.0 1.0 110.0 264.0 0.0 0.0 132.0 0.0 1.2 2.0 0.
299 68.0 1.0 4.0 144.0 193.0 1.0 0.0 141.0 0.0 3.4 2.0 2.
300 57.0 1.0 4.0 130.0 131.0 0.0 0.0 115.0 1.0 1.2 2.0 1.
301 57.0 0.0 2.0 130.0 236.0 0.0 2.0 174.0 0.0 0.0 2.0 1.
302 38.0 1.0 3.0 138.0 175.0 0.0 0.0 173.0 0.0 0.0 1.0 0.
303 rows × 14 columns
Let us first separate the input attributes and target attribute.
# Assign a new variable y as target
y = heart_data_imputed['num']
y = np.array(y)
array([0., 1., 1., 0., 0., 0., 1., 0., 1., 1., 0., 0., 1., 0., 0., 0., 1.,
0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., 0., 1., 0., 1., 1., 0.,
0., 0., 1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0.,
0., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 1., 1., 0.,
1., 1., 0., 1., 1., 1., 1., 0., 1., 0., 0., 1., 0., 0., 0., 1., 0.,
0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 1., 0., 0., 0., 0.,
0., 0., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 0., 0., 1.,
1., 1., 1., 0., 1., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0.,
1., 1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0.,
1., 0., 1., 0., 1., 1., 0., 1., 0., 0., 1., 1., 0., 0., 1., 0., 0.,
1., 1., 1., 0., 1., 1., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0.,
0., 1., 1., 1., 0., 1., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0.,
0., 0., 1., 1., 0., 0., 0., 1., 1., 0., 1., 1., 0., 0., 1., 1., 1.,
0., 0., 0., 0., 0., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0.,
0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 1., 1., 1., 1., 0., 1., 0.,
1., 0., 1., 0., 0., 0., 1., 0., 1., 0., 1., 0., 1., 1., 1., 0., 0.,
0., 1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 0.])
# Remove the target variable from heart_data
del heart_data_imputed['num']
heart_data_imputed.describe()
age sex cp trestbps chol fbs reste
count 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 303.0000
mean 54.438944 0.679868 3.158416 131.689769 246.693069 0.148515 0.9900
std 9.038662 0.467299 0.960126 17.599748 51.776918 0.356198 0.9949
min 29.000000 0.000000 1.000000 94.000000 126.000000 0.000000 0.0000
25% 48.000000 0.000000 3.000000 120.000000 211.000000 0.000000 0.0000
50% 56.000000 1.000000 3.000000 130.000000 241.000000 0.000000 1.0000
75% 61.000000 1.000000 4.000000 140.000000 275.000000 0.000000 2.0000
max 77.000000 1.000000 4.000000 200.000000 564.000000 1.000000 2.0000
Understanding the correlation between Input features
plt.figure(figsize=[15,15])
sns.heatmap(heart_data_imputed.corr(),annot = True, square = True)
plt.show()
Train and Test data split
# Let us split the data for training and testing
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(heart_data_imputed, y, test_size = 0.2
print('Shape of training data',X_train.shape)
print('Shape of training labels', y_train.shape)
print('Shape of testing data', X_test.shape)
print('Shape of testing labels',y_test.shape)
Shape of training data (227, 13)
Shape of training labels (227,)
Shape of testing data (76, 13)
Shape of testing labels (76,)
X_train
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope
173 62.0 0.0 4.0 140.0 394.0 0.0 2.0 157.0 0.0 1.2 2.0 0.
261 58.0 0.0 2.0 136.0 319.0 1.0 2.0 152.0 0.0 0.0 1.0 2.
37 57.0 1.0 4.0 150.0 276.0 0.0 2.0 112.0 1.0 0.6 2.0 1.
101 34.0 1.0 1.0 118.0 182.0 0.0 2.0 174.0 0.0 0.0 1.0 0.
166 52.0 1.0 3.0 138.0 223.0 0.0 0.0 169.0 0.0 0.0 1.0 0.
... ... ... ... ... ... ... ... ... ... ... ...
251 58.0 1.0 4.0 146.0 218.0 0.0 0.0 105.0 0.0 2.0 2.0 1.
192 43.0 1.0 4.0 132.0 247.0 1.0 2.0 143.0 1.0 0.1 2.0 0.
117 35.0 0.0 4.0 138.0 183.0 0.0 0.0 182.0 0.0 1.4 1.0 0.
47 50.0 1.0 4.0 150.0 243.0 0.0 2.0 128.0 0.0 2.6 2.0 0.
172 59.0 0.0 4.0 174.0 249.0 0.0 0.0 143.0 1.0 0.0 2.0 0.
As there
227 is a wide
rows × 13 variation
columns among the numerical values between features, it is a best practice
to normalize the features before training.
Normalizing features for training
# Instantiate the scaler to a variable and fit the train and test data
ss = StandardScaler()
X_train_norm = ss.fit_transform(X_train)
X_test_norm = ss.transform(X_test)
Perform Classification
LR = LogisticRegression()
classifier=LR.fit(X_train_norm, y_train)
score = LR.score(X_train_norm, y_train)
print("Training score: ", score)
#Make the prediction
y_pred = LR.predict(X_test_norm)
Training score: 0.8634361233480177
# Import the libraries
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
Confusion Matrix
A confusion matrix is a summary of prediction results on a classification problem.
Source
# visualizing the confusion matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import plot_roc_curve
class_names=["0","1"]
plot_confusion_matrix(classifier, X_test_norm, y_test,display_labels=class_names,cmap=plt.
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names, rotation=0)
plt.yticks(tick_marks, class_names)
plt.title('Confusion matrix')
plt.show()
/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarnin
warnings.warn(msg, category=FutureWarning)
CR = classification_report(y_test, y_pred)
print('Classification report \n')
print(CR)
Classification report
precision recall f1-score support
0.0 0.78 0.90 0.84 40
1.0 0.87 0.72 0.79 36
accuracy 0.82 76
macro avg 0.82 0.81 0.81 76
weighted avg 0.82 0.82 0.81 76
Hyperparameter tuning with RandomizedSearchCV and GridSearchCV
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
RandomizedSearchCV
# Create a hyperparameter grid for LogisticRegression
log_reg_grid_rs = {"C": np.logspace(-4, 4, 20),
"solver": ["liblinear"]}
# Tune LogisticRegression
np.random.seed(42)
# Setup random hypterparameter search for LogisticRegression with cross-validation
RS_log_reg = RandomizedSearchCV(LogisticRegression(),
param_distributions=log_reg_grid_rs,
cv=5,
n_iter=20,
verbose=True)
# Fit random hyperparamter search model for LogisticRegression
RS_log_reg.fit(X_train_norm, y_train)
Fitting 5 folds for each of 20 candidates, totalling 100 fits
RandomizedSearchCV(cv=5, estimator=LogisticRegression(), n_iter=20,
param_distributions={'C': array([1.00000000e-04, 2.63665090e-04,
6.95192796e-04, 1.83298071e-03,
4.83293024e-03, 1.27427499e-02, 3.35981829e-02, 8.85866790e-02,
2.33572147e-01, 6.15848211e-01, 1.62377674e+00, 4.28133240e+00,
1.12883789e+01, 2.97635144e+01, 7.84759970e+01, 2.06913808e+02,
5.45559478e+02, 1.43844989e+03, 3.79269019e+03, 1.00000000e+04]),
'solver': ['liblinear']},
verbose=True)
?LogisticRegression()
np.logspace(-4, 4, 20)
array([1.00000000e-04, 2.63665090e-04, 6.95192796e-04, 1.83298071e-03,
4.83293024e-03, 1.27427499e-02, 3.35981829e-02, 8.85866790e-02,
2.33572147e-01, 6.15848211e-01, 1.62377674e+00, 4.28133240e+00,
1.12883789e+01, 2.97635144e+01, 7.84759970e+01, 2.06913808e+02,
5.45559478e+02, 1.43844989e+03, 3.79269019e+03, 1.00000000e+04])
# Find the best hyperparameters
RS_log_reg.best_params_
{'C': 0.08858667904100823, 'solver': 'liblinear'}
RS_log_reg.score(X_train_norm, y_train)
0.8678414096916299
# Make predictions with tuned model
y_preds = RS_log_reg.predict(X_test_norm)
# Confusion matrix
print(confusion_matrix(y_test, y_preds))
[[36 4]
[10 26]]
?plot_roc_curve
# Plot ROC curve and calculate AUC metric
plot_roc_curve(RS_log_reg, X_test_norm, y_test)
/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarnin
warnings.warn(msg, category=FutureWarning)
<sklearn.metrics._plot.roc_curve.RocCurveDisplay at 0x7f8a13f6d190>
GridSearchCV
# Setup grid hyperparamter search for LogisticRegression
log_reg_grid_gs = {"C": np.logspace(-4, 4, 30),
"solver": ["liblinear"]}
GS_log_reg = GridSearchCV(LogisticRegression(),
param_grid=log_reg_grid_gs,
cv=5,
verbose=True)
# Fit grid hyperparameter search model
GS_log_reg.fit(X_train_norm, y_train);
Fitting 5 folds for each of 30 candidates, totalling 150 fits
np.logspace(-4, 4, 30)
array([1.00000000e-04, 1.88739182e-04, 3.56224789e-04, 6.72335754e-04,
1.26896100e-03, 2.39502662e-03, 4.52035366e-03, 8.53167852e-03,
1.61026203e-02, 3.03919538e-02, 5.73615251e-02, 1.08263673e-01,
2.04335972e-01, 3.85662042e-01, 7.27895384e-01, 1.37382380e+00,
2.59294380e+00, 4.89390092e+00, 9.23670857e+00, 1.74332882e+01,
3.29034456e+01, 6.21016942e+01, 1.17210230e+02, 2.21221629e+02,
4.17531894e+02, 7.88046282e+02, 1.48735211e+03, 2.80721620e+03,
5.29831691e+03, 1.00000000e+04])
# Check the best hyperparameters
GS_log_reg.best_params_
{'C': 0.1082636733874054, 'solver': 'liblinear'}
# Evaluate the grid search LogisticRegression model
GS_log_reg.score(X_train_norm, y_train)
0.8634361233480177
# Make predictions with tuned model
y_preds = GS_log_reg.predict(X_test_norm)
# Confusion matrix
print(confusion_matrix(y_test, y_preds))
[[36 4]
[10 26]]
