Machine Learning

Lesson 8: Ensemble Learning

© Simplilearn. All rights reserved.

 Concepts Covered

Ensemble Learning

Bagging and Boosting Algorithms

Model Selection

Cross-validation
 Learning Objectives

By the end of this lesson, you will be able to:

Explain ensemble learning

Evaluate performance of boosting models

Ensemble Learning
Topic 1: Overview
 Definition
Ensemble techniques combine individual models together to improve the stability and predictive power of the model
Ideology

Certain models do well in modeling one

aspect of the data, while others do well
in modeling another

Instead of learning a single complex

model, learn several simple models
and combine their output to produce
the final decision

In ensemble learning, other models

strength performs offset on individual
model variances and biases

Ensemble learning will provide a

composite prediction where the final
accuracy is better than the accuracy of
individual models
 Working

Inputs

Model 1

Model 2 Ensemble Predictions

model

Model 3
 Significance

Robustness Accuracy

Ensemble models Ensemble models deliver

incorporate the predictions accurate predictions and
from all the base learners have improved performances

01 02
 Ensemble Learning Methods

Techniques to create an Ensemble model

Combine all “weak” learners to form an ensemble

OR

Create an ensemble of well-chosen strong and diverse models

 Averaging

50
[50, 0, 0]
True setosa

petal width ≤ 0.8

150
[50, 50, 50] 54
setosa [0, 49, 5]
versicolor
False Petal width ≤ 1.75
100
[0, 50, 50]
versicolor
46
[0, 1, 45]
virginica

Decision Tree Logistic Regression SVM

p1 + p2 + p3
Equal weights are assigned to different models P=
3
 Weighted Averaging

50
[50, 0, 0]
True setosa

petal width ≤ 0.8

150
[50, 50, 50] 54
setosa [0, 49, 5]
versicolor
False Petal width ≤ 1.75
100
[0, 50, 50]
versicolor
46
[0, 1, 45]
virginica

Decision Tree Logistic Regression SVM

50
35

15

w1 + w2 + w3 = 1 Each model is assigned a different weight P = w1 * p1 + w2 * p2 + w3 * p3

 Bagging
Bagging or bootstrap aggregation reduces variance of an estimate by taking mean of multiple estimates

Original
D Training data

Create randomly sampled

STEP 1 datasets of the original
training data ( bootstrapping )

D1 D2 …. Dt-1 Dt Resamples

Build and fit several classifiers

STEP 2 to each of these diverse copies
Models
C1 C2 Ct-1 Ct

Take the average of all the predictions

STEP 3 to make final overall prediction
C*
 Boosting
Boosting reduces bias by training weak learners sequentially, each trying to correct its predecessor

headache, stomach headache is surely caused by

pain, and leg hurts X , but cannot explain others

stomach pain and leg hurts agree for X, leg hurts because of
(headache because of X) Y, but cannot explain stomach pain
 Boosting (Contd.)

headache is surely caused by X agree for X, leg hurts because of agree on X, sort of agree on Y,
, but cannot explain others Y, cannot explain stomach pain sure on stomach pain (Z)

Final Diagnosis

Result of the weighted opinions

that the doctors (sequentially)
 Boosting Algorithm

STEP: 01 STEP: 02 STEP: 03

Train a classifier H1 that best Identify the region where H1 Exaggerate those samples
classifies the data with produces errors, add for which H1 gives a
respect to accuracy weights to it and produce a different result from H2 and
H2 classifier produces H3 classifier

Repeat step 02 for a new

classifier
Ensemble Learning
Topic 2: Algorithms
 Random Forests

Random forests are utilized to produce decorrelated decision trees

Features
1, 2,3,4

RF’s create random subsets of the features

Features 1,2 Features 2,3 Features 3,4 Features 1,4 Smaller trees are built using these subsets creating tree diversity

i To overcome overfitting, diverse sets of decision trees are required

 Adaboost

+ -
+ +

-
Consider a scenario, where there are ‘+’ and ‘–’
-
Objective : Classify ‘+’ and ‘–’ + -

+ -
-
 Adaboost Working: Step 01

• Assign equal weights to each data point

• Apply a decision stump to classify them as + (plus) and – (minus)

• Decision stump (D1) has generated vertical plane at the left side to classify

• Apply higher weights to incorrectly predicted three + (plus) and add

Iteration 01
another decision stump
 Adaboost Working: Step 02

• Size of three incorrectly predicted + (plus) is made bigger as compared to

rest of the data points

• The second decision stump (D2) will try to predict them correctly

• Now, vertical plane (D2) has classified three mis-classified + (plus) correctly

• D2 has also caused mis-classification errors to three – (minus)

Iteration 02
 Adaboost Working: Step 03

• D3 adds higher weights to three – (minus)

• Horizontal line is generated to classify + (plus) and – (minus) based on

higher weight of mis-classified observation

Iteration 03
 Adaboost Working: Step 04

• D1, D2, and D3 are combined to form a strong prediction having complex

rule as compared to individual weak learner

Final Classifier
 Adaboost Algorithm

STEP STEP STEP STEP

1 2 3 4
Initially each data point A classifier ‘H1’ is The weighing factor 𝜶 Weight after time t is
is weighted equally picked up that best is dependent on given as :
with weight classifies the data with errors (𝝐𝒕 ) caused by
minimal error rate the H1 classifier
𝑊𝑖 = 1/𝑛 𝑾𝒕+𝟏
𝒊
𝒁
𝒆−𝜶𝒕.𝒉𝟏 𝒙 .𝒚(𝒙)

where n is the number 𝟏

𝜶𝒕 = ln 𝟏−𝝐𝒕
of samples 𝟐 𝝐 𝒕 where z is the
normalizing factor,
h1(x).y(x) is sign of
the current output
 Adaboost Flowchart

Weigh each data points

equally with weight,
1
𝑊𝑖 =
𝑛

Pick a classifier that

Pick 𝛼 , a weighing minimizes the error rate,
factor 𝜖𝑡

Calculate 𝑊 𝑡+1
 Gradient Boosting (GBM)
Gradient boosting involves three elements:

A loss function to be optimized

A weak learner to make predictions

An additive model to add

weak learners to minimize
the loss function

1 2 3
GBM minimizes the loss function (MSE) of a model by adding weak learners using a gradient descent procedure.
 GBM Mechanism

GBM predicts the residuals or errors of prior models

01 and then sums them to make the final prediction

One weak learner is added at a time and existing

weak learners in existing
the modelweakare left unchanged
learners
02
One new weak learner is added at a time and
in the model left unchanged

GBM repetitively leverages the patterns in residuals

03 and strengthens a model with weak predictions

Modeling is stopped when residuals do

not have any pattern that can be modeled 04
04
 GBM Algorithm

Step 01 Fit a simple regression or classification model

Step 02 Calculate error residuals ( actual value - predicted value )

Step 03 Fit a new model on error residuals as target variable with same input variables

Step 04 Add the predicted residuals to the previous predictions

Fit another model on residuals that are remaining and repeat steps 2
Step 05 and 5 until model is overfit or the sum of residuals becomes constant
 XGBoost
eXtreme Gradient Boosting is a library for developing fast and high-performance gradient boosting tree models.

custom tree
eXtreme Gradient
building algorithm
Boosting

Used for:
• Classification Interfaces for
• Regression Python and R,
• Ranking can be executed
With custom loss on YARN
functions

XGBoost is extensively used in ML competitions as it is almost 10 times faster than other

gradient boosting techniques
 XGBoost Parameters

1
General Parameters

Number of threads

3 2
Task Parameters Booster Parameters
• Objective • Step size
• Evaluation metric • Regularization
 XGBoost Library Features
XGBoost library features tools are built for the sole purpose of model performance and computational speed.

01 02 03

SYSTEM ALGORITHM MODELS

Parallelization Sparse Aware Gradient Boosting

Tree construction using all Automatic handling of Gradient boosting machine
CPU cores while training missing data values algorithm including learning rate

Distributed Computing Block Structure Stochastic Gradient Boosting

Training very large models Supports the Sub-sampling at the row, column
using a cluster of machines parallelization of tree and column per split levels
construction
Cache Optimization Regularized Gradient Boosting
Data structures make best Continued Training With both L1 and L2
use of hardware To boost an already fitted regularization
model on new data
 General Parameters
General parameters guide the overall functioning of XGBoost

• nthread
- Number of parallel threads
- If no value is entered, algorithm automatically detects the number of cores and
runs on all the cores
• booster
- gbtree: tree-based model
- gblinear: linear function

• Silent [default =0]

- if set to 1, no running messages will be printed.
Hence, keep it ‘0’ as the messages might help in understanding the model
 Booster Parameters
Booster parameters guide individual booster (Tree/Regression) at each step

Parameters for tree booster

• eta
- Step size shrinkage is used in update to prevent overfitting
- Range in [0,1], default 0.3
• gamma
- Minimum loss reduction required to make a split
- Range [0,∞ ], default 0
• max_depth
- Maximum depth of a tree
- Range [1, ∞ ], default 6
• min_child_weight
- Minimum sum of instance weight needed in a child
- Range [0, ∞], default 1
 Booster Parameters (Contd.)

Parameters for tree booster

• max_delta_step
- Maximum delta step allowed in each tree's weight estimation
- Range in [0, ∞ ], default 0
• subsample
- Subsample ratio of the training instance
- Range [0,1 ], default 1
• Colsample_bytree
- Subsample ratio of columns when constructing each tree
- Range [0, 1 ], default 1
 Booster Parameters (Contd.)

Parameters for Linear booster

• lambda
- L2 regularization term on weights
- default 0
• alpha
- L1 regularization term on weights
- default 0
• Lambda_bias
- L2 regularization term on bias
- default 0
 Task Parameters
Task parameters guide optimization objective to be calculated at each step

1. Objectives [default = reg:linear]

• "binary:logistic": logistic regression for binary classification, output is probability not class
• "multi:softmax": multiclass classification using the softmax objective, need to specify num_class

2. Evaluation Metric
• "rmse"
• "logloss"
• "error"
• "auc"
• "merror"
• "mlogloss"

i https://xgboost.readthedocs.io/en/latest/parameter.html
Assisted Practice
Boosting Duration: 15 mins.

Problem Statement: The Pima Indians Diabetes dataset has diagnostic measures like BMI, blood pressure of female
patients of more than 21 years old.

Objective:

• Classify whether the person is diabetic with maximum accuracy

Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
Unassisted Practice
Boosting Duration: 20 mins.

Problem Statement: The Iris plant has 3 species : Iris Setosa, Iris Versicolour, Iris Virginica
One class is linearly separable from the other two whereas the latter are not linearly separable from each other.

Objective:

• Import the iris dataset using sklearn

• Build a classification model using AdaBoost and XGBoost
• Compare accuracy of both the models

Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
 Step 1: Data Import

Code

datasets.load_iris
from sklearn.ensemble import AdaBoostClassifier

y = iris.target
from sklearn import datasets

X = iris.data
from sklearn.model_selection import train_test_split
from sklearn import metrics

iris = datasets.load_iris()

iris =
X = iris.data
y = iris.target

()
dataset
s.load_

iris.da

iris.ta
iris =

iris()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

rget
X =

y =
ta
 Step 2: Classifier
Code

abc = AdaBoostClassifier(n_estimators=50, learning_rate=1)

model = abc.fit(X_train, y_train)
y_pred = model.predict(X_test)
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

from sklearn import svm

from xgboost import XGBClassifier
clf = XGBClassifier()
clf.fit(X_train, y_train)
y_pred2 = clf.predict(X_test)
print(metrics.accuracy_score(y_test, y_pred2))
Ensemble Learning
Topic 3: Model Selection
 Model Evaluation

Models can be evaluated based on their measure of performance

 Assessing Model Performance

Train/Test Split Cross Validation Split

• Divide the training dataset ▪ Sets of train/test splits created
• Train on the first training set ▪ Accuracy for each set is
• Test on the second set checked
▪ Results are averaged

Techniques to assess model performance

 Train/Test Split

Examples
- + -
- +
-
- - -
+ +
+ +

- + - Train
+
+
+
- - -
+
Hypothesis Space
+

Creating the training dataset

 Train/Test Split (Contd.)

Testing set
-
-
-
-
+ + +

+
+
+
- -
Hypothesis space H
+

Creating the testing dataset

 Train/Test Split (Contd.)

Testing set
-
+
+
-
+
+ +
Test
- +
+
- -
- Hypothesis space H
 Train/Test Split (Contd.)

9/13 correct
Testing set
- -
- +
-+
- -
+ +
+ + + +

+ -
++
+ +
- - - -
Hypothesis space H
+ -

Verifying the results

 Common Splitting Strategies

K-Fold Cross-Validation Dataset

Train Test

Leave-one-out

Identifying the Train/Test split ratio

 Common Splitting Strategies (Contd.)

K-Fold Cross-Validation

Leave-one-out
 Train/Test Split vs. Cross-Validation

Cross-validation Train/Test Split

▪ More accurate estimate of out-of- ▪ Runs K-times faster than K-fold cross-
sample accuracy validation

▪ More efficient use of data(every ▪ Simpler to examine the detailed results

observation is used for both of testing process
training and testing)
Assisted Practice
Cross-validation Duration: 15 mins.

Problem Statement: Few learners have implemented random forest classifier on the Iris data but,
better accuracy can be achieved using cross-validation sampling technique.

Objective:
• Generate the random forest using cross validation splitting technique.
• Determine the accuracy such that it is the average of all the resultant accuracies.

Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and
password that are generated. Click on the Launch Lab button. On the page that appears, enter the
username and password in the respective fields, and click Login.
Unassisted Practice
Cross-Validation Duration: 20 mins.

Problem Statement: Mtcars, an automobile company in Chambersburg, United States has recorded the
production of its cars within a dataset. In order to classify cars, the company has come up with two classification
models (KNN and Logistic Regression).

Objective: Perform a model selection between the above two models using the sampling technique as 10-fold
cross- validation.

Note: This practice is not graded. It is only intended for you to apply the knowledge you have gained to solve real-
world problems.

Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
 Defining a 10-Fold KNN Model

Define a 10-fold KNN model and calculate the average of each accuracy
matrix obtained.

Code

from sklearn.cross_validation import cross_val_score

knn = KNeighborsClassifier(n_neighbors=4)
print(cross_val_score(knn, x, y, cv=10, scoring ='accuracy').mean())

The accuracy came out to be 56.66%.

 Defining a 10-Fold Logistic Regression Model

Define a 10-fold Logistic Regression model and calculate the average of

each accuracy matrix obtained.

Code

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression()
print (cross_val_score(logreg, x, y, cv=10, scoring = 'accuracy').mean())

The accuracy came out to be 28.33%.

Hence, you can infer that KNN model for the task is better as compared to Logistic
Regression model.
 Key Takeaways

Now, you are able to:

Explain ensemble learning

Evaluate performance of boosting models

 Knowledge
Check

© Simplilearn. All rights reserved.

 Knowledge
Check
Which of the following is not an ensemble method?
1

a. Decision Tree

b. Random Forest

c. Adaboost

d. None of the above

© Simplilearn. All rights reserved.

Knowledge
Check
Which of the following is not an ensemble method?
1

a. Decision Tree

b. Random Forest

c. Adaboost

d. None of the above

The correct answer is a. Decision Tree

In decision tree, single tree is built and no ensembling is required.

 Knowledge
Check
Some of the advantages of XGBoost include:
2

a. Parallelization

b. Handling missing values

c. Support for multiple GBM models

d. All of the above

© Simplilearn. All rights reserved.

Knowledge
Check
Some of the advantages of XGBoost include:
2

a. Parallelization

b. Handling missing values

c. Support for multiple GBM models

d. All of the above

The correct answer is d. All of the above

XGBoost is scalable and has accurate implementation of gradient boosting machines.

It has proven to push the limits of computing power for boosted trees.
 Lesson-End Project
Car Evaluation Database Duration: 30 mins

Problem Statement: Used car market has significantly grown in recent times with clients ranging from used car dealers
and buyers.You are provided with a car evaluation dataset that has features like price, doors, safety, and so on.
You are required to create a robust model that allows stakeholders to predict the condition of a used vehicle.
Objective:
▪ Predict the condition of a vehicle based on features
▪ Plot the most import features
▪ Train multiple classifiers and compare the accuracy
▪ Evaluate XGBoost model with K-fold cross-validation

Access: Click on the Labs tab on the left side panel of the LMS. Copy or note the username and password that are
generated. Click on the Launch Lab button. On the page that appears, enter the username and password in the
respective fields, and click Login.
 Thank You

© Simplilearn. All rights reserved.