Introduction to Boosting
Cynthia Rudin
PACM, Princeton
University
Advisors
Ingrid Daubechies and Robert
Schapire
Say you have a database of news articles…
( , +1) ( , +1) ( ) (
, -1 )
, -1
( , +1) ( , +1) ( ) (
, -1 )
, -1
( , +1) ( , +1) ( ) (
, -1 )
, -1
( , +1) ( , +1) ( ) (
, -1 )
, +1
where articles are labeled ‘+1’ if the category is “entertainment”,
and ‘-1’ otherwise.
Your goal is: Given a new article , find its label.
This is not easy, there are noisy datasets, high dimensions.
Examples of Statistical Learning Tasks:
• Optical Character Recognition (OCR) (post office,
banks), object recognition in images.
• Bioinformatics (analysis of gene array data for tumor
detection, protein classification, etc.)
• Webpage classification (search engines), email filtering,
document retrieval
• Semantic classification for speech, automatic .mp3
sorting
• Time-series prediction (regression)
Huge number of applications, but all have high dimensional data
Examples of classification algorithms:
• SVM’s (Support Vector Machines – large margin classifiers)
• Neural Networks
• Decision Trees / Decision Stumps (CART)
• RBF Networks
• Nearest Neighbors
• Bayes Net
Which is the best?
Depends on amount and type of data, and application!
It’s a tie between SVM’s and Boosted Decision Trees/Stumps
for general applications.
One can always find a problem where a particular algorithm is the best. Boosted convolutional neural nets are the best for
OCR (Yann LeCun et al).
Training Data: {(xi,yi)}i=1..m where (xi,yi) is chosen iid from an
unknown probability distribution on X×{-1,1}.
“space of all possible articles” “labels”
Huge Question: Given a new random example x, can we predict
its correct label with high probability? That is, can we generalize
from our training data?
X + +
_
_
+
? _
Huge Question: Given a new random example x, can we predict
its correct label with high probability? That is, can we generalize
from our training data?
Yes!!! That’s what the field of statistical learning is all about.
The goal of statistical learning is to characterize points from an
unknown probability distribution when given a representative
sample from that distribution.
How do we construct a classifier?
• Divide the space X into two sections, based on the sign of a
function f : X→R.
• Decision boundary is the zero-level set of f. f(x)=0
+
-
X + +
_
_
+
? _
Classifiers divide the space into two pieces for binary classification. Multiclass classification can always be reduced to binary.
Y Overview of Talk Z
• The Statistical Learning Problem (done)
• Introduction to Boosting and AdaBoost
• AdaBoost as Coordinate Descent
• The Margin Theory and Generalization
Say we have a “weak” learning algorithm:
• A weak learning algorithm produces weak classifiers.
• (Think of a weak classifier as a “rule of thumb”)
Examples of weak classifiers for “entertainment” application:
h1( ) = +1 if contains the term “movie”,
-1 otherwise
h2( ) = +1 if contains the term “actor”,
-1 otherwise
h3( ) = +1 if contains the term “drama”,
-1 otherwise
Wouldn’t it be nice to combine the weak classifiers?
Boosting algorithms combine weak
classifiers in a meaningful way.
Example:
f( ) = sign[.4 h1 ( ) + .3 h2 ( ) + .3 h3 ( )]
ASo
boosting
if the article
algorithm
contains
takesthe
asterm
input:
“movie”, and the word “drama”,
but not the word “actor”:
- the weak learning algorithm which produces the weak classifiers
- a large
Thetraining
value ofdatabase
f is sign[.4-.3+.3] = 1, so we label it +1.
and outputs:
- the coefficients of the weak classifiers to make the combined
classifier
Two ways to use a Boosting Algorithm:
• As a way to increase the performance of already `strong`
classifiers.
• Ex. neural networks, decision trees
• “On their own” with a really basic weak classifier
• Ex. decision stumps
AdaBoost
(Freund and Schapire ’95)
-Start with a uniform distribution (“weights”) over training examples.
(The
At theweights tell the
end, make weak learning
(carefully!) algorithm
a linear which
combination ofexamples
the weak are important.)
classifiers
obtained at all iterations.
-Request a weak classifier from the weak learning algorithm, hj :X→{-1,1}.
f final (x) = sign(λ1h1 (x) + K + λn hn (x)) t
-Increase the weights on the training examples that were misclassified.
-(Repeat)
AdaBoost
Define three important things:
d t ∈ R m := distribution (“weights”) over examples at time t
dt = [ .25 .3 .2 .25 ]
1 2 3 4
AdaBoost
Define three important things:
λ t ∈ R n := coeffs of weak classifiers for the linear combination
f t (x) = sign(λt ,1h1 (x) + ... + λt ,n hn (x))
AdaBoost
Define:
M ∈ R m×n :=matrix of hypotheses and data
Enumerate every possible weak
classifier which can be produced
by weak learning algorithm
h1 hj hn
“movie” “actor” “drama”
i Mij
# of data points ⎧ 1 if weak classif. h j classifies pt x i correctly
M ij := h j (x i ) yi = ⎨
⎩− 1 otherwise
The matrix M has too many columns to actually be enumerated. M acts as the only input to AdaBoost.
M AdaBoost λ final
dt , λ t
AdaBoost (Freund and Schapire 95)
λ1 = 0 Initialize coeffs to 0
for t = 1..T final
1
d t ,i = m
e −( Mλ t )i for all i Calculate (normalized) distribution
∑e
i '=1
− ( Mλ t ) i '
jt ∈ arg max j (d Tt M ) j Request weak classif. from
weak learning algorithm
rt = (d Tt M ) jt
α t = ln⎜⎜
1 t
⎟⎟
2 ⎝ 1 − rt ⎠
⎛1+ r ⎞
λ t +1 = λ t + α t e jt
} Update linear combo of weak classifiers
end for
AdaBoost (Freund and Schapire 95)
λ1 = 0
for t = 1..T final
1
d t ,i = m
e −( Mλ t )i for all i
∑e
i '=1
− ( Mλ t ) i '
jt ∈ arg max j (d Tt M ) j
rt = (d Tt M ) jt “Edge” or “correlation” of weak classifier jt.
m
1 ⎛1+ r ⎞ (d M ) jt = ∑ d t ,i yi h jt (x i ) = Edt [ yi h jt ]
T
α t = ln⎜⎜ t
⎟⎟ t
i =1
2 ⎝ 1 − rt ⎠
λ t +1 = λ t + α t e jt
end for
Y AdaBoost as Coordinate Descent Z
Breiman, Mason et al., Duffy and Helmbold, etc. noticed that
AdaBoost is a coordinate descent algorithm.
• Coordinate descent is a minimization algorithm like gradient
descent, except that we only move along coordinates.
• We cannot calculate the gradient because of the high
dimensionality of the space!
• “coordinates” = weak classifiers
“distance to move in that direction” = the update αt
AdaBoost minimizes the following function via coordinate descent:
m
F (λ ) := ∑ e −( Mλ )i
i =1
Choose a direction: jt ∈ arg max j (d Tt M ) j
Choose a distance to move in that direction:
rt = (d Tt M ) jt
1 ⎛1+ r ⎞
α t = ln⎜⎜ t
⎟⎟
2 ⎝ 1 − rt ⎠
λ t +1 = λ t + α t e jt
m
The function F (λ ) := ∑ e −( Mλ )i is convex:
i =1
1) If the data is non-separable by the weak classifiers, the minimizer
of F occurs when the size of λ is finite.
(This case is ok. AdaBoost converges to something we understand.)
2) If the data is separable, the minimum of F is 0
(This case is confusing!)
The original paper suggested that AdaBoost would probably
overfit…
But it didn’t in practice!
Why not?
The margin theory!
Y Boosting and Margins Z
• We want the boosted classifier (defined via λ) to
generalize well, i.e., we want it to perform well on
data that is not in the training set.
• The margin theory: The margin of a boosted
classifier indicates whether it will generalize well.
(Schapire et al. ‘98)
• Large margin classifiers work well in practice,
(but there’s more to this story).
Think of the margin as the confidence of a prediction.
Generalization Ability of Boosted Classifiers
Can we guess whether a boosted classifier f generalizes well?
• Can not calculate Prerror(f)
Minimize the rhs of a (loose) inequality such as this one (Schapire et al.)
When there are no training errors, with probability at least 1-δ,
⎛ ⎛ 2 m ⎞
1
2⎞
⎜ 1 ⎜ d log ( d ) ⎟ ⎟
Prerror ( f ) ≤ Ο⎜ + log( 1 ) ⎟.
⎜ m ⎜ ( µ ( f )) 2 δ ⎟ ⎟
⎝ ⎝ ⎠ ⎠
Probability that
classifier f makes # of training examples margin of f
an error on a
random position d=VC dim. of
x∈ X hyp. space, d≤m
The margin theory:
When there are no training errors, with high probability:
(Schapire et al, ‘98)
⎛ d ⎞
~⎜ m ⎟ d=VC dim. of
Prerror ( f ) ≤ Ο⎜ ⎟. hyp. space, d≤m
⎜ µ( f ) ⎟
⎝ ⎠ # of training examples
Probability that
classifier f makes margin of f
an error on a
random position
x∈ X
Large margin = better generalization = smaller probability of error
For Boosting, the margin of combined classifier f λ
(where fλ := sign(λ1h1 + …+ λnhn ) ) is defined by
(Mλ ) i
margin := µ ( fλ ) := min .
i λ1
Does AdaBoost produce maximum margin classifiers?
(AdaBoost was invented before the margin theory…)
(Grove and Schuurmans ’98)
- yes, empirically.
(Schapire, et al. ’98)
- proved AdaBoost achieves at least half the maximum possible
margin.
(Rätsch and Warmuth ’03)
- yes, empirically.
- improved the bound.
(R, Daubechies, Schapire ’04)
- no, it doesn’t.
AdaBoost performs mysteriously well!
AdaBoost performs better than algorithms which are
designed to maximize the margin
Still open: Why does AdaBoost work so well?
Does AdaBoost converge?
Better / more predictable boosting algorithms!
