Machine Learning
Support Vector Machine
Lecturer: Duc Dung Nguyen, PhD.
Contact:
[email protected]Faculty of Computer Science and Engineering
Hochiminh city University of Technology
Contents
1. Analytical Geometry
2. Maximum Margin Classifiers
3. Lagrange Multipliers
4. Non-linearly Separable Data
5. Soft-margin
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 1 / 33
Analytical Geometry
Analytical Geometry
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 2 / 33
Analytical Geometry
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 3 / 33
Maximum Margin Classifiers
Maximum margin classifiers
• Assume that the data are linearly separable
• Decision boundary equation:
y(x) = w.x + b
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 4 / 33
Maximum margin classifiers
• Margin: the smallest distance between the decision boundary and any of the samples.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 5 / 33
Maximum margin classifiers
• Margin: the smallest distance between the decision boundary and any of the samples.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 5 / 33
Maximum margin classifiers
• Support vectors: samples at the two margins.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 6 / 33
Maximum margin classifiers
• Scaling y (support vectors) to be 1 or -1:
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 7 / 33
Maximum margin classifiers
• Signed distance between the decision boundary and a sample xn :
y(xn )
||w||
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 8 / 33
Maximum margin classifiers
• Signed distance between the decision boundary and a sample xn :
y(xn )
||w||
• Absolute distance between the decision boundary and a sample xn :
tn .y(xn )
||w||
tn = +1 iff y(xn ) > 0 and tn = −1 iff y(xn ) < 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 8 / 33
Maximum margin classifiers
• Maximum margin:
1
arg max minn (tn .(w.xn + b))
w,b ||w||
with the constraint:
tn .(w.xn + b) ≥ 1
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 9 / 33
Maximum margin classifiers
• To be optimized:
1
arg min kwk2
w,b 2
with the constraint:
tn .(w.xn + b) ≥ 1
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 10 / 33
Lagrange Multipliers
Optimization using Lagrange multipliers
Joseph-Louis Lagrange born 25 January 1736 – Paris, 10
April 1813; also reported as Giuseppe Luigi Lagrange,
was an Italian Enlightenment Era mathematician and as-
tronomer. He made significant contributions to the fields
of analysis, number theory, and both classical and celestial
mechanics.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 11 / 33
Optimization using Lagrange multipliers
• Problem:
arg max f (x)
x
with the constraint:
g(x) = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 12 / 33
Optimization using Lagrange multipliers
• Solution is the stationary point of the Lagrange function:
L(x, λ) = f (x) + λ.g(x)
such that:
∂L(x, λ)/∂xn = ∂f (x)/∂xn + λ.∂g(x)/∂xn = 0
and
∂L(x, λ)/∂λ = g(x) = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 13 / 33
Optimization using Lagrange multipliers
• Example:
f (x) = 1 − u2 − v 2
with the constraint:
g(x) = u + v − 1 = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 14 / 33
Optimization using Lagrange multipliers
• Lagrange function:
L(x, λ) = f (x) + λ.g(x) = (1 − u2 − v 2 ) + λ.(u + v − 1)
∂L(x, λ)/∂u = ∂f (x)/∂u + λ.∂g(x)/∂u = −2u + λ = 0
∂L(x, λ)/∂v = ∂f (x)/∂v + λ.∂g(x)/∂v = −2v + λ = 0
∂L(x, λ)/∂λ = g(x) = u + v − 1 = 0
• Solution: u = 1/2 and v = 1/2
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 15 / 33
Optimization using Lagrange multipliers
• Example:
f (x) = 1 − u2 − v 2
with the constraint:
g(x) = u + v − 1 = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 16 / 33
Optimization using Lagrange multipliers
• Problem:
arg max f (x)
x
with the inequality constraint:
g(x) ≥ 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 17 / 33
Optimization using Lagrange multipliers
Solution is the stationary point of the Lagrange function:
L(x, λ) = f (x) + λ.g(x)
such that:
∂L(x, λ)/∂xn = ∂f (x)/∂xn + λ.∂g(x)/∂xn = 0
and
g(x) ≥ 0
λ≥0
λ.g(x) = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 18 / 33
Optimization using Lagrange multipliers
• To be optimized:
1
arg min kwk2
w,b 2
with the constraint:
tn .(w.xn + b) ≥ 1)
• Lagrange function for maximum margin classifier:
1 X
L(w, b, a) = kwk2 − an .(tn .(w.xn + b) − 1)
2
n=1..N
tn .(w.xn + b) − 1 ≥ 0
an ≥ 0
an .(tn .(w.xn + b) − 1) = 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 19 / 33
Optimization using Lagrange multipliers
• Lagrange function for maximum margin classifier:
1 X
L(w, b, a) = kwk2 − an .(tn .(w.xn + b) − 1)
2
n=1..N
• Solution for w:
∂(w, b, a)/∂w = 0
X
w= an .tn .xn
n=1..N
X
∂L(w, b, a)/∂b = an .tn = 0
n=1..N
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 20 / 33
Optimization using Lagrange multipliers
• Lagrange function for maximum margin classifier:
1 X
L(w, b, a) = kwk2 − an .(tn .(w.xn + b) − 1)
2
n=1..N
• Solution for a: dual representation to be optimized
X 1 X X
L∗ (a) = an − an .am .tn .tm .xn .xm
2
n=1..N n=1..N m=1..N
with the constraints:
an ≥ 0
X
an .tn = 0
n=1..N
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 21 / 33
Optimization using Lagrange multipliers
• Lagrange function for maximum margin classifier:
1 X
L(w, b, a) = kwk2 − an .(tn .(w.xn + b) − 1)
2
n=1..N
• Solution for a: dual representation to be optimized
X 1 X X
L∗ (a) = an − an .am .tn .tm .xn .xm
2
n=1..N n=1..N m=1..N
Why optimization via dual representation?
• Sparsity: an = 0 if xn is not a support vector.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 22 / 33
Optimization using Lagrange multipliers
• Lagrange function for maximum margin classifier:
1 X
L(w, b, a) = kwk2 − an .(tn .(w.xn + b) − 1)
2
n=1..N
an .(tn .(w.xn + b) − 1) = 0
• Solution for b:
1 X
b= am .tm .xm .xn
|S|
n∈S
where S is the set of support vectors (an 6= 0)
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 23 / 33
Optimization using Lagrange multipliers
• Classification: X
y(x) = w.x + b = an .tn .xn .x + b
n=1..N
y(x) > 0 → +1
y(x) < 0 → −1
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 24 / 33
Non-linearly Separable Data
Kernel trick for non-linearly separable data
• Mapping the data points into a high dimensional feature space.
• Example 1:
• Original space: (x)
• New space: (x, x2 )
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 25 / 33
Kernel trick for non-linearly separable data
• Example 2:
• Original space: (u, v)
• New space: ((u2 + v 2 )1/2 , arctan(v/u))
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 26 / 33
Kernel trick for non-linearly separable data
Example 3: XOR function
In1 In2 t
0 0 0
0 1 1
1 0 1
1 1 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 27 / 33
Kernel trick for non-linearly separable data
Example 3: XOR function
In1 In2 In3 Output
0 0 1 1
0 1 0 0
1 0 0 0
1 1 0 1
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 28 / 33
Kernel trick for non-linearly separable data
• Classification in the new space:
X
y(x) = w.φ(x) + b = an .tn .φ(xn ).φ(x) + b
n=1..N
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 29 / 33
Kernel trick for non-linearly separable data
• Classification in the new space:
X
y(x) = w.φ(x) + b = an .tn .φ(xn ).φ(x) + b
n=1..N
• Computational complexity of φ(xn ).φ(x) is high due to the high dimension of φ(.).
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 29 / 33
Kernel trick for non-linearly separable data
• Classification in the new space:
X
y(x) = w.φ(x) + b = an .tn .φ(xn ).φ(x) + b
n=1..N
• Computational complexity of φ(xn ).φ(x) is high due to the high dimension of φ(.).
• Kernel trick:
φ(xn ).φ(xm ) = K(xn , xm )
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 29 / 33
Kernel trick for non-linearly separable data
• A typical kernel function:
K(u, v) = (1 + u.v)2
√ √ √
φ((u1 .u2 , ..., ud )) = (1, 2u1 , 2u2 , ..., 2ud ,
√ √ √
2u1 .u2 , 2u1 .u3 , ..., 2ud−1 .ud ,
u21 , u22 , ..., u2d )
X X X X
φ(u).φ(v) = 1 + 2 ui .vi + 2 ui .vi .uj .vj + u2i vi2
i=1..d i=1..d−1 j=i+1..d i=1..d
φ(u.φ(v) = K(u, v)
• Is φ(x) guaranteed to be separable?
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 30 / 33
Soft-margin
Soft margin SVM
• Soft-margin SVM: to allow some of the training samples to be misclassified.
• Slack variable: ξ
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 31 / 33
Soft margin SVM
• New constraints:
tn .(w.xn + b) ≥ 1 − ξn
ξn ≥ 0
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 32 / 33
Soft margin SVM
• New constraints:
tn .(w.xn + b) ≥ 1 − ξn
ξn ≥ 0
• To be minimized:
1 X
||w||2 = C ξn
2
n=1..N
C > 0: controls the trade-off between the margin and slack variable penalty
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 32 / 33
Summary
• SVM is a sparse kernel method.
• Soft margin SVM is to deal with non-linearly separable data after kernel mapping.
Lecturer: Duc Dung Nguyen, PhD. Contact: [email protected] Machine Learning 33 / 33
