Revisiting Logistic Regression & Nave Bayes
Aarti Singh
Machine Learning 10-701/15-781 Jan 27, 2010
Generative and Discriminative Classifiers
Training classifiers involves learning a mapping f: X -> Y, or P(Y|X) Generative classifiers (e.g. Nave Bayes) Assume some functional form for P(X,Y) (or P(X|Y) and P(Y)) Estimate parameters of P(X|Y), P(Y) directly from training data Use Bayes rule to calculate P(Y|X) Discriminative classifiers (e.g. Logistic Regression) Assume some functional form for P(Y|X) Estimate parameters of P(Y|X) directly from training data
Logistic Regression
Assumes the following functional form for P(Y|X):
Alternatively,
(Linear Decision Boundary)
DOES NOT require any conditional independence assumptions
3
Connection to Gaussian Nave Bayes
There are several distributions that can lead to a linear decision boundary. As another example, consider a generative model:
Exponential family Observe that Gaussian is a special case
Connection to Gaussian Nave Bayes
Constant term
First-order term
Special case: P(X|Y=y) ~ Gaussian( y,y) where 0 = 1 (cij,0 = cij,1) Conditionally independent cij,y = 0 , i j (Gaussian Nave Bayes)
Generative vs Discriminative
Given infinite data (asymptotically), If conditional independence assumption holds, Discriminative and generative perform similar.
If conditional independence assumption does NOT holds, Discriminative outperforms generative.
Generative vs Discriminative
Given finite data (n data points, p features), Ng-Jordan paper
Nave Bayes (generative) requires n = O(log p) to converge to its asymptotic error, whereas Logistic regression (discriminative) requires n = O(p). Why? Independent class conditional densities * smaller classes are easier to learn * parameter estimates not coupled each parameter is learnt independently, not jointly, from training data.
7
Nave Bayes vs Logistic Regression
Verdict Both learn a linear decision boundary. Nave Bayes makes more restrictive assumptions and has higher asymptotic error, BUT converges faster to its less accurate asymptotic error.
8
Experimental Comparison (Ng-Jordan01)
UCI Machine Learning Repository 15 datasets, 8 continuous features, 7 discrete features
More in Paper
Nave Bayes
Logistic Regression
9
Classification so far (Recap)
10
Classification Tasks
Features, X
Diagnosing sickle cell anemia Tax Fraud Detection
Labels, Y
Anemic cell Healthy cell
Web Classification
Sports Science News
Drive to CMU, Rachels fan, Shop at SH Giant Eagle
Predict squirrel hill resident
Resident 11 Not resident
Classification
Goal:
Sports Science News
Features, X
Labels, Y
Probability of Error
12
Classification
Optimal predictor: (Bayes classifier)
Depends on unknown distribution
13
Classification algorithms
However, we can learn a good prediction rule from training data
Independent and identically distributed
Learning algorithm
So far
Decision Trees K-Nearest Neighbor Nave Bayes Logistic Regression
14
Linear Regression
Aarti Singh
Machine Learning 10-701/15-781 Jan 27, 2010
Discrete to Continuous Labels
Classification Sports Science News
X = Document
Anemic cell Healthy cell
Y = Topic
X = Cell Image
Y = Diagnosis
Regression
Stock Market Prediction
Y=?
X = Feb01
16
Regression Tasks
Weather Prediction
Y = Temp
X = 7 pm
Estimating Contamination
X = new location Y = sensor reading
17
Supervised Learning
Goal:
Sports Science News
Y=?
X = Feb01
Classification:
Regression:
Probability of Error
Mean Squared Error
18
Regression
Optimal predictor: (Conditional Mean)
Dropping subscripts for notational convenience
19
Regression
Optimal predictor: (Conditional Mean)
Intuition: Signal plus (zero-mean) Noise model
Depends on unknown distribution
20
Regression algorithms
Learning algorithm
Linear Regression Lasso, Ridge regression (Regularized Linear Regression) Nonlinear Regression Kernel Regression Regression Trees, Splines, Wavelet estimators,
Empirical Risk Minimizer:
21
Empirical mean
Linear Regression
Least Squares Estimator
- Class of Linear functions
Uni-variate case:
2 = slope
1 - intercept
Multi-variate case:
where
,
22
Least Squares Estimator
23
Least Squares Estimator
24
Normal Equations
p xp p x1 p x1
If
is invertible,
When is invertible ? (Homework 2) Recall: Full rank matrices are invertible. What is rank of What if is not invertible ? (Homework 2) Regularization (later)
25
Geometric Interpretation
Difference in prediction on training set:
is the orthogonal projection of onto the linear subspace spanned by the columns of
26
Revisiting Gradient Descent
Even when is invertible, might be computationally expensive if A is huge.
Gradient Descent Initialize: Update:
0 if
= < .
27
Stop: when some criterion met e.g. fixed # iterations, or
Effect of step-size
Large => Fast convergence but larger residual error Also possible oscillations Small => Slow convergence but small residual error
28
When does Gradient Descent succeed?
View of the algorithm is myopic.
http://www.ce.berkeley.edu/~bayen/
http://demonstrations.wolfram.com
Guaranteed to converge to local minima if
Converges as in jth direction Convergence depends on eigenvalue spread
29
Least Squares and MLE
Intuition: Signal plus (zero-mean) Noise model
log likelihood
Least Square Estimate is same as Maximum Likelihood Estimate under a Gaussian model ! 30
Regularized Least Squares and MAP
What if is not invertible ?
log likelihood
log prior
I) Gaussian Prior
Ridge Regression
Closed form: HW Prior belief that is Gaussian with zero-mean biases solution to small
31
Regularized Least Squares and MAP
What if is not invertible ?
log likelihood
log prior
II) Laplace Prior
Lasso
Closed form: HW Prior belief that is Laplace with zero-mean biases solution to small
32
Ridge Regression vs Lasso
Ridge Regression: Lasso:
HOT! Ideally l0 penalty, but optimization becomes non-convex
s with constant J() (level sets of J()) s with constant l2 norm
2
s with constant l1 norm
1
s with constant l0 norm
Lasso (l1 penalty) results in sparse solutions vector with more zero coordinates Good for high-dimensional problems dont have to store all coordinates!
33
Beyond Linear Regression
Polynomial regression
Regression with nonlinear features/basis functions
Kernel regression - Local/Weighted regression
Regression trees Spatially adaptive regression
34
Polynomial Regression
Univariate case: where ,
Weight of each feature
Nonlinear features
35
Nonlinear Regression
Basis coefficients Nonlinear features/basis functions
Fourier Basis
Wavelet Basis
Good representation for oscillatory functions
Good representation for functions localized at multiple scales
36
Local Regression
Basis coefficients Nonlinear features/basis functions
Globally supported basis functions (polynomial, fourier) will not yield a good representation
37
Local Regression
Basis coefficients Nonlinear features/basis functions
Globally supported basis functions (polynomial, fourier) will not yield a good representation
38
Kernel Regression (Local)
Weighted Least Squares Weigh each training point based on distance to test point
K Kernel
h Bandwidth of kernel
39
Nadaraya-Watson Kernel Regression
constant
40
Nadaraya-Watson Kernel Regression
constant
with box-car kernel #pts in h ball around X Sum of Ys in h ball around X Recall: NN classifier Average <-> majority41vote
Choice of Bandwidth
h
Should depend on n, # training data (determines variance) Should depend on smoothness of function (determines bias)
Large Bandwidth average more data points, reduce noise (Lower variance) Small Bandwidth less smoothing, more accurate fit (Lower bias) Bias Variance tradeoff : More to come in later lectures
42
Spatially adaptive regression
h
If function smoothness varies spatially, we want to allow bandwidth h to depend on X Local polynomials, splines, wavelets, regression trees
43
Regression trees
Binary Decision Tree Num Children? 2 <2
Average (fit a constant ) on the leaves
44
Regression trees
Quad Decision Tree
h
h
- Polynomial fit on each leaf
If Else stop
, then split Compare residual error with and without split
45
Summary
Discriminative vs Generative Classifiers - Nave Bayes vs Logistic Regression Regression - Linear Regression Least Squares Estimator Normal Equations Gradient Descent Geometric Interpretation Probabilistic Interpretation (connection to MLE) - Regularized Linear Regression (connection to MAP) Ridge Regression, Lasso - Polynomial Regression, Basis (Fourier, Wavelet) Estimators - Kernel Regression (Localized) - Regression Trees
46
