Closed
Description
Describe the bug
The example on sparse Logreg surprised me because even going at extremely low regularizations (large C), only 1 feature enters the model:
Investigation lead me to check if StandardScaling X changed the graph. When X is preprocessed, the liblinear solver does not seem to converge.
Steps/Code to Reproduce
from sklearn.svm import l1_min_c
from sklearn import datasets
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np
from time import time
print(__doc__)
# Author: Alexandre Gramfort <[email protected]>
# License: BSD 3 clause
iris = datasets.load_iris()
X = iris.data
y = iris.target
X = X[y != 2]
y = y[y != 2]
# #############################################################################
# Demo path functions
cs = l1_min_c(X, y, loss='log') * np.logspace(0, 10, 30)
X = StandardScaler().fit_transform(X)
print("Computing regularization path ...")
start = time()
clf = linear_model.LogisticRegression(penalty='l1', solver='liblinear',
tol=1e-6, max_iter=int(1e6),
warm_start=True,
intercept_scaling=10000.)
coefs_ = []
for c in cs:
clf.set_params(C=c)
clf.fit(X, y)
coefs_.append(clf.coef_.ravel().copy())
print(clf.coef_.ravel(), clf.intercept_)
Expected Results
the code should run fast (iris has 4 features)
Actual Results
the code gets stuck after the first regularization parameter
I guess something is happening with the scaled intercept column added because liblinear does not fit an unregularized intercept.
Going to a lower intercept scaling (intercept_scaling=1) gives me the way more reasonable graph:
where I still suspect numerical errors to be responsible for the bumps when C becomes large
Versions
Happy to help if it's a known issue
@agramfort