[MRG] Improve stability of SGDClassifier / SGDRegressor with gradient clipping #3883
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The l2 weight decay rescaling is also kept positive (or null) in case of strong regularization.
|
Can you bench this against master please? |
|
+1 for a benchmark otherwise looks good to me |
|
I am wondering wether there is a better way to compute the clipping in cython. |
|
LGTM. As long as you use |
|
The benchmark seems to show that the change is fine. Here is my script: import numpy as np
from time import time
from sklearn.linear_model import SGDClassifier
rng = np.random.RandomState(42)
n_samples = int(1e6)
data = rng.randn(n_samples, 100)
target = rng.randint(0, 2, n_samples)
durations = []
for i in range(10):
t0 = time()
SGDClassifier(n_iter=5, random_state=10).fit(data, target)
d = time() - t0
durations.append(d)
print("%0.3fs" % d)
print("%0.3f+/-%0.3fs" % (np.mean(durations), np.std(durations)))
|
|
Thanks @larsmans for the tip. Shall I merge? |
larsmans
added a commit
that referenced
this pull request
Nov 26, 2014
Improve stability of SGDClassifier / SGDRegressor with gradient clipping
|
Thanks! Let me add a whats_new.rst entry. |
|
Great job! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
squared_hingeloss ofSGDClassifier(and potentially thesquaredloss ofSGDRegressor) tend to trigger numerical overflows even on normalized data for some hyper parameter combinations.This PR fixes that issue by clipping
dlossto1e12. All existing still tests pass.I have also had to prevent strong l2 regularization with large learning rates to trigger negative scales (which are meaningless and can also cause numerical divergence if lower than -1). Instead I set the weights to zero in that case. A new non regression tests highlights this case as well.
Both non regression tests were inspired by #3040. They both fail at epoch #2 and #3 of the iris data with the
sgd_fast.pyximplementation from master.