DEP PassiveAggressiveClassifier and PassiveAggressiveRegressor #29097

lorentzenchr · 2024-05-24T08:52:32Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

This PR deprecates the 2 classes PassiveAggressiveClassifier and PassiveAggressiveRegressor. A user can easily use SGDClassifier and SGDRegressor instead and I cannot figure out the added value of these 2 classes.

@scikit-learn/core-devs ping for decision (in particular in case you are against the deprecation)

github-actions · 2024-05-24T08:53:53Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 05b7bbd. Link to the linter CI: here}

adrinjalali · 2024-05-24T09:12:03Z

I'm +1 on this:

I haven't seen anybody use it (except maybe @amueller ?)
If we deprecate and people start complaining, we can revert deprecation. So in a sense deprecation is a way to check usage here
The deprecation message clearly should always point to an alternative, which I think is the case here.

glemaitre · 2024-05-24T09:21:38Z

The deprecation message clearly should always point to an alternative, which I think is the case here.

I think that we should do a bit more and specify how you get a similar behaviour by setting the right parameters.

lorentzenchr · 2024-05-25T08:36:35Z

In fa3d842, I added C to SGDClassifier and SGDRegressor to make it more accessible.
I could imagine to put the equivalent PA estimators into the SGD docstring - maybe once the deprecation is carried out.

adrinjalali · 2024-05-25T11:02:30Z

Nice. We need to add more information on the docstring about C though. It's very short and not clear what it does when I read it. Otherwise LGTM.

adrinjalali · 2024-10-22T13:31:25Z

@lorentzenchr getting close to the release, would you mind resolving issues here?

lorentzenchr · 2024-10-22T14:47:16Z

@lorentzenchr getting close to the release, would you mind resolving issues here?

I can resolve merge conflicts, but I won't do

We need to add more information on the docstring about C though.

adrinjalali · 2024-10-22T14:51:18Z

sklearn/linear_model/_passive_aggressive.py

+            SGDRegressor(
+                penalty=None,
+                alpha=1.0,
+                C=1.0


I must be missing something, our SGDRegressor doesn't have a C param.

BaseSGDClassifier has an attibute C. This PR exposes this attribute to SGDClassifier (same for regressors).

I don't really like that. Proposition: I revert the exposition of C and write in the deprecation note, using somewhat private API:

clf = SGDClassifier( penalty=None, alpha=1.0, eta0=1.0, learning_rate="pa1", loss="hinge", ) clf.C = 1.0 # THIS IS USING PRIVATE API

See c24c95f. I can revert if you dislike it.

If we're deprecating these estimators and telling users to use SGDRegressor and SGDClassifier instead, it makes sense to actually have a public way of doing that. So I'm in favor of exposing C in both SGD estimators, document them, and then here simply create and equivalent estimator using them.

How about leaving them private for the time being?

I would like to have a dedicated discussion whether to

expose C in SGD; xor

remove C and all passive aggressive things altogether

I think this PR is the right place to have that discussion. I'd like others to weigh in about this point.

maybe @ogrisel @amueller @GaelVaroquaux @adam2392 would have an opinion?

I'm not super familiar with passive aggressive vs SGD but a loose read suggests passive aggressive allows more online updating. Though i thought SGD easily allows online training too, so why would someone use one over the other?

Is it true that the only parameter that makes these different is "C" step size parameter?

Pragmatically: if no one uses "C", then i kinda agree might be simplest to just remove it? If there's something I'm missing tho, lmk.

PassiveAggressiveClassifier was first implemented in #1259 in 2012. I have never seen it used anywhere and I think our SGD... offer enough for online learning (partial_fit). So I would be fine with removing it completely, i.e. also removing C in SGD. Unless someone steps in for keeping it.

Having read through the code and partially papers, I could also live with making learning_rate="pa1" and "pa2" as well as C public in SGD.... In that case I would rename C to something like pa_C.

I am partially in favor of just simplification and deprecating those parameters / the PAClf altogether, unless someone has an actual use case for them.

lorentzenchr · 2024-10-22T18:43:53Z

The decorator deprecated interferes with the signature, e.g. in check_do_not_raise_errors_in_init_or_set_params:

    params = signature(Estimator).parameters

I need help how to avoid that (I won't fix deprecated) or rather to (temporarily) switch of those tests for PassiveAggressiveXX.

lorentzenchr · 2024-10-24T20:35:44Z

Note that 306a5fa should be made a PR of its own in case this PR is not merged.

Edit: I opened #30145.

alexshtf · 2025-05-22T08:00:50Z

I think that the main advantage of those two are that these are algorithms that work without a learning rate. It's one less parameter to tune, so they are theoretically easier to use.

I think the main issue is with their name. I don't have a better name for "Like SGD but you don't need to know the best learning rate", but I think that the lack of their usage is primarily a marketing issue stemming from the name.

In one of the videos in the probabl youtube channel, Vincent also demonstrated when it is useful.

lorentzenchr · 2025-05-23T05:59:10Z

@scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team @mblondel @zaxtax ping for a decision

The Passive-Aggressive algorithm was added in #1259 over 12 years ago. Almost no active core developer seems to know it well and google searches don't show many hits indicating little usage. This raises the question what to do with it. Options:

Keep status quo.
Remove the class PassiveAggressiveClassifier via a deprication cycle, but make it possible to parametrise a SGDClassifier equivalently (which needs to expose the additional parameter C).
Remove it altogether, i.e. class PassiveAggressiveClassifier and private parameter C in SGD.

adrinjalali · 2025-05-23T07:43:11Z

I'm happy to remove these classes, but it'd be nice for having the option of running them somehow, hence I'm in favor of option 2.

betatim · 2025-05-23T08:47:22Z

I tried to quickly educate myself about what C is and my "university of ten minutes effort" degree suggests it is some form of regularisation. Given that SGDClassifier has a way to have regularisation ... how about removing PassiveAggressive* and waiting to see if anyone comes complaining? If there are complaints about it missing and that SGDClassifier isn't a good replacement we can still add C as a hyper-parameter (though maybe with a better description than "max step size")

DEP PassiveAggressiveClassifier and PassiveAggressiveRegressor

d26659a

github-actions bot added the module:linear_model label May 24, 2024

lorentzenchr added 3 commits May 25, 2024 10:33

ENH add C to SGD init

fa3d842

DOC equivvalent estimator

7bdf688

MNT redundant parameter validation

0111071

lorentzenchr added 3 commits May 25, 2024 10:38

DOC whatsnew 1.6

3051404

Merge branch 'main' into dep_passive_aggressive

3af1f2f

MNT after merging main

4f23bc0

lorentzenchr added this to the 1.6 milestone Jun 17, 2024

adrinjalali reviewed Oct 22, 2024

View reviewed changes

lorentzenchr added 3 commits October 22, 2024 19:20

Merge branch 'main' into dep_passive_aggressive

eb1c83e

DOC add new whatsnew

df3825e

CLN make C private again

c24c95f

lorentzenchr added 2 commits October 23, 2024 21:25

FIX signature of deprecated classes

306a5fa

FIX _parameter_constraints in SGD classes

c367698

lorentzenchr force-pushed the dep_passive_aggressive branch from 7b21cfb to c367698 Compare October 23, 2024 21:20

lorentzenchr added 2 commits October 24, 2024 22:32

FIX numpydoc GL09 placement of deprecation in docstring

67f6404

TST remove PA von other tests

246cd04

lorentzenchr mentioned this pull request Oct 24, 2024

FIX signature of deprecated classes #30145

Merged

FIX tests by using learning_rate="pa1"

05b7bbd

glemaitre removed this from the 1.6 milestone Nov 25, 2024

glemaitre added this to the 1.7 milestone Nov 25, 2024

Uh oh!

DEP PassiveAggressiveClassifier and PassiveAggressiveRegressor #29097

Are you sure you want to change the base?

DEP PassiveAggressiveClassifier and PassiveAggressiveRegressor #29097

Uh oh!

Conversation

lorentzenchr commented May 24, 2024

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Uh oh!

github-actions bot commented May 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

adrinjalali commented May 24, 2024

Uh oh!

glemaitre commented May 24, 2024

Uh oh!

lorentzenchr commented May 25, 2024

Uh oh!

adrinjalali commented May 25, 2024

Uh oh!

adrinjalali commented Oct 22, 2024

Uh oh!

lorentzenchr commented Oct 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lorentzenchr Oct 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lorentzenchr commented Oct 22, 2024

Uh oh!

lorentzenchr commented Oct 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexshtf commented May 22, 2025

Uh oh!

lorentzenchr commented May 23, 2025

Uh oh!

adrinjalali commented May 23, 2025

Uh oh!

betatim commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 24, 2024 •

edited

Loading

lorentzenchr Oct 24, 2024 •

edited

Loading

lorentzenchr commented Oct 24, 2024 •

edited

Loading

betatim commented May 23, 2025 •

edited

Loading