Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH Add option to disable subsampling to preprocessing.QuantileTransformer #27636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 21 commits into from
Mar 8, 2024
Merged

ENH Add option to disable subsampling to preprocessing.QuantileTransformer #27636

merged 21 commits into from
Mar 8, 2024

Conversation

@RUrlus
Copy link
Contributor

@RUrlus RUrlus commented Oct 21, 2023

Reference Issues/PRs

Closes #27373

What does this implement/fix? Explain your changes.

The current behaviour of the preprocessing.QuantileTransformer is to disable subsampling when subsampling > n_samples but this is not documented.
This PR adds a documented option (subsample=None) to disable subsampling as discussed with @lorentzenchr.

Any other comments?

Arguably the default should be changed to None in the future as subsampling introduces measurable bias, see #27373 for details.

@github-actions
Copy link

github-actions bot commented Oct 21, 2023 •
edited
Loading

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: a4d4386. Link to the linter CI: here

@RUrlus RUrlus changed the title [MRG] ENH Add option to disable subsampling to preprocessing.QuantileTransformer ENH Add option to disable subsampling to preprocessing.QuantileTransformer Oct 21, 2023
betatim
betatim reviewed Oct 23, 2023
View reviewed changes
betatim
betatim reviewed Oct 23, 2023
View reviewed changes
betatim
betatim reviewed Oct 23, 2023
View reviewed changes
@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from dfc209f to 93e9936 Compare October 25, 2023 15:54
@RUrlus
Copy link
Contributor Author

RUrlus commented Oct 25, 2023

@betatim Thanks for the review, I've incorporated all your feedback

@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from 93e9936 to f37795a Compare October 27, 2023 05:57
betatim
betatim reviewed Oct 30, 2023
View reviewed changes
@glemaitre glemaitre self-requested a review October 30, 2023 14:37
@glemaitre
Copy link
Member

glemaitre commented Oct 30, 2023

In addition to @betatim comment, could you add an entry in the changelog as an enhancement.

glemaitre
glemaitre reviewed Oct 30, 2023
View reviewed changes
GaelVaroquaux and others added 10 commits October 30, 2023 10:31
@GaelVaroquaux @RUrlus
DOC update the sponsors list on landing and about pages (scikit-learn…
be31290 
…#27660)
@lesteve @RUrlus
CI Add workflow to update lock files (scikit-learn#27622)
d5feb2b
@punndcoder28 @RUrlus
TST add missing assert statement (scikit-learn#27677)
3ee449b
@StefanieSenger @adrinjalali @RUrlus
FEA Add metadata routing to SelectFromModel (scikit-learn#27490)
e8d9ba9 
Co-authored-by: Adrin Jalali <[email protected]>
@0urobor0s @RUrlus
DOC use latex notation for d2_tweedie_score docstring (scikit-learn#2…
7301629 
…7681)
@glemaitre @ogrisel @RUrlus
MAINT downcast indices dtype when converting sparse arrays (scikit-le…
af00b3f 
…arn#27372)

Co-authored-by: Olivier Grisel <[email protected]>
@glemaitre @ArturoAmorQ
@thomasjpfan @RUrlus
DOC add an example regarding the multiclass strategies (scikit-learn#…
60a6399 
…25228)

Co-authored-by: Arturo Amor <[email protected]>
Co-authored-by: Thomas J. Fan <[email protected]>
@marenwestermann @RUrlus
TST Extend tests for scipy.sparse.*array in `sklearn/cluster/tests/…
6ffa35f 
…test_optics.py` (scikit-learn#27104)
@RUrlus
ENH Add option to disable subsampling in QuantileTransformer
5014894
@RUrlus
TST test disabling subsampling in QuantileTransformer
62439bd
@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from af60c36 to 2f9fad3 Compare October 30, 2023 17:55
glemaitre
glemaitre reviewed Oct 30, 2023
View reviewed changes
@glemaitre
Copy link
Member

glemaitre commented Oct 30, 2023

@RUrlus Could you merge main in your branch. It seems that something went wrong.

@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from c78ceef to 15bdb4b Compare October 30, 2023 19:16
betatim
betatim reviewed Oct 31, 2023
View reviewed changes
glemaitre
glemaitre reviewed Oct 31, 2023
View reviewed changes
glemaitre
glemaitre reviewed Oct 31, 2023
View reviewed changes
@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from 15bdb4b to b357a5d Compare October 31, 2023 16:51
@RUrlus RUrlus force-pushed the qt_add_optional_subsampling branch from b357a5d to 27e4a81 Compare October 31, 2023 16:55
@RUrlus
Copy link
Contributor Author

RUrlus commented Nov 2, 2023

@betatim, @glemaitre I've incorporated all the feedback. Can we merge this one in?

@RUrlus
Copy link
Contributor Author

RUrlus commented Nov 22, 2023

Hi @betatim, @glemaitre,

Polite ping on the above. Would be nice if we could this one done.

@glemaitre glemaitre self-requested a review November 22, 2023 17:39
@glemaitre
Copy link
Member

glemaitre commented Nov 22, 2023

I'll have a look tomorrow. Thanks for pinging.

glemaitre
glemaitre approved these changes Nov 23, 2023
View reviewed changes
Copy link
Member

@glemaitre glemaitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM apart of those small doc changes.

@glemaitre glemaitre added the Waiting for Second Reviewer First reviewer is done, need a second one! label Nov 23, 2023
jeremiedbb
jeremiedbb approved these changes Mar 8, 2024
View reviewed changes
Copy link
Member

@jeremiedbb jeremiedbb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I applied the remaining requested changes.

LGTM. Thanks @RUrlus

@jeremiedbb jeremiedbb enabled auto-merge (squash) March 8, 2024 10:34
@RUrlus
Copy link
Contributor Author

RUrlus commented Mar 8, 2024

@jeremiedbb thanks for the approval and the fixing the last comments. I was waiting to get feedback from the second before making the changes but this works!

@jeremiedbb jeremiedbb merged commit f4c058a into scikit-learn:main Mar 8, 2024
@RUrlus RUrlus deleted the qt_add_optional_subsampling branch March 8, 2024 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@betatim betatim betatim left review comments

@glemaitre glemaitre glemaitre approved these changes

@jeremiedbb jeremiedbb jeremiedbb approved these changes

Assignees

No one assigned

Labels

module:preprocessing Waiting for Second Reviewer First reviewer is done, need a second one!

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

QuantileTransformer's default subsampling introduces artefacts for unbounded distributions

10 participants

@RUrlus @glemaitre @betatim @jeremiedbb @GaelVaroquaux @lesteve @punndcoder28 @StefanieSenger @0urobor0s @marenwestermann