Codestin Search App

snath-xoc · 2025-11-21T14:18:16Z

Reference Issues/PRs

Follow up on previously closed PR #31147 and issue #30707

What does this implement/fix? Explain your changes.

Adds sample weights within the QuantileTransformer under preprocessing using the _weighted_percentile function.

…rcentile, add XFAIL for sparse_data

github-actions · 2025-11-21T14:19:29Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: fa29103. Link to the linter CI: here}

snath-xoc · 2025-11-21T14:30:25Z

@ogrisel and @kaekkr

snath-xoc · 2025-11-22T22:56:08Z

Hmmmm I am running with an issue that seems similar to here: actions/runner#449

Mohataseem89 · 2025-12-08T15:12:15Z

the integration of _weighted_percentile and the updates based on earlier review feedback look like a solid step toward bringing sample-weight support to QuantileTransformer.
good to see the switch to the unified _weighted_percentile logic — that aligns with recent changes in utils.stats.
as @ogrisel mentioned earlier, using percentile ranks correctly (per feature and per reference value) will be key for correctness.

antoinebaker

Thanks @snath-xoc for the PR ! The logic seems fine to me.

I made a few suggestions regarding the docstrings / warning message etc, but I don't have a strong opinion. I let you make your own mind :)

Do not hesitate to ping me when it's ready for another review.

antoinebaker · 2026-02-17T16:19:27Z

+        if sample_weight is None:
+            n_samples = X.shape[0]
+        else:
+            sample_weight = _check_sample_weight(sample_weight, X, dtype=X.dtype)
+            n_samples = np.sum(sample_weight)


If n_samples stands for the sum of sample weights, we should perhaps mention it in the docstring and warning messages ? But on the other hand it may be too verbose (at each time saying "the number of samples or the total sum of weights if provided").

Perhaps a middle ground would be to replace total number of samples -> effective number of samples in the warning messages and the n_quantiles docstring ? Maybe we should replace n_samples by a better name ?

@antoinebaker yes we could use effective_sample_size this was at some point tried in the #30751? Let me update.

check if it makes sense now?

yep I like the name !

antoinebaker · 2026-02-17T16:36:49Z

        if self.n_quantiles > n_samples:
            warnings.warn(
                "n_quantiles (%s) is greater than the total number "
                "of samples (%s). n_quantiles is set to "
                "n_samples." % (self.n_quantiles, n_samples)
            )


Here is the warning message in question in #32761 (comment)

snath-xoc · 2026-07-03T11:29:56Z

@antoinebaker let me know what you think?

jeremiedbb · 2026-07-29T10:24:30Z

The n_unique solution seemed cleaner in my mind and is more inline with KBinsDiscretizer but then with all the other raised errors it seems to not be so good.

the n_unique solution is a good idea but it should be done before computing the quantiles otherwise we waste time computing too many quantiles. n_quantiles could instead be determined from the number of unique rows in X. However it also has a cost and should not be done of large datasets. And is probably overkill for the issue we're trying to solve.

So in the end I believe that just dropping the cap on n_quantiles if fine. It adds some overhead on small datasets because we compute more quantiles than necessary, but I think it's acceptable. Here's a quick benchmark:

  config                no-cap n_q_  capped n_q_   no-cap ms   capped ms    diff ms   slowdown
----------------------------------------------------------------------------------------------
  n=10,   f=5                  1000           10       0.603       0.493      0.110      1.22x
  n=10,   f=50                 1000           10       2.766       1.805      0.961      1.53x
  n=50,   f=5                  1000           50       0.626       0.519      0.107      1.21x
  n=50,   f=50                 1000           50       3.039       2.095      0.945      1.45x
  n=100,  f=5                  1000          100       0.668       0.575      0.093      1.16x
  n=100,  f=50                 1000          100       3.406       2.580      0.826      1.32x
  n=500,  f=10                 1000          500       2.652       2.553      0.099      1.04x
  n=500,  f=100                1000          500      22.863      21.794      1.069      1.05x
  n=2000, f=10                 1000         1000      22.814      22.815     -0.001      1.00x
  n=2000, f=100                1000         1000     222.813     222.601      0.212      1.00x
  n=5000, f=50                 1000         1000      26.866      27.036     -0.170      0.99x

The slowdown is only significant for tiny datasets (up to 50%), but the purpose of QuantileTransformer is to deal with large datasets. I don't think we worry too much about performance for datasets that small for which QuantileTransformer will never be used in practice (besides testing).

jeremiedbb · 2026-07-29T10:26:09Z

-    n_quantiles_ : int
-        The actual number of quantiles used to discretize the cumulative
-        distribution function.
-


We can't just remove a public attribute like that. Either we keep it and it's always equal to n_quantiles, or we go through a deprecation cycle. (note that to separate concern we can keep it for now in this PR and decide to deprecate later)

antoinebaker · 2026-07-29T15:24:26Z

+        landmarks used to discretize the cumulative distribution function. If
+        n_quantiles is larger than the effective sample size (sum of sample
+        weights if provided, total number of samples otherwise), n_quantiles is
+        set to the effective sample size as a larger number of quantiles does
+        not give a better approximation of the cumulative distribution function


To be removed if we no longer cap the number of quantiles, and probably add a ..versionchanged directive to explain the change.

antoinebaker · 2026-07-29T15:28:32Z

+            raise NotImplementedError(
+                "sample_weight is not supported for sparse input."
            )
-        self.n_quantiles_ = max(1, min(self.n_quantiles, n_samples))


Following on https://github.com/scikit-learn/scikit-learn/pull/32761/changes#r3673274161 we should (for now) keep the attribute:

self.n_quantiles_ = self.n_quantiles

antoinebaker · 2026-07-29T15:35:28Z

@antoinebaker let me know what you think?

Sorry for the late answer :( Just removing the criterion seems a good solution, I agree with @jeremiedbb that the performance drawback is not a serious one:

The slowdown is only significant for tiny datasets (up to 50%), but the purpose of QuantileTransformer is to deal with large datasets. I don't think we worry too much about performance for datasets that small for which QuantileTransformer will never be used in practice (besides testing).

snath-xoc · 2026-07-31T11:42:21Z

Thanks @antoinebaker and @jeremiedbb have updated seems fine now.

antoinebaker

LGTM @snath-xoc :)

EDIT: there is a doctest failure to fix in preprocessing.rst

jeremiedbb · 2026-08-16T13:01:33Z

EDIT: there is a doctest failure to fix in preprocessing.rst

We should set n_quantiles=150 in this doc example now that it's not capped.

jeremiedbb · 2026-08-16T13:04:47Z

+            self.quantiles_ = _weighted_percentile(
+                X,
+                sample_weight=sample_weight,
+                percentile_rank=references,
+                average=True,
+            )
+            self.quantiles_ = np.asarray(self.quantiles_).T


There's some sort of inconsistency here: input with NaNs that have non-zero weights are not discarded while in the non-weighted case inputs with non-zero weights are discarded.

Let's add a test for that

Good catch, turns out the NaN handling is not done correctly, I added in a fix for it, let me know what you think.

jeremiedbb · 2026-08-16T13:07:28Z

+                self.quantiles_.append(
+                    np.nanquantile(
+                        column_data, references / 100, method="averaged_inverted_cdf"
+                    )
+                )
        self.quantiles_ = np.transpose(self.quantiles_)


the change of method is a change of behavior. We should document it in the changelog.

Also, now we're doing

references = self.references_ * 100 np.nanquantile(column_data, references / 100, method="averaged_inverted_cdf")

Let's just use self.references_ directly.

snath-xoc · 2026-09-11T17:09:12Z

@ogrisel @jeremiedbb and @antoinebaker this should be ready now?

snath-xoc and others added 8 commits November 20, 2025 20:24

lazy test added

9fcf536

merge main

3418041

add sample_weight parameter in fit function

efcbf37

add support for sample_weight in QuantileTransformer

3debd10

remove custom weighted_percentile function, use _averaged_weighted_pe…

8d34d12

…rcentile, add XFAIL for sparse_data

add review

831e5ef

add missed reviews

2a40be3

Merge branch 'main' into quantile-transformer-sample-weight

5d03d5d

github-actions Bot added module:preprocessing module:utils labels Nov 21, 2025

imports

d17f120

snath-xoc added 4 commits November 21, 2025 14:30

add changelog

9538f52

add changelog

98d0147

fix keyword error

907c066

fix bugs

a42843a

snath-xoc and others added 5 commits November 22, 2025 23:22

fix tests

c468f7f

add docstring

0146e82

add space

a4241e9

Merge branch 'main' into quantile-transformer-sample-weight

fa29103

Merge branch 'main' into quantile-transformer-sample-weight

7024897

adrinjalali added this to Labs Dec 11, 2025

github-project-automation Bot moved this to Todo in Labs Dec 11, 2025

adrinjalali assigned snath-xoc Dec 11, 2025

adrinjalali moved this from Todo to In progress in Labs Dec 11, 2025

adrinjalali unassigned snath-xoc Jan 8, 2026

antoinebaker reviewed Feb 17, 2026

View reviewed changes

remove redundant doc string

77a52f5

jeremiedbb reviewed Jul 29, 2026

View reviewed changes

antoinebaker reviewed Jul 29, 2026

View reviewed changes

snath-xoc and others added 2 commits July 30, 2026 23:07

Merge branch 'main' into quantile-transformer-sample-weight

6b3720d

remove n_quantile cap

da8f052

antoinebaker approved these changes Jul 31, 2026

View reviewed changes

jeremiedbb reviewed Aug 16, 2026

View reviewed changes

snath-xoc and others added 8 commits August 30, 2026 14:57

add tests and clean up docs

dcf8c83

filter out NaNs in dense fit

e72b6ad

add NaN handling

bd41e01

fix nan handling second try

3786fe2

Merge branch 'main' into quantile-transformer-sample-weight

9556a02

fix docstring

fdd536d

Merge branch 'main' into quantile-transformer-sample-weight

ce211c9

Merge branch 'main' into quantile-transformer-sample-weight

0c339ac

github-actions Bot added the CI:Linter failure The linter CI is failing on this PR label Sep 7, 2026

ogrisel moved this from In progress to PR waiting for reviews in Labs Sep 7, 2026

snath-xoc commented Sep 7, 2026

View reviewed changes

Comment thread sklearn/preprocessing/tests/test_data.py

snath-xoc and others added 3 commits September 7, 2026 14:40

Update sklearn/preprocessing/tests/test_data.py

912c509

Merge branch 'main' into quantile-transformer-sample-weight

1a499ec

remove whitespaces

737f2b4

github-actions Bot removed the CI:Linter failure The linter CI is failing on this PR label Sep 10, 2026

Merge branch 'main' into quantile-transformer-sample-weight

b3ac71a

Uh oh!

Uh oh!

Conversation

snath-xoc commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Uh oh!

github-actions Bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

snath-xoc commented Nov 21, 2025

Uh oh!

snath-xoc commented Nov 22, 2025

Uh oh!

Mohataseem89 commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antoinebaker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snath-xoc Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

antoinebaker Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snath-xoc commented Jul 3, 2026

Uh oh!

jeremiedbb commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antoinebaker commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snath-xoc commented Jul 31, 2026

Uh oh!

antoinebaker left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremiedbb commented Aug 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

snath-xoc commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

snath-xoc commented Nov 21, 2025 •

edited

Loading

github-actions Bot commented Nov 21, 2025 •

edited

Loading

Mohataseem89 commented Dec 8, 2025 •

edited

Loading

snath-xoc Feb 26, 2026 •

edited

Loading

antoinebaker Feb 17, 2026 •

edited

Loading

antoinebaker commented Jul 29, 2026 •

edited

Loading

antoinebaker left a comment •

edited

Loading