Codestin Search App

itamarst · 2026-06-24T22:38:24Z

Instead of checking every value in the array for being a scalar, _get_counts just uses Counter's default logic, which seems to work just fine.

I also optimized is_scalar_nan since it is still called quite a lot, e.g. for every unique value, so relevant in cases where the number of unique values is the same as the number of values this can perhaps add up.

Big picture benchmark

Before, using Python 3.14t (for the better concurrency) on https://gist.github.com/ogrisel/1b24301bfc90d61ab2138bb7fbf7f623:

n_jobs: 1, duration: 18.465 s, speedup: 1.00x, Best R2: 0.8654
n_jobs: 2, duration: 10.455 s, speedup: 1.77x, Best R2: 0.8654
n_jobs: 4, duration: 6.404 s, speedup: 2.88x, Best R2: 0.8654
n_jobs: 8, duration: 4.636 s, speedup: 3.98x, Best R2: 0.8654

After:

n_jobs: 1, duration: 13.872 s, speedup: 1.00x, Best R2: 0.8654
n_jobs: 2, duration: 8.879 s, speedup: 1.56x, Best R2: 0.8654
n_jobs: 4, duration: 5.704 s, speedup: 2.43x, Best R2: 0.8654
n_jobs: 8, duration: 4.265 s, speedup: 3.25x, Best R2: 0.8654

Microbenchmark of `is_scalar_nan`

Before (3.14t):

In [3]: %timeit is_scalar_nan("")
470 ns ± 0.248 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [4]: %timeit is_scalar_nan(np.nan)
809 ns ± 0.24 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

After:

In [3]: %timeit is_scalar_nan("")
235 ns ± 0.231 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [4]: %timeit is_scalar_nan(np.nan)
507 ns ± 5.64 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

…heck.

itamarst · 2026-06-24T22:42:25Z

            ["a", "b", "c", "e"],
            [16, 4, 20, 0],
        ),
+        # Before #34385 was fixed, the result was [2, 6, 6, 6]. In practice, in


If the old behavior is desirable, it can easily be added back; as the comment says, I don't think it's particularly relevant since this code path is mostly just for strings.

I think this new behaviors makes more sense. But it impacts _unique and there the behavior is not good anymore, indeed it breaks this invariant:remains true:

_, counts = _unique(values, return_counts=True) assert len(values) == sum(counts)

You can add this check in test_unique_util_with_all_missing_values it will break with your changes but not in main.

Let's modify _unique_python so that this invariant is preserved. Codex suggested this fix:

191 - ret += (_get_counts(values, uniques),) 191 + counts = _get_counts(values, uniques) 192 + if missing_values.nan: 193 + counts[-1] = sum(is_scalar_nan(value) for value in values) 194 + ret += (counts,)

A bit brittle, but that works.

Why do you think it destroys the invariant? I am sleep deprived, so will think about this more, but at first glance it fixes the invariant, previously the invariant was not correct when there were multiple kinds of nans, now it is correct. Nans do get counted in this PR.

That change will slow things down (it's O(n) on values), if some change is necessary there are likely better ways to do it.

To be more specific: previously the result in new test was [2, 6, 6, 6], and the sum of that is not len(values) == 8. Whereas now it is.

Sorry this is messy (but not my fault, it's the sklearn code's fault 😜 ), it comes from the interaction between the _unique_python and _get_counts.

Just add those two lines at the end of test_unique_util_with_all_missing_values, run the test with pdb, and you'll get it I think:

_, counts = _unique(values, return_counts=True) assert len(values) == sum(counts)

That change will slow things down (it's O(n) on values)

Yes sorry this is quite ugly. I let codex write it without realizing it was bad, I'll try to propose something better.

cakedev0 · 2026-06-25T10:37:06Z

Looks like another cool PR 😄 I'm going to take a look.

cakedev0 · 2026-06-25T11:04:01Z

A small focused benchmark shows that fit goes from 23ms to 7ms (~3x speedup) on this example 🚀

from timeit import timeit

from sklearn.datasets import fetch_openml
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder

X, _ = fetch_openml(data_id=42165, as_frame=True, return_X_y=True)
X = X.loc[:, X.select_dtypes(include=["object", "string"]).columns]

one_hot = OneHotEncoder(handle_unknown="ignore", max_categories=10)
ordinal = OrdinalEncoder(max_categories=10)

for name, encoder in [("one hot", one_hot), ("ordinal", ordinal)]:
    print(name, "fit", round(timeit(lambda : encoder.fit(X), number=100) * 10, 1), "ms")
    encoder.fit(X)
    print(name, "transform", round(timeit(lambda : encoder.transform(X), number=100) * 10, 1), "ms")

shipitdev

This is an awesome speedup. I mostly read these PRs to learn, and seeing a massive performance bump just from swapping to a built-in Python collection is super cool.

I was looking at the test file diff where the output changed from [2, 6, 6, 6]. I did some digging and it looks like the old custom _NaNCounter explicitly grouped all NaNs together, whereas the native Counter treats separate float("nan") objects as distinct keys because of their memory addresses. Honestly, the new behavior feels more intuitive to me anyway since they are technically separate objects.

Awesome work man, and thanks for documenting the test changes so clearly!

cakedev0

Overall LGTM and great speedup, let's just fix a small thing.

Also: can you add a changelog? Something like this I guess:

Improved the speed of :meth:`preprocessing.OneHotEncoder.fit`
and :meth:`preprocessing.OrdinalEncoder.fit` on object/string categorical
features when category counts are needed, for instance with `min_frequency`
or `max_categories`.

cakedev0 · 2026-06-25T11:33:34Z

            ["a", "b", "c", "e"],
            [16, 4, 20, 0],
        ),
+        # Before #34385 was fixed, the result was [2, 6, 6, 6]. In practice, in


I think this new behaviors makes more sense. But it impacts _unique and there the behavior is not good anymore, indeed it breaks this invariant:remains true:

_, counts = _unique(values, return_counts=True) assert len(values) == sum(counts)

You can add this check in test_unique_util_with_all_missing_values it will break with your changes but not in main.

Let's modify _unique_python so that this invariant is preserved. Codex suggested this fix:

191 - ret += (_get_counts(values, uniques),) 191 + counts = _get_counts(values, uniques) 192 + if missing_values.nan: 193 + counts[-1] = sum(is_scalar_nan(value) for value in values) 194 + ret += (counts,)

A bit brittle, but that works.

itamarst · 2026-06-25T13:04:19Z

Ok I added a news file. See my inline replies in the thread to requested code change, it's not clear me to that a change is necessary there.

itamarst · 2026-06-25T13:13:25Z

4 separate commits later, the changelog entry is finally in (so sleep deprived); thanks for writing it, I wouldn't have been able to give a good description of the impact.

I assume the Linter failure is something on main that will be fixed in a bit.

cakedev0

Ok, I think I have a clear understanding on how to go with this now, see my suggestions.

Handling various nans makes things very painful 😭

itamarst · 2026-06-25T15:12:27Z

OK, how's that? Fixes the nan merging issue, while not being O(unique items).

cakedev0

A few nits about comments but otherwise LGTM

Nice solution for the "multi-nans" handling 👍

lorentzenchr

Looks like a nice improvement, even cleaner and less code.

ogrisel

Overall LGTM once the above suggestions have been addressed.

ogrisel · 2026-06-30T15:29:11Z

FYI, I tried #34386 (comment) on my Apple M4 laptop and I measure a 4x speed-up for both encoders.

…tely.

…' into 34385-unique-counts-object-dtype

itamarst · 2026-06-30T16:45:46Z

OK, I think I addressed everything.

ogrisel

Thank you very much. Much cleaner and more efficient code ;)

lorentzenchr

Nice improvement

…ding (scikit-learn#34386)

itamarst added 4 commits June 24, 2026 17:50

Speed up in cases where there are many duplicate values.

9c89bfa

Microoptimizations: fewer attribute lookups, and remove unnecessary c…

1e2787e

…heck.

Plain old Counter seems to work just fine...

e40e52c

Explain motivation

b315a0e

github-actions Bot added the module:utils label Jun 24, 2026

itamarst commented Jun 24, 2026

View reviewed changes

itamarst changed the title ~~PERF Speed up unique counts for strings~~ PERF Speed up unique counts for strings, thereby speeding up encoding Jun 24, 2026

itamarst marked this pull request as ready for review June 25, 2026 10:11

shipitdev approved these changes Jun 25, 2026

View reviewed changes

cakedev0 reviewed Jun 25, 2026

View reviewed changes

itamarst added 2 commits June 25, 2026 09:02

Add a news entry.

edbfe5f

News entry

34477e4

itamarst added 2 commits June 25, 2026 09:07

Correct name

1d746a9

Rename harder

4bdccbc

cakedev0 reviewed Jun 25, 2026

View reviewed changes

Comment thread sklearn/utils/_encode.py

Comment thread sklearn/utils/tests/test_encode.py Outdated

Combine counts of all nans, to match _unique() better.

f7526d6

Merge branch 'main' into 34385-unique-counts-object-dtype

14b4135

cakedev0 approved these changes Jun 25, 2026

View reviewed changes

Comment thread sklearn/utils/_encode.py Outdated

Comment thread sklearn/utils/tests/test_encode.py Outdated

Improve comments.

c3211ed

lorentzenchr reviewed Jun 26, 2026

View reviewed changes

Comment thread doc/whats_new/upcoming_changes/sklearn.preprocessing/34386.efficiency.rst Outdated

Comment thread sklearn/utils/tests/test_encode.py Outdated

Comment thread sklearn/utils/tests/test_encode.py Outdated

Comment thread sklearn/utils/_encode.py Outdated

This was referenced Jun 26, 2026

PRF: Encoders (very) fast path for pandas/polars categoricals #34393

Closed

BENCH Add benchmark for encoders #34394

Open

ogrisel added the Performance label Jun 26, 2026

ogrisel approved these changes Jun 30, 2026

View reviewed changes

itamarst added 4 commits June 30, 2026 11:35

Update changelog entry.

ab0ff9d

Clearer explanation of what is going on when comparing lists.

1dd0a4f

Add an attribute to MissingValues instead of passing it around separa…

f792073

…tely.

Merge remote-tracking branch 'origin/34385-unique-counts-object-dtype…

a14c8f0

…' into 34385-unique-counts-object-dtype

itamarst requested a review from lorentzenchr June 30, 2026 16:45

ogrisel approved these changes Jul 1, 2026

View reviewed changes

lorentzenchr approved these changes Jul 3, 2026

View reviewed changes

lorentzenchr changed the title ~~PERF Speed up unique counts for strings, thereby speeding up encoding~~ ENH PERF Speed up unique counts for strings, thereby speeding up encoding Jul 3, 2026

lorentzenchr merged commit dfe68af into scikit-learn:main Jul 3, 2026
39 checks passed

prady0t pushed a commit to prady0t/scikit-learn that referenced this pull request Sep 2, 2026

ENH PERF Speed up unique counts for strings, thereby speeding up enco…

af44c3c

…ding (scikit-learn#34386)

jeremiedbb mentioned this pull request Sep 8, 2026

Release 1.9.1 #34905

Merged

14 tasks

Uh oh!

Uh oh!

Conversation

itamarst commented Jun 24, 2026

Big picture benchmark

Microbenchmark of is_scalar_nan

Uh oh!

itamarst Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

cakedev0 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

itamarst Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itamarst Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

cakedev0 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

cakedev0 commented Jun 25, 2026

Uh oh!

cakedev0 commented Jun 25, 2026

Uh oh!

shipitdev left a comment

Choose a reason for hiding this comment

Uh oh!

cakedev0 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cakedev0 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

itamarst commented Jun 25, 2026

Uh oh!

itamarst commented Jun 25, 2026

Uh oh!

cakedev0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

itamarst commented Jun 25, 2026

Uh oh!

cakedev0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Jun 30, 2026

Uh oh!

itamarst commented Jun 30, 2026

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Microbenchmark of `is_scalar_nan`

itamarst Jun 25, 2026 •

edited

Loading

cakedev0 left a comment •

edited

Loading