Codestin Search App

adam2392 · 2026-05-11T00:40:05Z

Reference Issues/PRs

Follow-up to #33354 (review)

Towards: #33965

What does this implement/fix? Explain your changes.

Add random splitting capabilities using the bitset as a deterministic hash random seed number to allow splitting up to 2**24 possible categories (encoded via float32). This is a minor modification from the original implementation in NOCATS: Categorical splits for tree-based learners (ctnd.) #12866

AI usage disclosure

Used AI to find the random splitting algorithm mix uint32 function, and then help implement.

Any other comments?

~~Open questions:~~

Within random splitter, allow >256 categories when random splitting? -> yes, we allow up to 2^24-1
Within "best" splitter, allow random splits to be there when certain features exceed the total categories? This would enable pseudo-support for categorical variables with high cardinality -> yes, but in downstream PR
Within random splitter, should we allow categorical features to evaluate many random categorical splits, or just pick one completely at random? -> no, doesn't make sense
- right now, it's evaluate MAX_RANDOM_CATEGORICAL_SPLIT_ATTEMPTS

cakedev0 · 2026-07-23T09:01:18Z

Note: if you merge main/rebase, I'll be happy to take a look at the diff. The idea (deterministic hash with random seed) sounds fun and promising!

adam2392 · 2026-07-23T13:46:19Z

Will do. Was planning on revisiting these PRs this weekend. I can probably finish this off since it's close. Are you looking at other features in the list of #33965 ?

cakedev0 · 2026-07-24T08:11:03Z

Are you looking at other features in the list of #33965 ?

I commented there. But I won't have the bandwidth to really work on trees for the next few weeks.

adam2392 · 2026-07-25T12:53:39Z

This is ready for an initial review if you want to take a quick look, and have any opinions on the open questions @cakedev0 and @lorentzenchr ?

cc: @adrinjalali and @thomasjpfan if you have any quick opinions

Afterwards, I can iterate and then this will be completely ready for review.

cakedev0

I started reviewing without having read your "open questions" 😅

My answers:

Yes! Let's allow >256 categories when random splitting (=> some changes to do in _classes.py
Let's not do it for now, I'll explain why in a subsequent comment.
just pick one completely at random! (see my reasons in a comment of this review)

I'm really excited by this PR! It's fairly simple but so powerful! 🚀

I have a couple of minor/easy-to-resolve comments plus a critical one about MAX_RANDOM_CATEGORICAL_SPLIT_ATTEMPTS.

cakedev0 · 2026-07-27T09:39:41Z

Re 2 (random splits "best" splitter):

I don't think we should do it for now, as it will just be too slow: 256 x n_node_samples...

But let's revisit this idea once we have histogram-based splitting: this would allow adapting the same idea to get a 256 x n_categories complexity. Or let's try it in HGB first.
I do think it's a promising idea, even for categories with a cardinality smaller than 256: it allows a kind of "regularized approximated best split" (regularized because you don't select the best out of $2^{n_c}$ possible splits).

And we should clearly not be done it in this PR:

it's easy to put in a separate PR
I think it's quite a big change, as it requires some random splitter logic to be put into the best splitter.
it's a big decision, that changes the contract exposed to users ("best" is not exactly best anymore)
it needs some dedicated perf measures

adam2392 · 2026-07-29T13:28:45Z

This was all started with @adrinjalali original work.

I’ll wait to see if @lorentzenchr , @thomasjpfan, @adrinjalali or perhaps @ogrisel want to take a second look.

IIRC, I think we still need one more maintainer review for this non trivial change(?) but lmk if me and Arthur are sufficient 😅

ogrisel

Thanks for the PR. Here is a first pass of review.

We will need a follow-up PR to add the categorical_features argument to ExtraTrees ensemble classes.

ogrisel · 2026-08-03T15:48:01Z



+def random_categorical_goes_left(seed, x):
+    return np.array([_py_mix_uint32(seed ^ int(category)) & 1 for category in x])


Please add an inline comment to explain the maths behind this line of code.

Also, let's rename category to category_idx to make it explicit that we expected the ordinal representation of categorical values.

ogrisel · 2026-08-03T16:08:12Z

+    # - for SPLIT_CATEGORICAL_BITSET: left_cat_bitset stores the set of
+    #   categories that go to the left child;
+    # - for SPLIT_CATEGORICAL_HASH: left_cat_bitset[0] stores the hash seed.
    BITSET_DTYPE_C left_cat_bitset


Shouldn't we rename this field to left_cat_bitset_or_hashseed?

Note that since we have to also store a new split_kind field, pickling backward compat is broken between scikit-learn versions anyways.

ogrisel · 2026-08-04T10:16:34Z

                n_categories=int(tree.tree_._n_categories[ftr]),
            )
+        elif split_kind == SPLIT_CATEGORICAL_HASH:
+            threshold = int(tree.tree_._left_cat_bitset[0, 0])


I don't understand this line. Maybe an inline comment could help improve readability.

ogrisel · 2026-08-04T12:20:32Z

This PR also need some conflict resolution and a changelog entry.

adam2392 · 2026-08-04T15:16:23Z

Thanks for the review!

This PR also need some conflict resolution and a changelog entry.

Resolved conflicts, and the changelog entry error I don't understand. Perhaps it's because I changed an older changelog entry? I added one for this PR it seems

adam2392

I added a changelog entry in doc/whats_new/upcoming_changes/sklearn.tree/33972.major-feature.rst. Unsure why the CI is still erroring.

cakedev0 · 2026-08-04T20:00:34Z

The retry strategy looks good 👍 (much cleaner to implement than what I thought, nice!)

lorentzenchr · 2026-08-14T15:01:18Z

    ('n_node_samples',          np.intp),            # 8 bytes  (offset 72)
    ('weighted_n_node_samples', np.float64),         # 8 bytes  (offset 80)
    ('missing_go_to_left',      np.uint8),           # 1 byte   (offset 88)
+    ('split_kind',              np.int8),            # 1 byte  (offset 89)


I guess this extension does not matter much as long as we stay under 3*32 = 96 bytes (or even 2 * 64 = 128 bytes).

Just added for completeness :p

ogrisel

Besides the resolving the pending discussion point above, LGTM. Thanks @adam2392!

ogrisel · 2026-08-17T14:52:31Z

+                n_left = current_split.pos - start
+                n_right = end - current_split.pos
+                if n_left != 0 and n_right != 0:
+                    break


I tried to investigate if it was possible to trigger an infinite loop here passing float values and declaring them as categorical values, but apparently it's not possible from the public API because of the internal ordinal encoding preprocessing.

Still, wouldn't it be safer to cap the number of retries to 100 or something (and raise a warning asking the user to report a minimal reproducer if that limit is reached)?

You're saying cap the retries here directly within Cython? Sure I can add that

Capped hash-seed retries at MAX_RANDOM_CATEGORICAL_SPLIT_RETRIES = 100. If that limit is reached we emit a UserWarning asking for a minimal reproducer and skip the feature. That branch is marked # pragma: no cover and left untested, since ordinal encoding makes it unreachable from the public API.

… feat/rand-cat

adam2392 · 2026-08-20T01:43:05Z

Does @cakedev0 approval count for the +2? heh.

I'll let @lorentzenchr take a look and merge if he's happy with the changes now? As next steps, I think we can now enable in Sklearn.ensemble the categorical support for the forest estimators.

ogrisel · 2026-08-24T07:41:57Z

Could you please duplicate doc/whats_new/upcoming_changes/sklearn.tree/33354.major-feature.rst to doc/whats_new/upcoming_changes/sklearn.tree/33972.major-feature.rst? This will register both PR numbers to the entry at the time of changelog consolidation by towncrier.

adam2392 · 2026-08-25T01:42:21Z

Could you please duplicate doc/whats_new/upcoming_changes/sklearn.tree/33354.major-feature.rst to doc/whats_new/upcoming_changes/sklearn.tree/33972.major-feature.rst? This will register both PR numbers to the entry at the time of changelog consolidation by towncrier.

Done. Let me know if this is what you had in mind @ogrisel

ogrisel · 2026-08-25T07:46:41Z

Thanks @adam2392 for the final push. Merged!

…rt up to 2**24 possible categories (scikit-learn#33972)

adam2392 added 2 commits May 10, 2026 15:15

Nocats

9f060d9

Random splitting with deterministic hashing

a1c31ce

github-actions Bot added the cython label May 11, 2026

adam2392 mentioned this pull request May 22, 2026

FEA Categorical split support for DecisionTree*, and RandomForest* #33354

Merged

Merge main

dd41d6c

github-actions Bot added the CI:Linter failure The linter CI is failing on this PR label Jul 25, 2026

adam2392 added 3 commits July 24, 2026 21:15

Cleanup

d7e61b2

Fixed compilation

88609db

Adding split kind

26424dd

github-actions Bot removed the CI:Linter failure The linter CI is failing on this PR label Jul 25, 2026

adam2392 marked this pull request as ready for review July 25, 2026 12:52

adam2392 added 3 commits July 25, 2026 18:08

Clean up tests

a71b516

Clean up documentation

06640ae

Merge branch 'main' into feat/rand-cat

bc51224

adam2392 added the module:tree label Jul 25, 2026

adam2392 commented Jul 25, 2026

View reviewed changes

Comment thread doc/whats_new/upcoming_changes/sklearn.tree/33354.major-feature.rst

adam2392 mentioned this pull request Jul 25, 2026

Native categorical splitting enhancements in DecisionTree, ExtraTree, RandomForest and ExtraTrees #33965

Open

10 tasks

adam2392 requested review from adrinjalali, lorentzenchr and thomasjpfan July 25, 2026 22:25

cakedev0 reviewed Jul 27, 2026

View reviewed changes

cakedev0 moved this to In progress in Labs Jul 27, 2026

cakedev0 added this to Labs Jul 27, 2026

adam2392 added 2 commits July 27, 2026 10:11

Address feedback

d322c77

Merge branch 'main' into feat/rand-cat

cfead8f

ogrisel moved this from In progress to PR waiting for reviews in Labs Aug 3, 2026

ogrisel reviewed Aug 4, 2026

View reviewed changes

Merge branch 'main' into feat/rand-cat

15115d6

github-actions Bot added the CI:Linter failure The linter CI is failing on this PR label Aug 4, 2026

Address olivier's comments

ca12b2c

github-actions Bot removed the CI:Linter failure The linter CI is failing on this PR label Aug 4, 2026

adam2392 commented Aug 4, 2026

View reviewed changes

Comment thread sklearn/tree/tests/test_split.py

Comment thread sklearn/tree/_utils.pxd

cakedev0 mentioned this pull request Aug 10, 2026

RFC: Unknown categories should not inherit the meaning of missing values in HGB / trees #34691

Open

Merge branch 'main' into feat/rand-cat

2399bb4

adam2392 requested a review from ogrisel August 14, 2026 14:32

lorentzenchr reviewed Aug 14, 2026

View reviewed changes

ogrisel approved these changes Aug 17, 2026

View reviewed changes

adam2392 added 2 commits August 18, 2026 21:52

Address olivier and christian comments

779e801

Merge branch 'feat/rand-cat' of github.com:adam2392/scikit-learn into…

8a11527

… feat/rand-cat

adam2392 added 2 commits August 24, 2026 21:41

Update to make towncrier happy

3bc59fa

Merge branch 'main' into feat/rand-cat

0ed5645

ogrisel merged commit daece67 into scikit-learn:main Aug 25, 2026
37 checks passed

github-project-automation Bot moved this from PR waiting for reviews to Done in Labs Aug 25, 2026

adam2392 deleted the feat/rand-cat branch August 26, 2026 15:23

betatim mentioned this pull request Aug 28, 2026

Change to scikit-learn tree node type breaks treelite.sklearn.export_model dmlc/treelite#696

Closed

prady0t pushed a commit to prady0t/scikit-learn that referenced this pull request Sep 2, 2026

[ENH] ExtraTree: Add random splitting via deterministic hash to suppo…

3916cb6

…rt up to 2**24 possible categories (scikit-learn#33972)

jeremiedbb mentioned this pull request Sep 8, 2026

Release 1.9.1 #34905

Merged

14 tasks



		def random_categorical_goes_left(seed, x):
		return np.array([_py_mix_uint32(seed ^ int(category)) & 1 for category in x])

Uh oh!

Uh oh!

Conversation

adam2392 commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

AI usage disclosure

Any other comments?

Uh oh!

cakedev0 commented Jul 23, 2026

Uh oh!

adam2392 commented Jul 23, 2026

Uh oh!

cakedev0 commented Jul 24, 2026

Uh oh!

adam2392 commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

cakedev0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cakedev0 commented Jul 27, 2026

Uh oh!

adam2392 commented Jul 29, 2026

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Aug 4, 2026

Uh oh!

adam2392 commented Aug 4, 2026

Uh oh!

adam2392 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cakedev0 commented Aug 4, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adam2392 commented Aug 20, 2026

Uh oh!

ogrisel commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adam2392 commented May 11, 2026 •

edited

Loading

adam2392 commented Jul 25, 2026 •

edited

Loading

ogrisel commented Aug 24, 2026 •

edited

Loading