You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Version of glibc (cause of the slowdown): conda-forge compiles scikit-learn against a sysroot_linux-64=2.17 (glibc 2.17, from 2012-12-25). That old glibc's <math.h> defines the isnan() macro as a dispatch to glibc-internal __isnanf/__isnan functions instead of __builtin_isnan*. GCC can't recognize or inline those, so sklearn/tree/_partitioner.pyx's isnan(value) calls — sitting in the hottest loop of tree-building (find_min_max, partition_samples) — become genuine non-inlined function calls. This is fit-specific (confirmed: predict, which doesn't call these functions, shows no slowdown) and fully reproduced from source once the exact sysroot was pinned.
This is very likely not scikit-learn-specific: any conda-forge package with a hot-loop isnan()/isinf() call is plausibly exposed the same way.
Replace from libc.math cimport isnan / isnan(value) in sklearn/tree/_partitioner.pyx and sklearn/tree/_utils.pxd (and any other tree/HGB hot loop doing the same) with the self-inequality NaN test value != value. Pure comparison, nothing to inline, immune to this class of bug on any compiler/libc, forever.
Root cause, in detail
(pure LLM write-up)
Details
sklearn/tree/_partitioner.pyx (DensePartitioner.find_min_max, .partition_samples) and sklearn/tree/_utils.pxd call isnan(value) on float32_t data, via from libc.math cimport isnan. This runs once per sample per feature scan during fit; never during predict.
The feedstock's CI pin file (.ci_support/linux_64_python3.12.____cpython.yaml at the 1.9.0 release commit) sets c_stdlib: sysroot, c_stdlib_version: '2.17'. That sysroot's math.h has:
Once the preprocessor expands this, the identifier isnan is gone from
the translation unit — GCC has nothing left to recognize as a builtin, so __isnanf (an ordinary external glibc symbol, not a compiler intrinsic)
gets a real out-of-line call. Modern glibc instead defines isnan(x) as __builtin_isnan(x) directly, which GCC folds to 2 instructions
(ucomiss/setp). PyPI/manylinux wheels don't hit this because they're
built with devtoolset-style toolchains: modern compiler and modern
headers, achieving old-system compatibility via glibc symbol versioning
at link time rather than by literally compiling against old headers.
Confirmed end to end:
perf-profiled real installed PyPI vs conda-forge packages
(ExtraTreesRegressor.fit, n_features=30): conda-forge spends 12.6% of
samples in __isnanf alone; PyPI doesn't show it in the top 15 at all.
Disassembled both real .so files directly: conda-forge's find_min_max has call *0x1791f(%rip) # <__isnanf@GLIBC_2.2.5>;
PyPI's has no call at all.
Reproduced from source: rebuilding the actual v1.9.0 tag with the exact
matching GCC (14.3.0-19, confirmed via the real binary's own .comment section) reproduced nothing untilsysroot_linux-64=2.17
was pinned explicitly (every environment had otherwise resolved the
latest default, 2.39) — at which point the non-inlined call and the
larger function size reappeared byte-for-byte.
Same isnan() call pattern, same functions, confirmed still present on
current main (the categorical-feature/Breiman-shortcut work added
since 1.9.0 didn't touch this code path — find_min_max compiles to the
identical byte size on both).
Everything else that was tried and ruled out
Before finding the sysroot, a long list of variables was controlled for one at a time and did not reproduce the real-world gap when rebuilding from source with the real conda-forge gcc_linux-64/gxx_linux-64 toolchain:
Details
-O2 vs -O3 in isolation (~1.7% effect — this is the Meson bug above)
-march=nocona -mtune=haswell vs no override
GCC version: 16.2.0, 13.4.0, and the exact 14.3.0-19 patch build
(pinning gcc_impl_linux-64/gxx_impl_linux-64/binutils_impl_linux-64
precisely, since gcc_impl_linux-64 has its own build-number sequence
independent of the gcc_linux-64 wrapper)
Cython version (real binary uses 3.2.5; early rebuilds had picked up
3.3.0 by default — controlled for, no effect)
While looking at builds comparison results from my benchmarks (https://github.com/probabl-ai/scikit-learn-benchmarks), I noticed
ExtraTrees*.fitwas often ~25% slower on conda-forge builds ofscikit-learn 1.9.0 than on PyPI wheels.
Two distinct bugs were found along the way; only the second one explains the slowdown:
O3 vs O2 (documented in Investigation: Meson silently downgrades -O3 to -O2 under conda-forge CFLAGS #34865):
-O3in meson doesn't survive an environmentCFLAGSthat also sets-O2(which conda-forge's compiler activation does).Version of
glibc(cause of the slowdown): conda-forge compiles scikit-learn against asysroot_linux-64=2.17(glibc 2.17, from 2012-12-25). That old glibc's<math.h>defines theisnan()macro as a dispatch to glibc-internal__isnanf/__isnanfunctions instead of__builtin_isnan*. GCC can't recognize or inline those, sosklearn/tree/_partitioner.pyx'sisnan(value)calls — sitting in the hottest loop of tree-building (find_min_max,partition_samples) — become genuine non-inlined function calls. This isfit-specific (confirmed:predict, which doesn't call these functions, shows no slowdown) and fully reproduced from source once the exact sysroot was pinned.This is very likely not scikit-learn-specific: any conda-forge package with a hot-loop
isnan()/isinf()call is plausibly exposed the same way.=> TODO: Open an issue in conda-forgeDone: I posted a comment here: conda-forge/ctng-compiler-activation-feedstock#108Proposed fix
Replace
from libc.math cimport isnan/isnan(value)insklearn/tree/_partitioner.pyxandsklearn/tree/_utils.pxd(and any other tree/HGB hot loop doing the same) with the self-inequality NaN testvalue != value. Pure comparison, nothing to inline, immune to this class of bug on any compiler/libc, forever.Root cause, in detail
(pure LLM write-up)
Details
sklearn/tree/_partitioner.pyx(DensePartitioner.find_min_max,.partition_samples) andsklearn/tree/_utils.pxdcallisnan(value)onfloat32_tdata, viafrom libc.math cimport isnan. This runs once per sample per feature scan duringfit; never duringpredict.The feedstock's CI pin file (
.ci_support/linux_64_python3.12.____cpython.yamlat the 1.9.0 release commit) setsc_stdlib: sysroot,c_stdlib_version: '2.17'. That sysroot'smath.hhas:Once the preprocessor expands this, the identifier
isnanis gone fromthe translation unit — GCC has nothing left to recognize as a builtin, so
__isnanf(an ordinary external glibc symbol, not a compiler intrinsic)gets a real out-of-line call. Modern glibc instead defines
isnan(x)as__builtin_isnan(x)directly, which GCC folds to 2 instructions(
ucomiss/setp). PyPI/manylinux wheels don't hit this because they'rebuilt with devtoolset-style toolchains: modern compiler and modern
headers, achieving old-system compatibility via glibc symbol versioning
at link time rather than by literally compiling against old headers.
Confirmed end to end:
perf-profiled real installed PyPI vs conda-forge packages(
ExtraTreesRegressor.fit,n_features=30): conda-forge spends 12.6% ofsamples in
__isnanfalone; PyPI doesn't show it in the top 15 at all..sofiles directly: conda-forge'sfind_min_maxhascall *0x1791f(%rip) # <__isnanf@GLIBC_2.2.5>;PyPI's has no call at all.
matching GCC (14.3.0-19, confirmed via the real binary's own
.commentsection) reproduced nothing untilsysroot_linux-64=2.17was pinned explicitly (every environment had otherwise resolved the
latest default, 2.39) — at which point the non-inlined call and the
larger function size reappeared byte-for-byte.
isnan()call pattern, same functions, confirmed still present oncurrent
main(the categorical-feature/Breiman-shortcut work addedsince 1.9.0 didn't touch this code path —
find_min_maxcompiles to theidentical byte size on both).
Everything else that was tried and ruled out
Before finding the sysroot, a long list of variables was controlled for one at a time and did not reproduce the real-world gap when rebuilding from source with the real conda-forge
gcc_linux-64/gxx_linux-64toolchain:Details
-O2vs-O3in isolation (~1.7% effect — this is the Meson bug above)-march=nocona -mtune=haswellvs no override14.3.0-19patch build(pinning
gcc_impl_linux-64/gxx_impl_linux-64/binutils_impl_linux-64precisely, since
gcc_impl_linux-64has its own build-number sequenceindependent of the
gcc_linux-64wrapper)3.3.0 by default — controlled for, no effect)
mainpython -m build -wwheel build (matchingbuild.shexactly)-fno-merge-constants(the extra flag conda-forge adds underCONDA_BUILD=1)identical between real PyPI and conda-forge installs
Interest in fixing the bug
Yes, I'll open the PR very soon.