Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Investigation: conda-forge scikit-learn slower than PyPI (ExtraTrees case) #34869

Description

@cakedev0

While looking at builds comparison results from my benchmarks (https://github.com/probabl-ai/scikit-learn-benchmarks), I noticed ExtraTrees*.fit was often ~25% slower on conda-forge builds of
scikit-learn 1.9.0 than on PyPI wheels.

Two distinct bugs were found along the way; only the second one explains the slowdown:

  1. O3 vs O2 (documented in Investigation: Meson silently downgrades -O3 to -O2 under conda-forge CFLAGS #34865): -O3 in meson doesn't survive an environment CFLAGS that also sets -O2 (which conda-forge's compiler activation does).

  2. Version of glibc (cause of the slowdown): conda-forge compiles scikit-learn against a sysroot_linux-64=2.17 (glibc 2.17, from 2012-12-25). That old glibc's <math.h> defines the isnan() macro as a dispatch to glibc-internal __isnanf/__isnan functions instead of __builtin_isnan*. GCC can't recognize or inline those, so sklearn/tree/_partitioner.pyx's isnan(value) calls — sitting in the hottest loop of tree-building (find_min_max, partition_samples) — become genuine non-inlined function calls. This is fit-specific (confirmed: predict, which doesn't call these functions, shows no slowdown) and fully reproduced from source once the exact sysroot was pinned.

This is very likely not scikit-learn-specific: any conda-forge package with a hot-loop isnan()/isinf() call is plausibly exposed the same way.

=> TODO: Open an issue in conda-forge Done: I posted a comment here: conda-forge/ctng-compiler-activation-feedstock#108

Proposed fix

Replace from libc.math cimport isnan / isnan(value) in sklearn/tree/_partitioner.pyx and sklearn/tree/_utils.pxd (and any other tree/HGB hot loop doing the same) with the self-inequality NaN test value != value. Pure comparison, nothing to inline, immune to this class of bug on any compiler/libc, forever.

Root cause, in detail

(pure LLM write-up)

Details

sklearn/tree/_partitioner.pyx (DensePartitioner.find_min_max, .partition_samples) and sklearn/tree/_utils.pxd call isnan(value) on float32_t data, via from libc.math cimport isnan. This runs once per sample per feature scan during fit; never during predict.

The feedstock's CI pin file (.ci_support/linux_64_python3.12.____cpython.yaml at the 1.9.0 release commit) sets c_stdlib: sysroot, c_stdlib_version: '2.17'. That sysroot's math.h has:

#  define isnan(x) \
     (sizeof (x) == sizeof (float) ? __isnanf (x) \
      : sizeof (x) == sizeof (double) ? __isnan (x) : __isnanl (x))

Once the preprocessor expands this, the identifier isnan is gone from
the translation unit — GCC has nothing left to recognize as a builtin, so
__isnanf (an ordinary external glibc symbol, not a compiler intrinsic)
gets a real out-of-line call. Modern glibc instead defines isnan(x) as
__builtin_isnan(x) directly, which GCC folds to 2 instructions
(ucomiss/setp). PyPI/manylinux wheels don't hit this because they're
built with devtoolset-style toolchains: modern compiler and modern
headers, achieving old-system compatibility via glibc symbol versioning
at link time rather than by literally compiling against old headers.

Confirmed end to end:

  • perf-profiled real installed PyPI vs conda-forge packages
    (ExtraTreesRegressor.fit, n_features=30): conda-forge spends 12.6% of
    samples in __isnanf alone; PyPI doesn't show it in the top 15 at all.
  • Disassembled both real .so files directly: conda-forge's
    find_min_max has call *0x1791f(%rip) # <__isnanf@GLIBC_2.2.5>;
    PyPI's has no call at all.
  • Reproduced from source: rebuilding the actual v1.9.0 tag with the exact
    matching GCC (14.3.0-19, confirmed via the real binary's own
    .comment section) reproduced nothing until sysroot_linux-64=2.17
    was pinned explicitly (every environment had otherwise resolved the
    latest default, 2.39) — at which point the non-inlined call and the
    larger function size reappeared byte-for-byte.
  • Same isnan() call pattern, same functions, confirmed still present on
    current main (the categorical-feature/Breiman-shortcut work added
    since 1.9.0 didn't touch this code path — find_min_max compiles to the
    identical byte size on both).

Everything else that was tried and ruled out

Before finding the sysroot, a long list of variables was controlled for one at a time and did not reproduce the real-world gap when rebuilding from source with the real conda-forge gcc_linux-64/gxx_linux-64 toolchain:

Details
  • -O2 vs -O3 in isolation (~1.7% effect — this is the Meson bug above)
  • -march=nocona -mtune=haswell vs no override
  • GCC version: 16.2.0, 13.4.0, and the exact 14.3.0-19 patch build
    (pinning gcc_impl_linux-64/gxx_impl_linux-64/binutils_impl_linux-64
    precisely, since gcc_impl_linux-64 has its own build-number sequence
    independent of the gcc_linux-64 wrapper)
  • Cython version (real binary uses 3.2.5; early rebuilds had picked up
    3.3.0 by default — controlled for, no effect)
  • Exact v1.9.0 source tag vs current main
  • Editable install vs literal python -m build -w wheel build (matching
    build.sh exactly)
  • -fno-merge-constants (the extra flag conda-forge adds under
    CONDA_BUILD=1)
  • joblib version, core-count detection, BLAS threading config — all
    identical between real PyPI and conda-forge installs

Interest in fixing the bug

Yes, I'll open the PR very soon.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions