ENH add sparse output to SplineTransformer - #24145

There is a bunch of sparse matrices being constructed in this implementation. For each feature:

1. A CSR design matrix is constructed
2. This matrix can be converted into a lil matrix
3. hstack converts it all back to CSR.

What is the runtime for transform with sparse output compared to dense output?

import numpy as np
from sklearn.preprocessing import SplineTransformer

X = np.linspace([-1, -10, 100], [1, 10, 100], 10000)
st_sparse = SplineTransformer(sparse_output=True, extrapolation="error").fit(X)
st_dense = SplineTransformer(extrapolation="error").fit(X)

%timeit st_dense.transform(X)
2.13 ms ± 20.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit st_sparse.transform(X)
43.7 ms ± 336 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

That's unfortunate. Having n_features=1 doesn't change the result.
But memory consumption should be better.

The bottleneck seems to be in scipy's _bspl.pyx function _make_design_matrix. In contrast, evaluate_spline does all loops explicitly. We might report this upstream.

@ev-br @egorz734 friendly ping

Not sure what's going on here TBH, but csr->lil->csr does sound expensive. Where is the bottleneck, what does the profiler say?

What's the impact of scipy/scipy#16840 on the scikit-learn benchmark in https://github.com/scikit-learn/scikit-learn/pull/24145/files#r940357974?

Are there scipy nightly builds to install? Otherwise, this is above my current time budget to benchmark.
Scipy 1.10 will not only speed up BSpline.design_matrix but also make it easier to implement spline extrapolation in scikit-learn, thereby further reducing runtime and memory consumption.

SciPy has nightly builds:

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy

but their CI is failing and not updating the nightly builds. I opened MacPython/scipy-wheels#175 to track the SciPy issue.

With scipy==1.10.0.dev0, one gets improved runtime:

import numpy as np
from sklearn.preprocessing import SplineTransformer

X = np.linspace([-1, -10, 100], [1, 10, 100], 10000)

st_sparse = SplineTransformer(sparse_output=True, extrapolation="error").fit(X)
st_dense = SplineTransformer(extrapolation="error").fit(X)

%timeit st_dense.transform(X)
1.89 ms ± 10.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit st_sparse.transform(X)
4.89 ms ± 53.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

On my laptop, I get (scipy 1.10.1)

%timeit st_dense.transform(X)
2.12 ms ± 46.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit st_sparse.transform(X)
4.61 ms ± 162 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

This comment seems to imply that with scipy 1.10, the .tolil conversion would no longer be necessary.

However, as far as I understand, we still have to go through this condition when extrapolate="periodic", even with scipy 1.10 or later:

• https://github.com/scikit-learn/scikit-learn/pull/24145/files#diff-690d1b546a74ed69a4dc3e77edaa88363fc0f8297a71b1a3619a0f1cf79b2621R899-R906

Would there be a way to avoid the .tolil conversion completely with recent scipy versions?

At the moment, sparse periodic extrapolation is more than 20x slower than its dense counterpart:

In [1]: import numpy as np
   ...: from sklearn.preprocessing import SplineTransformer
   ...: 
   ...: X = np.linspace([-1, -10, 100], [1, 10, 101], 10000)
   ...: extrapolation="periodic"
   ...: st_sparse = SplineTransformer(sparse_output=True, extrapolation=extrapolation).fit(X)
   ...: st_dense = SplineTransformer(extrapolation=extrapolation).fit(X)

In [2]: %timeit st_dense.transform(X)
1.29 ms ± 1.25 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [3]: %timeit st_sparse.transform(X)
64.1 ms ± 851 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

scipy version: 1.10.1

It is slower, yes, but this is not a performance regression as sparse output is a new feature and the default is dense output.

And yes, with scipy >= 1.10, we can get rid of (at least most of) the lil-conversions as we can use design_matrix(..., extrapolate=True).