It would also be nice to add similar tests for non-constant but significantly non-centered random data to check the consistency between the sparse and dense array variants (incremental or not).

Also the marked XFAIL tests of #19527 and the test referenced in #19527 (review) should also be updated to test constant values larger than 1.

Reading the code, I realized that for the first call, incr_mean_variance_axis0 is delegating to _csr_mean_variance_axis0 and _csc_mean_variance_axis0 so those function would need to be fixed first.

Then one would need to actually chunk the constant feature into small batches of say 10 samples at a time and check that calling incr_mean_variance_axis0 repeatedly on such data is also numerically stable.

Note that the dense implementation _incremental_mean_and_var uses np.float64 accumulators even when the input data is np.float32. This is not the case for the sparse implementation and could therefore explain part of the discrepancy for np.float32 data case. But this does not explain the discrepancy observed for np.float64 data.

The main culprit seems to be the lines:

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/sparsefuncs_fast.pyx#L155

and:

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/sparsefuncs_fast.pyx#L259

        variances[i] += (sum_weights[i] - sum_weights_nz[i]) * means[i]**2

A small rounding error in the computation of (sum_weights[i] - sum_weights_nz[i]) is multiplied by a large quantity: the squared mean value of the column.

I am not sure how to fix this. The easiest way would be to materialize a dense representation of the the sparse columns, one at a time to avoid memory explosion and call the numerically stable _incremental_mean_and_var on each such dense column.

A slightly more efficient way would be to iterate explicitly over the zero value elements and multiply the weight by the squared mean before summing it to the partial unnormalized variance accumulator. That would still be significantly more expensive than the current code but probably less than materializing the columns. But I am not so sure because of the non-uniform memory access patterns. Finally the code to manipulate row and column indices w.r.t. the CSR or CSC datastructures would probably be much more complex.

Interesting.

Thoughts:

1. In the weighted case, are we already doing operations with O(n_samples) memory on the sample weights? (or is it possible that the sample weights are memmapped?) If so, maybe dense-column-at-a-time is okay.
2. Your option to "iterate explicitly over the zero value elements and multiply the weight by the square mean" sounds like it might be reasonable in a CSC with sorted indices. Perhaps also a CSR with sorted indices.

In the weighted case, are we already doing operations with O(n_samples) memory on the sample weights? (or is it possible that the sample weights are memmapped?) If so, maybe dense-column-at-a-time is okay.

I agree.

Your option to "iterate explicitly over the zero value elements and multiply the weight by the square mean" sounds like it might be reasonable in a CSC with sorted indices. Perhaps also a CSR with sorted indices.

I agree also, but the code is going to be complex, I think. Not sure it's worth it.

Also for the non-weighted case, sum_weights and sum_weights_nz would be integers:

    variances[i] += (sum_weights[i] - sum_weights_nz[i]) * means[i]**2

so if we specialize this function for the non-weighted cases to use uint64 for instance, then we do not have catastrophic cancellation anymore and the code would still be fast and memory efficient.