FIX stabilize weighted confusion_matrix_at_thresholds on float32-only devices - #34827

cc @david-cortes-intel as we discussed this during a meeting.

In my tests your adaptive scale idea works great in all but one easily rectifiable case.

The case goes as w = 1e-8 on the top 10% by score, 1.0 on the rest, n = 1e7:

  scale=1.11e+06  round(1e-8*scale)=0
  tps[500000]  exact 0.0045005   #34827v2 0

tps would be wrong for the starting ones.
This can be fixed by consuming more of 64 bits you got by something in the line of

scale = max(int(0.9 * (2**63 - 1) / (n_samples * weight_max)), 1)

Generally summing in descending order is numerically problematic, the idea to use int64 is quiet good, the classic way of dealing with it is using compensated summing[1] that I tested a vectorized version[2] of it head to head to this version and the current pr is better.

def compensated_cumsum(x):
    c = np.cumsum(x, dtype=f32)
    a = np.concatenate([np.zeros(1, dtype=f32), c[:-1]])
    s = a + x
    bb = s - a
    e = (a - (s - bb)) + (x - bb)
    d = (s - c) + e
    return c + np.cumsum(d, dtype=f32)

[1]: Higham, Nicholas J. 4.6 In Accuracy and Stability of Numerical Algorithms, 2nd ed.
[2]: Knuth D. ch 4.2.2 The Art of Computer Programming, Vol. 2

@Fazel94 this PR has been update with what you suggested. Let me know if you have any further feedback.

In my tests your adaptive scale idea works great in all but one easily rectifiable case.

The case goes as w = 1e-8 on the top 10% by score, 1.0 on the rest, n = 1e7:

  scale=1.11e+06  round(1e-8*scale)=0
  tps[500000]  exact 0.0045005   #34827v2 0

tps would be wrong for the starting ones. This can be fixed by consuming more of 64 bits you got by something in the line of

scale = max(int(0.9 * (2**63 - 1) / (n_samples * weight_max)), 1)

Generally summing in descending order is numerically problematic, the idea to use int64 is quiet good, the classic way of dealing with it is using compensated summing[1] that I tested a vectorized version[2] of it head to head to this version and the current pr is better.

def compensated_cumsum(x):
    c = np.cumsum(x, dtype=f32)
    a = np.concatenate([np.zeros(1, dtype=f32), c[:-1]])
    s = a + x
    bb = s - a
    e = (a - (s - bb)) + (x - bb)
    d = (s - c) + e
    return c + np.cumsum(d, dtype=f32)

[1]: Higham, Nicholas J. 4.6 In Accuracy and Stability of Numerical Algorithms, 2nd ed. [2]: Knuth D. ch 4.2.2 The Art of Computer Programming, Vol. 2

Wouldn't that also fail when the compensation terms exceed the mantissa? For example, cumsum of 40 million ones.

@david-cortes-intel It would break on 40M, nice catch.
For purely theoretical interest,
We have a self-similar problem of calculating compensation term, we can apply a round of Two Sum on the compensation term, essentially instead of np.cumsum(d, dtype=f32) we can recurse and alleviate the problem to some extend, but on arbitrary distributed inputs the needed recursion depth can vary and a fix depth can not be accurate on all inputs.
I suspect using Kahan method could help better but that would require an if branch on each term.
But again as Higham puts as his first solution to summation numerical stability, "Just use more bits"(4.6.1) is the correct approach that @ogrisel implemented here.

I have the feeling that this PR is a net improvement over main and from my shallow understanding of the discussion it does not seem to be easy to do better on float32-only devices.

However, if volunteers would like to explore alternatives, feel free to open a concurrent PR and we can evaluate robustness/code complexity/speed tradeoffs.

There is another alternative which would be to perform the weighted counts on CPU if the the device is float32 only.

There is another alternative which would be to perform the weighted counts on CPU if the the device is float32 only.

I like that alternative better. Not sure how it fits into the whole array API design though.

There is another alternative which would be to perform the weighted counts on CPU if the the device is float32 only.

I like that alternative better. Not sure how it fits into the whole array API design though.

I don't see any other way than checking to see if the device doesn't support float64, then pulling the data back and using numpy. Since numpy code here is efficient it don't see a problem with it. Saying that it doesn't feel natural.

If there is cases of algorithms facing similar kind of problems it would make sense to create some sort of fixed point helper functions to use in such cases.

If there is cases of algorithms facing similar kind of problems it would make sense to create some sort of fixed point helper functions to use in such cases.

We have this note in our array API support document:

https://scikit-learn.org/dev/modules/array_api.html#note-on-device-support-for-float64

This note is typically linked to from estimators & function listed in this list:

https://scikit-learn.org/dev/modules/array_api.html#support-for-array-api-compatible-inputs

For metric functions that are often fast to compute, I think moving to CPU if needed is fine. For estimators where fit can often be the computational bottlneck of a user's workfload, the tradeoff is harder to break.