Why would _averaged_weighted_percentile not work when len(distinct_values) <= max_bins?

>>> import numpy as np
>>> from sklearn.utils.stats import _weighted_percentile, _averaged_weighted_percentile
>>> a = np.random.randint(0, 10, size=10000).astype(np.float64)
>>> max_bins = 30
>>> max_bins > np.unique(a).shape[0]
True
>>> np.array(
...     [_averaged_weighted_percentile(a, np.ones_like(a), percentile) for percentile in percentiles]
... )
array([0., 0., 0., 1., 1., 1., 1., 2., 2., 2., 3., 3., 3., 4., 4., 4., 5.,
       5., 5., 6., 6., 6., 7., 7., 7., 8., 8., 8., 9., 9., 9.])
>>> np.array(
...     [_weighted_percentile(a, np.ones_like(a), percentile) for percentile in percentiles]
... )
array([0., 0., 0., 1., 1., 1., 1., 2., 2., 2., 3., 3., 3., 4., 4., 4., 5.,
       5., 5., 6., 6., 6., 7., 7., 7., 8., 8., 8., 9., 9., 9.])

When distinct values are less than max_bins and sample weights are given we get a discrepancy between weighted and repeated under special cases (see test_zero_sample_weights_classification under test_gradient_boosting.py). Need to see what to do here.

The manual midpoints between distinct values in the weighted case is probably responsible for this discrepancy.

We should probably explore if it's possible to post process the output of _averaged_weighted_percentile to trim duplicated thresholds. If the number of trimmed thresholds is lower than max_bins then it might be possible to further post-process to recover thresholds that match the unweighted case by shifting the thresholds to use 0.5 midpoints.

Could you expand this test (maybe with @pytest.mark.parametrize) to include a case where n_bins > np.unique(X[:, some_arbitrary_feature_idx]).shape[0]?

It's expected to fail but we need to solve this case anyway.

Yes it fails but good to have this test in

Please do not comment out this case since we need to find a way to make this test pass.

Uncommented, we get the fail now, it may be worth also trying a strategy e.g.,

midpoints = np.array([_averaged_weighted_percentile ...

if np.unique(midpoints).shape[0]!=midpoints.shape[0]:
     midpoints, counts = np.unique(midpoints, return_counts=True)

and do some calculations based off that during trimming... need to think a bit more about it

UPDATE: test passes now after some fixes

I think this variable should be renamed bin_thresholds since they are no midpoints when n_unique > max_bins.

Bin thresholds are set to mid_points when there are fewer observed unique feature values than number of bins.

nitpick: the convention is to use TODO which is easier to grep consistently and often syntax highlighted by IDEs.

Same here, those are no longer midpoints since we use method="average_inverted_cdf".

We should avoid calling np.unique(midpoints) repeatedly. At this stage, the input array has max_bins values so it's not that costly, but still, let store this is a local variable.

Actually, this cannot correctly match the midpoint-based bin threshold of the sample_weight=None side of the condition.

Maybe we should just abandon the idea of trying to set midpoint-based thresholds for len(distinct_values) < max_bins case.

Instead, we should just call np.percentile(..., method="averaged_inverted_cdf") for the unweighted case and _averaged_weighted_percentile() for the weighted case and trim the redundant thresholds a posteriori.

The code will be simpler and the equivalence guaranteed by construction in all the cases.

Currently I have midpoints in within the PR. I have tested locally without the if len(distinct_values) <= len(midpoints), the following tests fail:

• test_bin_mapper_repeated_values_invariance (we seem to get one more bin value than expected so the shapes don't match)
• test_find_binning_thresholds_small_regular_data: the last test which expects midpoints is now failing (we could instead just change the expected output here
• test_binmapper_weighted_vs_repeated_equivalence[42-None]: we get an inhomogeneous array since each column returns a different number of unique percentile values
• test_bin_mapper_identity_small: need to dig into this one more would not expect it to fail but it may be due to np.percentile calculation
  amongst others. I think for some if we change the expected result it may be fine, but not sure what to do when non-homogeneous arrays are returned, let me know your thoughts.