Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: np.percentile fails with internal overflow when using float16 input on large arrays #29003

Open
@friedlerido

Description

@friedlerido

Describe the issue:

When using np.percentile or np.nanpercentile on a large float16 array from real data (≈9 million elements), I get:
ValueError: kth(=-9223372036845518721) out of bounds (9257087)

This happens even for low percentiles like 20 or 50, and is resolved by casting the array to float64.

⚠️ Expected Behavior:
Either: NumPy automatically upcasts float16 to float64
Or: raises a UserWarning if float16 is passed to percentile

Reproduce the code example:

# This does not happen with random arrays of similar shape, but does happen consistently  #with my real-world data:
arr = np.load("example.npy")  # I can share this privately if needed
print(arr.dtype)  # float16
print(arr.shape)  # (9257087,)

np.percentile(arr, 50)  # → raises ValueError

# ✅ This works:
np.percentile(arr.astype(np.float64), 50)  # works fine

Error message:

/opt/venv/lib/python3.11/site-packages/numpy/lib/_function_base_impl.py:107: RuntimeWarning: overflow encountered in cast
  get_virtual_index=lambda n, quantiles: (n - 1) * quantiles,
/opt/venv/lib/python3.11/site-packages/numpy/lib/_function_base_impl.py:4750: RuntimeWarning: overflow encountered in cast
  indexes_above_bounds = virtual_indexes >= valid_values_count - 1
/opt/venv/lib/python3.11/site-packages/numpy/lib/_function_base_impl.py:4655: RuntimeWarning: invalid value encountered in multiply
  lerp_interpolation = asanyarray(add(a, diff_b_a * t, out=out))
/opt/venv/lib/python3.11/site-packages/numpy/lib/_function_base_impl.py:4656: RuntimeWarning: invalid value encountered in scalar multiply
  subtract(b, diff_b_a * (1 - t), out=lerp_interpolation, where=t >= 0.5,
np.float16(nan)

Python and NumPy Versions:

2.2.4
3.11.3 (main, May 23 2023, 13:34:03) [GCC 10.2.1 20210110]

Runtime Environment:

[{'numpy_version': '2.2.4',
'python': '3.11.3 (main, May 23 2023, 13:34:03) [GCC 10.2.1 20210110]',
'uname': uname_result(system='Linux', , release='5.15.0-1039-aws', version='#44~20.04.1-Ubuntu SMP Thu Jun 22 12:21:12 UTC 2023', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/opt/venv/lib/python3.11/site-packages/numpy.libs/libscipy_openblas64_-6bb31eeb.so',
'internal_api': 'openblas',
'num_threads': 64,
'prefix': 'libscipy_openblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.28'}]

Context for the issue:

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions