@friedlerido Could you share the example.npy? If possible try to reduce the example data to a minimal case that can be generated by plain python. That will help us to investigate.

example.zip

For the example above we end up with values_count=n equal to the shape of the input array and quantiles equal to np.array([50], dtype=arr.dtype) inside _quantile. There a call is made to the get_virtual_index of the linear method which is defined as

get_virtual_index=lambda n, quantiles: (n - 1) * quantiles,

and the overflow occurs.

The issue might have been introduced in #23912 @seberg. There the dtype of the input array is set on the quantiles, e.g.

Here is a small reproducer:

import numpy as np
arr = np.zeros(65521, dtype=np.float16)
arr[:10] = 1
z = np.percentile(arr, 50)
print(z)

Likely NEP 50 itself that changed the dtype, not so much the specific code change? I am not sure immediately if there are some internal calculations that should maybe always use float64 at least (because they are related to the length of an array, so need full float64 mantissa to be pretty correct).

To some degree, float16 tends to overflow, but this may also not be ideal for float32. The other thing is that a lerp might do a better job (e.g. by being a ufunc), but then I am not sure there is a hot-fix.

This indeed seems tricky to get right: the output of get_virtual_index=lambda n, quantiles: (n - 1) * quantiles is an array with virtual indices virtual_indexes that is used in two ways (roughly):

• floor(virtual_indexes) and ceil(virtual_indices) are indices of the two data points (in the sorted data) used for interpolation
• g= virtual_indexes % 1 is used as a factor to interpolate between the two data points

We can use np.interp (or some other way) to calculate the value of (n - 1) * quantiles without overflows, but the result has to be in float64 (at least higher than float16 which is not enough). Then also g will be float64 and the final result of np.quantile is (1-g)*y[j] + g*y[j+1] which will also be float64. This is not ideal since we would like to have the output of np.quantile(float16_array, [0, .5, .99]) to be float16. On the other hand, np.quantile([0, 1, 3], [0]) is also promoted to float64.

We could downcast g to the dtype of the input array. (since g is in the 0 to 1 range that seems fine), but that might break cases with integer input.

Several other methods also give incorrect results:

import numpy as np
arr = np.zeros(65521, dtype=np.float16)
arr[:10] = 1
methods = ['inverted_cdf','averaged_inverted_cdf','closest_observation','interpolated_inverted_cdf','hazen','weibull',
           'linear', 'median_unbiased','normal_unbiased']

for method in methods:
    z = np.percentile(arr, 50, method=method)
    print(f'{method=} output={z} {z.dtype=}')

Output:

method='inverted_cdf' output=0.0 z.dtype=dtype('float16')
method='averaged_inverted_cdf' output=1.0 z.dtype=dtype('float16')
method='closest_observation' output=0.0 z.dtype=dtype('float16')
method='interpolated_inverted_cdf' output=nan z.dtype=dtype('float16')
method='hazen' output=nan z.dtype=dtype('float16')
method='weibull' output=nan z.dtype=dtype('float16')
method='linear' output=nan z.dtype=dtype('float16')
method='median_unbiased' output=nan z.dtype=dtype('float16')
method='normal_unbiased' output=nan z.dtype=dtype('float16')