nan_policy effects all non-finite values if set to omit #9558
Description
In scipy.stats the use of nan_policy effects all invalid values rather than only nan values if nan_policy='omit'. From the sem docstring:
Defines how to handle when the input contains nan.
'omit' performs the calculations ignoring nan
This occurs because _nan_policy checks for nan values only, however, if a nan is found it is handled by using ma.masked_invalid where invalid values are defined as: Not a Number, positive infinity and negative infinity. For example in scipy.stats.sem:
Lines 2211 to 2215 in 517fb55
Reproducing code example:
>>> import numpy as np
>>> from scipy.stats import sem
>>> sem([1, 2, 3, 4, 5])
0.7071067811865476
>>>sem([1, 2, 3, 4, 5, np.nan], nan_policy='omit')
0.7071067811865476 # Omits nans as expected returning the same value.
>>> sem([1, 2, 3, 4, 5, np.inf], nan_policy='omit')
/home/kai/envs/scipy-dev/lib/python3.6/site-packages/numpy/core/_methods.py:117:
RuntimeWarning: invalid value encountered in subtract x = asanyarray(arr - arrmean)
nan # Warns user and returns nan, as expected.
>>> sem([1, 2, 3, 4, 5, np.inf, np.nan], nan_policy='omit')
0.7071067811865476 # nan values AND inf values are omitted.
Scipy/Numpy/Python version information:
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.2.0.dev0+5c01c69 1.15.4 sys.version_info(major=3, minor=6, micro=5, releaselevel='final', serial=0)
List of occurrences in scipy.stats
- mode
- tmin
- tmax
- moment
- variation
- skew
- kurtosis
- kurtosistest
- normaltest
- describe
- skewtest
- Spearmanr
- kendalltau
- ttest_1samp
- ttest_ind
- kruskal
- ttest_rel
- brunnermunzel