BUG: SciPy test failures due to changes in NumPy main or Python 3.11 #22006

mdhaber · 2022-07-17T21:50:52Z

Describe the issue:

Some SciPy tests involving masked arrays have been failing since ~7/15 (e.g. see CI in scipy/scipy#16610). It looks like some of the failures are resolved by gh-21977, but there are some that remain.

=================================== FAILURES ===================================
______________________ TestCorr.test_kendalltau_seasonal _______________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:405: in test_kendalltau_seasonal
    assert_almost_equal(output['global p-value (indep)'], 0.008, 3)
        output     = {'chi2 total': 0.9138890055275067, 'chi2 trend': 0.03518368100765739, 'global p-value (dep)': 0.9592698679548645, 'global p-value (indep)': 0.9415878180349753, ...}
        self       = <scipy.stats.tests.test_mstats_basic.TestCorr object at 0x7f75a3bf2650>
        x          = masked_array(
  data=[[--, 4.0, 3.0, --],
        [--, 3.0, 2.0, 6.0],
        [4.0, 5.0, 5.0, 11.0],
        [2.0, 3.... False, False, False],
        [False, False, False, False],
        [False, False,  True, False]],
  fill_value=1e+20)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:189: in assert_almost_equal
    raise AssertionError(msg)
E   AssertionError: 
E   Items are not equal:
E    ACTUAL: 0.9415878180349753
E    DESIRED: 0.008
        actual     = 0.9415878180349753
        decimal    = 3
        desired    = 0.008
        err_msg    = ''
        msg        = '\nItems are not equal:\n ACTUAL: 0.9415878180349753\n DESIRED: 0.008'
        verbose    = True
_______________________ TestTtest_rel.test_fully_masked ________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1217: in test_fully_masked
    assert_array_equal(p, (np.nan, np.nan))
        outcome    = masked_array(
  data=[[--, -0.4495544480666782],
        [--, 0.6433801600172545],
        [--, 0.08038096427837299]],
  mask=[[ True, False],
        [ True, False],
        [ True, False]],
  fill_value=1e+20)
        p          = 1.0
        pair       = ([nan, nan], [1.0, 2.0])
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_rel object at 0x7f75a2eedb50>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
_______________________ TestTtest_ind.test_fully_masked ________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1316: in test_fully_masked
    assert_array_equal(p, (np.nan, np.nan))
        outcome    = masked_array(
  data=[[--, -0.4495544480666782],
        [--, 0.6433801600172545],
        [--, 0.08038096427837299]],
  mask=[[ True, False],
        [ True, False],
        [ True, False]],
  fill_value=1e+20)
        p          = 1.0
        pair       = (masked_array(data=[--, --, --],
             mask=[ True,  True,  True],
       fill_value=1e+20,
            dtype=f...1600172545,
                   0.08038096427837299],
             mask=[False, False, False],
       fill_value=1e+20))
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_ind object at 0x7f75a3837150>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
______________________ TestTtest_1samp.test_fully_masked _______________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py:1403: in test_fully_masked
    assert_array_equal(p, expected)
        expected   = (nan, nan)
        outcome    = masked_array(data=[--, --, --],
             mask=[ True,  True,  True],
       fill_value=1e+20,
            dtype=float64)
        p          = 1.0
        pair       = ((nan, nan), 0.0)
        self       = <scipy.stats.tests.test_mstats_basic.TestTtest_1samp object at 0x7f75a35bfa50>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7f75996c5150>
        t          = masked
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:225: in assert_array_equal
    assert_array_compare(operator.__eq__, x, y,
        err_msg    = ''
        verbose    = True
        x          = 1.0
        y          = (nan, nan)
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <built-in function eq>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not equal'
        m          = False
        verbose    = True
        x          = masked_array(data=1.,
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not equal
E   
E   x and y nan location mismatch:
E    x: array(1.)
E    y: array([nan, nan])
        args       = (<built-in function eq>, array(1.), array([nan, nan]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
_________________________ test_chisquare_masked_arrays _________________________
../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_stats.py:3579: in test_chisquare_masked_arrays
    mat.assert_array_almost_equal(g, expected_g, decimal=15)
        chi2       = <scipy.stats._continuous_distns.chi2_gen object at 0x7f75ad58ae50>
        chisq      = masked_array(data=[24.0, 0.5],
             mask=[False, False],
       fill_value=1e+20)
        expected_chisq = array([24. ,  0.5])
        expected_g = array([22.18070978,  0.50534308])
        g          = array([nan, nan])
        mask       = array([[0, 1],
       [0, 1],
       [0, 0],
       [0, 0],
       [1, 0]])
        mobs       = masked_array(
  data=[[8, --],
        [8, --],
        [16, 3],
        [32, 4],
        [--, 5]],
  mask=[[False,  T...,
        [False,  True],
        [False, False],
        [False, False],
        [ True, False]],
  fill_value=999999)
        obs        = array([[ 8, -1],
       [ 8, -1],
       [16,  3],
       [32,  4],
       [-1,  5]])
        p          = array([nan, nan])
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:265: in assert_array_almost_equal
    assert_array_compare(compare, x, y, err_msg=err_msg, verbose=verbose,
        compare    = <function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>
        decimal    = 15
        err_msg    = ''
        verbose    = True
        x          = array([nan, nan])
        y          = array([22.18070978,  0.50534308])
/home/runner/.local/lib/python3.11/site-packages/numpy/ma/testutils.py:213: in assert_array_compare
    return np.testing.assert_array_compare(comparison,
        comparison = <function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>
        err_msg    = ''
        fill_value = True
        header     = 'Arrays are not almost equal'
        m          = False
        verbose    = True
        x          = masked_array(data=[nan, nan],
             mask=False,
       fill_value=1e+20)
        y          = masked_array(data=[22.18070978,  0.50534308],
             mask=False,
       fill_value=1e+20)
/opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal
E   
E   x and y nan location mismatch:
E    x: array([nan, nan])
E    y: array([22.18071 ,  0.505343])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7f758bbc84a0>, array([nan, nan]), array([22.18070978,  0.50534308]))
        func       = <function assert_array_compare at 0x7f75b19c16c0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f75b19a91d0>
============================= slowest 10 durations =============================
16.80s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_kappa4_array_gh13582
12.20s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-skewnorm-arg91]
9.72s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_direct.py::TestDIRECT::test_segmentation_fault[False]
7.68s call     build/testenv/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_precond_inverse[case1]
7.33s call     build/testenv/lib/python3.11/site-packages/scipy/_lib/tests/test_import_cycles.py::test_modules_importable
5.79s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_lsq_linear.py::TestTRF::test_large_rank_deficient
5.76s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_lsq_linear.py::TestBVLS::test_large_rank_deficient
4.71s call     build/testenv/lib/python3.11/site-packages/scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-truncweibull_min-arg100]
3.93s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/tests/test_optimize.py::TestOptimizeSimple::test_minimize_callback_copies_array[fmin]
3.68s call     build/testenv/lib/python3.11/site-packages/scipy/optimize/_trustregion_constr/tests/test_report.py::test_gh12922
=========================== short test summary info ============================
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestCorr::test_kendalltau_seasonal
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_rel::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_ind::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_mstats_basic.py::TestTtest_1samp::test_fully_masked
FAILED ../testenv/lib/python3.11/site-packages/scipy/stats/tests/test_stats.py::test_chisquare_masked_arrays
= 5 failed, 36748 passed, 2162 skipped, [1211](https://github.com/scipy/scipy/runs/7379137176?check_suite_focus=true#step:7:1212)4 deselected, 139 xfailed, 7 xpassed in 538.49s (0:08:58) =

I am having trouble creating a MWE involving only NumPy because I haven't been successful at building NumPy main. I hope someone with (any relatively modern version of) SciPy and NumPy main can help me find one.

This test below passes in NumPy 1.21.2, but I believe it will fail with NumPy main. Can someone confirm and tell me what the value of res is in NumPy main?

Reproduce the code example:

import numpy as np
from numpy import ma
from scipy.stats import mstats
outcome = ma.masked_array(np.random.randn(3, 2), mask=[[1, 0], [1, 0], [1, 0]])
res = mstats.ttest_rel(outcome[:, 0], outcome[:, 1])
# I don't think it is good practice to compare a masked element to a NaN,
# but this is what was in the SciPy test.
np.testing.assert_array_equal(res, (np.nan, np.nan))
# In any case, if this no longer passes, why not? 
# What is the value of `res` using NumPy main?
print(res)

Error message:

NumPy/Python version information:

1.21.2 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]

The text was updated successfully, but these errors were encountered:

charris · 2022-07-18T00:36:25Z

I haven't been successful at building NumPy main

Wheels are built from the main branch on a weekly basis and are available at https://anaconda.org/scipy-wheels-nightly/numpy/files

mdhaber · 2022-07-18T01:35:46Z

Thanks @charris. I installed from the latest wheel, but it looks like it doesn't have the breaking change. Looks like I'll need to wait for a few days. In the meantime, if someone would be willing to run the snippet and report the result, I'd appreciate it.

mattip · 2022-07-18T04:01:59Z

#21993 was meant to fix an issue that cropped up after ~~#21993~~ #21977. I manually triggered wheel builds to produce wheels with that change. They should show up on the link @charris mentioned above when this workflow finishes.

mattip · 2022-07-18T04:05:17Z

You could try manually editing this change into the NumPy you have to see if it fixes the problem.

rcomer · 2022-07-18T10:16:07Z

I tried this with my numpy-dev environment and did not reproduce the error:

ipython session

In [1]: %paste
import numpy as np
from numpy import ma
from scipy.stats import mstats
outcome = ma.masked_array(np.random.randn(3, 2), mask=[[1, 0], [1, 0], [1, 0]])
res = mstats.ttest_rel(outcome[:, 0], outcome[:, 1])
# I don't think it is good practice to compare a masked element to a NaN,
# but this is what was in the SciPy test.
np.testing.assert_array_equal(res, (np.nan, np.nan))
# In any case, if this no longer passes, why not? 
# What is the value of `res` using NumPy main?
print(res)

## -- End pasted text --
[git-path]/numpy/numpy/testing/_private/utils.py:713: UserWarning: Warning: converting a masked element to nan.
  x = np.asanyarray(x)
Ttest_relResult(statistic=masked, pvalue=nan)

In [3]: res
Out[3]: Ttest_relResult(statistic=masked, pvalue=nan)

In [4]: print(res)
Ttest_relResult(statistic=masked, pvalue=nan)

In [5]: print(res[0])
--

In [6]: print(res[1])
nan

In [7]: print(res[0].data)
0.0

In [8]: np.__version__
Out[8]: '1.24.0.dev0+554.g64707310d'

This is with python 3.9. Perhaps you need a newer python version to reproduce the error? I think there was a change to nan-handling at python 3.10, but I don't know the details.

rossbar · 2022-07-18T20:52:07Z

I managed to successfully bisect this which identified 6d77c59 as the source of the failures.

FWIW the MRE in the OP didn't recreate the error (on either Python 3.9 or 3.10), I used the scipy test suite to bisect, i.e. at each step I ran pytest --pyargs scipy.stats.tests.test_mstats_basic.

My guess is this is indeed a masked array issue, and not related to Python nan handling. Relevant version info from the bisection:

Python 3.10.5
scipy 1.9.0rc2

mdhaber · 2022-07-18T21:18:53Z

Thanks. As I mentioned in the OP, I wasn't sure whether the snippet would reproduce the issue or not. Now that I know that it does not, perhaps I can dig a little more to see what is really going on.

mdhaber · 2022-07-18T21:40:21Z

Ah, I can reproduce the issue now. I will try to post a real MWE involving only NumPy.

mdhaber · 2022-07-18T22:10:45Z

Well this is not the main issue, but it is unusual

import numpy as np
a = np.ma.array(1, mask=True).sum()  # or np.ma.masked
b = (np.nan, np.nan)
np.testing.assert_array_equal(a, b)

Passes on both versions of NumPy (1.21.2 an the most recent weekly wheel).

Actually, assert_array_equal(a, b) appears to pass if a is MaskedConstant(0.) (displays as just masked in the console)
and b is any sequence or array. (Fails when b is a scalar, though.) Very unusual the SciPy test was performing this comparison in the first place, so I'll fix that, but should I also open an issue about the misbehaving assert_array_equal?

mdhaber · 2022-07-18T22:21:18Z

OK here we go. It doesn't just involve NumPy, unfortunately, but there is a behavior that was changed recently:

import numpy as np
from scipy import special
df = np.ma.array([1])
t = np.ma.masked
special.stdtr(df, t)

This used to produce a masked array with the only element masked; now it produces an unmasked array([0.5]).

A little simpler:

import numpy as np
import scipy.special
special.ndtr(np.ma.masked)  # was `masked`, now `0.5`

seberg · 2022-10-26T10:34:28Z

Is there still an issue here? The change to use __array_ufunc__ in NumPy has long been reverted, so the above produces the old results, unless there is something very weird with Python 3.11 going on?

seberg · 2022-12-07T23:58:30Z

Closing, @mdhaber please ping if there was something remaining, but I think we reverted the change and both the NumPy and SciPy releases are long ago by now.

mdhaber added the 00 - Bug label Jul 17, 2022

mdhaber mentioned this issue Jul 17, 2022

TST: stats.mstats.ttests: fix mistakes in tests scipy/scipy#16610

Closed

mdhaber changed the title ~~BUG: SciPy test failures due to changes in NumPy main~~ BUG: SciPy test failures due to changes in NumPy main or Python 3.11 Jul 18, 2022

mdhaber mentioned this issue Jul 22, 2022

MAINT: stats.mode: revert gh-15423 scipy/scipy#16429

Merged

Micky774 mentioned this issue Jul 25, 2022

FIX: Updated dtype resolution in _stack_along_minor_axis scipy/scipy#16628

Merged

patrickzib mentioned this issue Oct 26, 2022

[MNT] python 3.11 support sktime/sktime#3647

Closed

seberg closed this as completed Dec 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: SciPy test failures due to changes in NumPy main or Python 3.11 #22006

BUG: SciPy test failures due to changes in NumPy main or Python 3.11 #22006

mdhaber commented Jul 17, 2022 •

edited

Loading

charris commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022 •

edited

Loading

Uh oh!

mattip commented Jul 18, 2022 •

edited

Loading

Uh oh!

mattip commented Jul 18, 2022

Uh oh!

rcomer commented Jul 18, 2022

Uh oh!

rossbar commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022 •

edited

Loading

Uh oh!

mdhaber commented Jul 18, 2022 •

edited

Loading

Uh oh!

seberg commented Oct 26, 2022

Uh oh!

seberg commented Dec 7, 2022

Uh oh!

Uh oh!

BUG: SciPy test failures due to changes in NumPy main or Python 3.11 #22006

BUG: SciPy test failures due to changes in NumPy main or Python 3.11 #22006

Comments

mdhaber commented Jul 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue:

Reproduce the code example:

Error message:

NumPy/Python version information:

charris commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Jul 18, 2022

Uh oh!

rcomer commented Jul 18, 2022

Uh oh!

rossbar commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022

Uh oh!

mdhaber commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdhaber commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Oct 26, 2022

Uh oh!

seberg commented Dec 7, 2022

Uh oh!

mdhaber commented Jul 17, 2022 •

edited

Loading

mdhaber commented Jul 18, 2022 •

edited

Loading

mattip commented Jul 18, 2022 •

edited

Loading

mdhaber commented Jul 18, 2022 •

edited

Loading

mdhaber commented Jul 18, 2022 •

edited

Loading