[MRG] Make tests runnable with pytest without error #8246

lesteve · 2017-01-30T15:24:17Z

Related to #7319. This is the minimal set of changes to make the tests runnable with pytest. All in all it amounts to:

splitting a test function in two functions. I made sure that this was still testing correctly the fix in c31ad7a.
moving check_ functions using closures to module-level

Errors were due to pytest quirks with (deprecated) yield support. AFAICT the underlying reason is that pytest first collects the tests (collecting the test function and the arguments) and then runs them. It is somewhat easy to modify the arguments at collection time which creates suprises during the test run.

See below for an example test highlighting the quirk:

class A(object):
    def __init__(self, a):
        self.a = a

args = [A(i) for i in [0]]


def check(x, y):
    print('\nin check:', 'x.a:', x.a, 'y.a:', y.a)
    assert x.a == y.a


def test():
    for arg in args:
        arg_copy = A(arg.a)

        print('\nin test: arg.a:', arg.a, 'arg_copy.a:', arg_copy.a)
        yield check, arg, arg_copy

        arg_copy.a = -100
        assert arg_copy.a == -100

Output:

❯ pytest -s test_pytest.py
======================================================== test session starts =========================================================
platform linux -- Python 3.5.2, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /tmp, inifile: 
plugins: cov-2.3.1
collecting 0 items
in test: arg.a: 0 arg_copy.a: 0
collected 1 items 

test_pytest.py 
in check: x.a: 0 y.a: -100
F

============================================================== FAILURES ==============================================================
______________________________________________________________ test[0] _______________________________________________________________

x = <test_pytest.A object at 0x7efcd0c6ab38>, y = <test_pytest.A object at 0x7efcd0716048>

    def check(x, y):
        print('\nin check:', 'x.a:', x.a, 'y.a:', y.a)
>       assert x.a == y.a
E       assert 0 == -100
E        +  where 0 = <test_pytest.A object at 0x7efcd0c6ab38>.a
E        +  and   -100 = <test_pytest.A object at 0x7efcd0716048>.a

test_pytest.py:10: AssertionError
======================================================= pytest-warning summary =======================================================
WC1 /tmp/test_pytest.py yield tests are deprecated, and scheduled to be removed in pytest 4.0
============================================ 1 failed, 1 pytest-warnings in 0.05 seconds =============================================

lesteve · 2017-02-01T10:51:47Z

ping @karandesai-96 just in case you are interested to have some kind of explanation about the weird failure we were seeing with yield in joblib.

kdexd · 2017-02-01T12:18:37Z

Pytest >= 3.0.0 has dropped support of yield based tests. Basically we can look at yield statements in tests to be invocations of the test for different test cases. Pytest has a mechanism which registers the total number of tests (multiple invocations of same test included). Multiple invocations are registered through the @parametrize marker, or by looking whether the fixture required by test is parametrized (that is, whether it has params kwarg in @pytest.fixture decorator).

So yield based tests in their original form throw a pytest warning, but fiddling with them by passing fixtures or parametrize markers broke them. That's all I can say here. 😄

lesteve · 2017-02-02T09:17:39Z

~~Tests added.~~ Ooops wrong PR ...

lesteve · 2017-02-09T07:50:02Z

ping @jnothman @ogrisel.

jnothman · 2017-02-09T13:32:48Z

@lesteve, I'm not yet familiar with py.tests or its quirks, so I feel under-qualified to give a thorough review. But I appreciate the risk of merge conflicts, so will do what I can.

jnothman · 2017-02-09T13:35:55Z

Regarding the quirk you describe above, is one option to wrap the generators so that all yielded arguments are deepcopied during iteration?

jnothman · 2017-02-09T13:36:53Z

I realise that won't fix the issue with closures.

jnothman · 2017-02-09T13:38:31Z

Without a wrapper like that, it seems one needs to carefully check that all arguments are immutable or otherwise unmodified. I'd rather something with assurances like a deepcopy of all arguments..?

jnothman · 2017-02-09T13:40:50Z

I think if we're going to support pytest we need CI to ensure it continues to work. Add a travis build that uses pytest??

lesteve · 2017-02-09T14:24:33Z

I think if we're going to support pytest we need CI to ensure it continues to work. Add a travis build that uses pytest??

I was thinking about pytest as a "use at your own risks" kind of thing for now, hence the minimal PR. Given the limited set of changes I had to do to get the tests running without error with pytest, I am strongly encouraged to think that going forward the tests pass will pass with pytest if they pass with nose.

As soon as this PR is merged, I will use pytest to run the tests and I am willing to iron out any quirks, if any, that we find along the road.

jnothman · 2017-02-10T01:24:08Z

I don't see much harm in a pytest CI run during the transition. @ogrisel?

codecov · 2017-02-14T11:04:23Z

Codecov Report

Merging #8246 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8246      +/-   ##
==========================================
+ Coverage   94.75%   94.75%   +<.01%     
==========================================
  Files         342      342              
  Lines       60801    60804       +3     
==========================================
+ Hits        57609    57612       +3     
  Misses       3192     3192

Impacted Files	Coverage Δ
sklearn/neighbors/tests/test_kd_tree.py	`97.45% <100%> (ø)`	✅
sklearn/neighbors/tests/test_kde.py	`98.85% <100%> (ø)`	✅
sklearn/gaussian_process/tests/test_kernels.py	`98.85% <100%> (+0.02%)`	✅
sklearn/neighbors/tests/test_ball_tree.py	`98% <100%> (ø)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80c1bf1...ba64c10. Read the comment docs.

lesteve · 2017-02-14T15:15:15Z

I added a build on Travis that uses pytest.

Note that at the moment pytest does not run doctests from the rst in the doc folder. It seems like there is no easy way to do that and that pytest and nosetests are quite different on this topic so I would advise to tackle this when we drop nose.

lesteve · 2017-02-16T09:14:09Z

@ogrisel feel free to take a look at this if you have a bit of time

Errors were due to pytest quirks with (deprecated) yield support.

and tweak pytest settings in setup.cfg

codecov-io · 2017-02-20T08:30:13Z

Codecov Report

Merging #8246 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8246      +/-   ##
==========================================
+ Coverage   94.75%   94.75%   +<.01%     
==========================================
  Files         342      342              
  Lines       60801    60804       +3     
==========================================
+ Hits        57609    57612       +3     
  Misses       3192     3192

Impacted Files	Coverage Δ
sklearn/neighbors/tests/test_ball_tree.py	`98% <100%> (ø)`	✅
sklearn/neighbors/tests/test_kde.py	`98.85% <100%> (ø)`	✅
sklearn/neighbors/tests/test_kd_tree.py	`97.45% <100%> (ø)`	✅
sklearn/gaussian_process/tests/test_kernels.py	`98.85% <100%> (+0.02%)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80c1bf1...af94767. Read the comment docs.

codecov-io · 2017-02-20T08:30:13Z

Codecov Report

Merging #8246 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8246      +/-   ##
==========================================
+ Coverage   94.75%   94.75%   +<.01%     
==========================================
  Files         342      342              
  Lines       60801    60816      +15     
==========================================
+ Hits        57609    57624      +15     
  Misses       3192     3192

Impacted Files	Coverage Δ
sklearn/neighbors/tests/test_kde.py	`98.85% <100%> (ø)`	✅
sklearn/gaussian_process/tests/test_kernels.py	`98.85% <100%> (+0.02%)`	✅
sklearn/neighbors/tests/test_ball_tree.py	`98% <100%> (ø)`	✅
sklearn/neighbors/tests/test_kd_tree.py	`97.45% <100%> (ø)`	✅
sklearn/linear_model/tests/test_randomized_l1.py	`100% <ø> (ø)`	✅
sklearn/model_selection/tests/test_search.py	`99.31% <ø> (ø)`	✅
sklearn/linear_model/randomized_l1.py	`94.76% <ø> (+0.06%)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80c1bf1...af94767. Read the comment docs.

ogrisel

LGTM, I think this a good first step and it's nice to have both nosetests and pytest work for the next release. After 0.19 we can refactor the test suite deeper to make it more py.test native (using parametrize instead of yields) and drop nosetests support in 0.20.

ogrisel · 2017-02-20T12:08:46Z

Squash merged. The changes are small so it should be easy to revert shall it causes any problem.

kdexd · 2017-02-20T15:03:54Z

This is a good move 😄

jnothman · 2017-02-23T11:25:41Z

build_tools/travis/test_script.sh

-        nosetests -s --with-coverage --with-timer --timer-top-n 20 sklearn
-    else
-        nosetests -s --with-timer --timer-top-n 20 sklearn
+        TEST_CMD="$TEST_CMD --with-coverage"


I'm not sure you mean to repeat --with-coverage here and in setting TEST_CMD above

Good point, I'll fix that.

* Make tests runnable with pytest without error. Errors were due to pytest quirks with (deprecated) yield support. * Add pytest build on Travis and tweak pytest settings in setup.cfg * Tweak comment

lesteve mentioned this pull request Feb 1, 2017

Move to pytest? #7319

Closed

lesteve force-pushed the make-pytest-pass branch from 1b55331 to 99dad77 Compare February 14, 2017 11:04

lesteve force-pushed the make-pytest-pass branch from 99dad77 to abc2787 Compare February 14, 2017 13:53

lesteve force-pushed the make-pytest-pass branch from abc2787 to dff34c3 Compare February 16, 2017 10:39

lesteve added 2 commits February 16, 2017 13:11

Make tests runnable with pytest without error.

e5634ca

Errors were due to pytest quirks with (deprecated) yield support.

Add pytest build on Travis

ba64c10

and tweak pytest settings in setup.cfg

lesteve force-pushed the make-pytest-pass branch from dff34c3 to ba64c10 Compare February 16, 2017 12:11

Tweak comment

af94767

ogrisel approved these changes Feb 20, 2017

View reviewed changes

ogrisel merged commit 0ec6664 into scikit-learn:master Feb 20, 2017

lesteve deleted the make-pytest-pass branch February 20, 2017 12:11

jnothman reviewed Feb 23, 2017

View reviewed changes

Przemo10 mentioned this pull request Mar 17, 2017

update fork (#1) #8606

Closed

lesteve mentioned this pull request Jul 26, 2017

Make pytest run the doctests in doc rst files #9445

Closed

massich mentioned this pull request Jul 26, 2017

switch travis + doc from nosetest to pytest mne-tools/mne-python#4412

Closed

Uh oh!

[MRG] Make tests runnable with pytest without error #8246

[MRG] Make tests runnable with pytest without error #8246

Uh oh!

Conversation

lesteve commented Jan 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lesteve commented Feb 1, 2017

Uh oh!

kdexd commented Feb 1, 2017

Uh oh!

lesteve commented Feb 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lesteve commented Feb 9, 2017

Uh oh!

jnothman commented Feb 9, 2017

Uh oh!

jnothman commented Feb 9, 2017

Uh oh!

jnothman commented Feb 9, 2017

Uh oh!

jnothman commented Feb 9, 2017

Uh oh!

jnothman commented Feb 9, 2017

Uh oh!

lesteve commented Feb 9, 2017

Uh oh!

jnothman commented Feb 10, 2017

Uh oh!

codecov bot commented Feb 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lesteve commented Feb 14, 2017

Uh oh!

lesteve commented Feb 16, 2017

Uh oh!

codecov-io commented Feb 20, 2017

Codecov Report

Uh oh!

codecov-io commented Feb 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Feb 20, 2017

Uh oh!

kdexd commented Feb 20, 2017

Uh oh!

jnothman Feb 23, 2017

Choose a reason for hiding this comment

Uh oh!

lesteve Feb 23, 2017

Choose a reason for hiding this comment

Uh oh!

lesteve Feb 23, 2017

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lesteve commented Jan 30, 2017 •

edited

Loading

lesteve commented Feb 2, 2017 •

edited

Loading

codecov bot commented Feb 14, 2017 •

edited

Loading

codecov-io commented Feb 20, 2017 •

edited

Loading