Flaky re-runs xfail tests #184
Description
If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.
test_it.py:
import pytest
@pytest.mark.xfail
def test_xfail():
assert False
def test_fail():
assert False
Run with pytest:
% pytest -vv --force-flaky
============================================ test session starts ============================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/virtualenvs/tmp-d591713aeced424/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/flakybug
plugins: flaky-3.7.0
collected 2 items
test_it.py::test_xfail XFAIL [ 50%]
test_it.py::test_xfail XFAIL [ 50%]
test_it.py::test_fail FAILED [100%]
================================================= FAILURES ==================================================
_________________________________________________ test_fail _________________________________________________
def test_fail():
> assert False
E assert False
test_it.py:8: AssertionError
===Flaky Test Report===
test_xfail failed (1 runs remaining out of 2).
<class 'AssertionError'>
assert False
[<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_xfail failed; it passed 0 out of the required 1 times.
<class 'AssertionError'>
assert False
[<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_fail failed (1 runs remaining out of 2).
<class 'AssertionError'>
assert False
[<TracebackEntry /private/tmp/flakybug/test_it.py:8>]
test_fail failed; it passed 0 out of the required 1 times.
<class 'AssertionError'>
assert False
[<TracebackEntry /private/tmp/flakybug/test_it.py:8>]
===End Flaky Test Report===
========================================== short test summary info ==========================================
FAILED test_it.py::test_fail - assert False
======================================= 1 failed, 2 xfailed in 0.06s ========================================
The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.
% pip freeze
attrs==21.2.0
flaky==3.7.0
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.6
pytest==6.2.5
toml==0.10.2