Fixtures marked (scope='session') are run on every re-run

[dangercrow]

I would expect that session fixtures are run once per session - but it seems if a flaky test reruns, it will run session scoped fixtures' teardown code, and then their setup again.

This can result in costly setup/teardown being unnecessarily rerun, and also seems to go against the whole point of labelling the fixture session

Reproduction:

@pytest.fixture(scope="session")
def example_fixture():
    print("\nSetup")
    yield
    print("Teardown")

@pytest.mark.flaky(reruns=1)
def test_example_fixture(example_fixture):
    assert 0

Running with pytest -s -v ... I get:

plugins: timeout-1.2.0, rerunfailures-2.2, repeat-0.4.1, ordering-0.5
collected 1 items 

scripts/autotest/tests/conftest.py::test_example_fixture 
Setup
Teardown
RERUN
scripts/autotest/tests/conftest.py::test_example_fixture 
Setup
Teardown
FAILED

[sallner]

@dangercrow Thanks for the report. Just for completeness: Which pytest version do you use?

I could reproduce the behaviour you reported. It seems that pytest produces a complete teardown if there are no more items available for execution.

When I use the following example, setup and teardown seem to be only called once.

import pytest


@pytest.fixture(scope="session")
def example_fixture():
    print("\nSetup")
    yield
    print("\nTeardown")


fail_twice = (x for x in [False, False, True])


@pytest.mark.flaky(reruns=1)
def test_example_fixture_1(example_fixture):
    assert next(fail_twice)

def test_stay_true():
    assert True

The respective result of pytest -s -v ... is:

platform linux -- Python 3.5.4, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 -- 
cachedir: ../.cache
rootdir: /test, inifile: pytest.ini
plugins: rerunfailures-3.0, flake8-0.8.1, cov-2.4.0
collected 2 items 

rerun_fixtures.py::test_example_fixture_1 
Setup
RERUN
rerun_fixtures.py::test_example_fixture_1 FAILED
rerun_fixtures.py::test_stay_true PASSED
Teardown

Could you maybe comment on that, whether your failing test case was always the last one in your test setup?

I am not completely sure, whether it is possible to fix this easily as runtestprotocol requires a next item. One could try to pass the same item again if no other test is waiting in the queue and rerun is enabled. But then we have to ensure somehow that test is only run once if it passes.

[sallner]

@dangercrow Do you have comments on that issue anymore? I see that this seems to be a problem, but without further confirmation it is hard to work on that.

[dangercrow]

Ta for the ping.

pytest version is 3.1.2

It seems your test case illustrates the issue quite well - I see the same behaviour as you!
When the test_stay_true is renamed such that test_example_fixture_1 is the final test, I see the multiple runs of the session fixture, but when it is not the final test, behaviour is as semantically expected.

[robertrmartinez]

I have had the same issue as well.

I have a session fixture which opens and closes a tunnel to Sauce Labs,

This 'bug' causes the final test in my suite to close the session, then once again re-open it for the rerun. Causes a good minute or two overhead each time the final test fails.

Tests before the final test work as expected.

So, it looks like pytest sees the final test has just run, says "Okay lets tear this session down", and after that rerun triggers.

I haven't looked into the code, but seems like pytest needs to have visibility of a rerun before it makes the decision to initiate a tear down?

Thanks for all of your hard work!

[sallner]

@dangercrow @robertrmartinez it seems that we have another issue reported that is related to this one (#56) I will try to have a look into this issue over the next weeks.

[dnut]

I am observing the same behavior.
pytest 4.5.0
pytest-rerunfailures 7.0

[sbhonde1]

Hello, I am observing the same behavior.
I am using pytest for testing embedded code.
pytest 5.0.1

[joaonc]

I actually would argue that this is the expected and correct behavior.

In certain cases when re-running flaky tests, you may want to start from scratch as the flakyness may be in the setup of the tests or the setup of the tests may in some way cause flakyness down the line.

This looks like a scope that pytest is not intended to handle.. IMO, should be handled by the plugin as a command line option added with the pytest_addoption hook (ex. --rerun-session-fixtures or something like that).

[sokumar4]

@sallner I am also experiencing the same with pytest 5.3.5 & pytest-rerunfailures==7.0
Is there any workaround for now?
Is there any plan to fix it?

[dnut]

Is there any workaround for now?

One idea is to set a global variable when the fixture runs and checking the variable before executing. Since session fixtures are being re-run, state may be reset between re-runs. There must be state somewhere that persists though. pytest-dev/pytest#2461

[andyleejordan]

For what it's worth this happens for session="class" fixtures too. If the last test in a test class is marked flaky, and it fails, the whole class fixture is unexpectedly torn down and then set back up for the single test.

[bstaletic]

As of recently-ish the hack with a trivial test at the end of the file doesn't cut it any more. I attempted to roll back both pytest and rerunfailures, to no avail. The attempted pull request is here and you can see repeated Captured log teardown.

I'm hoping someone will have an idea, else I'll have to rewrite the test suite in unittest... and I really don't want to do that.

[henzef]

Any progress on this? This is quite a major bug for my setup :-/

[icemac]

@henzef Sorry, there is no progress on this. If someone wants to volunteer in the investigation and/or in fixing this issue: any help is welcome.

[HusuSama]

version: 10.3
I am observing the same behavior

[galloramiro]

Im observing the same behavior with scope=session and scope=class.

pytest = {index = "pypi",version = "==7.4"}
flaky = {index = "pypi",version = "==3.6.1"}

[AntonioMarrazzo]

Observing same behaviour, fixed using this

def xxx_fixture():
    if not hasattr(xxx_fixture, "already_instantiated"):
        xxx_fixture.already_instantiated = None
    if xxx_fixture.already_instantiated is None:
        code...
        yield something
        finisher code
    else:
        yield xxx_fixture.already_instantiated