xdist misidentifies failing test on teardown crash

[reginaldl]

Hi!

I'm seeing xdist flagging the wrong test when a crash happens at test teardown.
Given 2 tests:

• test_a, passes but crashes at teardown
• test_b, passes

xdist will identify test_b as failing.
I have been able to reproduce while using pytest-xdist and pytest-timeout.
Here are the steps I've been following:

• Test file:

import pytest
import time

@pytest.fixture
def timeout_fixture(request):
    def long_wait():
        time.sleep(20)
    request.addfinalizer(long_wait)

def test_a(timeout_fixture):
    print("hello")

def test_b():
    print("world")

def test_c():
    print("bonjour")

def test_d():
    print("monde")

• Run test with --timeout-method=thread, so pytest-timeout will kill our process and xdist will see it crashed:
  $ py.test -s -x -vv --timeout=2 --timeout-method=thread -n 1

Result:

[...]
test_bug.py::test_a 
[gw0] PASSED test_bug.py::test_a 
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
[...]
[gw0] node down: Not properly terminated
[gw0] FAILED test_bug.py::test_b 
Replacing crashed slave gw0
[gw1] darwin Python 2.7.10 cwd: /private/tmp/test
[gw1] Python 2.7.10 (default, Jul 30 2016, 19:40:32)  -- [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
gw1 [4]
test_bug.py::test_d 
[gw1] PASSED test_bug.py::test_d 
test_bug.py::test_c 
[gw1] PASSED test_bug.py::test_c 
========================================================================================================================================================== FAILURES ==========================================================================================================================================================
________________________________________________________________________________________________________________________________________________________ test_bug.py _________________________________________________________________________________________________________________________________________________________
[gw0] darwin -- Python 2.7.10 /usr/bin/python
Slave 'gw0' crashed while running 'test_bug.py::test_b'
============================================================================================================================================= 1 failed, 3 passed in 4.01 seconds =============================================================================================================================================

test_b is seen as failing but hasn't been executed at all.

I believe this is due to the way xdist handles crashed items. When a node crashes, it pops the running item and reschedule it on a new node. However, when the node crashes at teardown, the running item has already been dequeued.

[reginaldl]

Hi! I see that the PR has been merged, thanks!

Any ETA for a release?

[nicoddemus]

That's out in 1.16.0, no?

[reginaldl]

This is looking at master, v1.16.0 tag does not include this merge.
I think my initial update to the Changelog is at the wrong place, I'll send an update.

[nicoddemus]

Fixed in 1.17.0.

[nicoddemus]

We had to pull this back in 1.17.1 (#157). I will see if I can reproduce the problem in pytest-xdist itself.

Also, @reginaldl would you like to give this another try? 😁