Crash when assert fails in multiprocessing task

[adborden]

Seems that pytest crashes when trying to get the code snippet of an assertion failure when the assertion failure occurs in a multiprocessing task.

Here's a log of the pytest run https://gist.github.com/adborden/376f2dd3defae8fced33b655913e5f91#file-pytest_crash-log

Here's the test case that causes the crash
https://gist.github.com/adborden/376f2dd3defae8fced33b655913e5f91#file-test_pytest_crash-py

Here's a snippet of the traceback (see above for full stack).

INTERNALERROR>   File "/home/adborden/.pyenv/versions/3.4.5/envs/pytest-crash/lib/python3.4/site-packages/_pytest/_code/code.py", line 373, in __init__
INTERNALERROR>     self.traceback = _pytest._code.Traceback(self.tb, excinfo=self)
INTERNALERROR>   File "/home/adborden/.pyenv/versions/3.4.5/envs/pytest-crash/lib/python3.4/site-packages/_pytest/_code/code.py", line 270, in __init__
INTERNALERROR>     list.__init__(self, tb)
INTERNALERROR> TypeError: 'NoneType' object is not iterable

Using pytest-3.0.3 on Ubuntu 15.10

Virtualenv:

$ pip list
pip (8.1.1)
py (1.4.31)
pytest (3.0.3)
setuptools (20.10.1)

[RonnyPfannschmidt]

indeed, py.test currently expects real tracebacks, and multiprocessing is NOT supplying that
it seems like an issue we will have to work around :(

[Daenyth]

This is affecting me really heavily - I'm trying to covert a large python2 app to python3 and this is preventing most of the suite from running.

[nicoddemus]

I noticed this only happens because of the code which tries to print chained exceptions in Python 3+. As a workaround, it is possible to use pytest --tb=native:

_____________________________ test_multitask_job ______________________________
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\pytest\.tmp\test_foo.py", line 5, in process_task
    assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\pytest\.tmp\test_foo.py", line 15, in test_multitask_job
    multitask_job()
  File "C:\pytest\.tmp\test_foo.py", line 11, in multitask_job
    pool.map(process_task, tasks, chunksize=1)
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 608, in get
    raise self._value
AssertionError
========================== 1 failed in 0.37 seconds ===========================

[Daenyth]

That works, thanks!

[Daenyth]

Thanks, that works! I suspect that it also happens for threading, not just
multiprocessing, since I'm not actually using multiprocessing directly,
though maybe flask's test utilities do?

On Tue, Oct 25, 2016 at 8:33 PM Bruno Oliveira notifications@github.com
wrote:

I noticed this only happens because of the code which tries to print
chained exceptions in Python 3+. As a workaround, it is possible to use pytest
--tb=native:

_____________________________ test_multitask_job ______________________________
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(_args, *_kwds))
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\pytest.tmp\test_foo.py", line 5, in process_task
assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\pytest.tmp\test_foo.py", line 15, in test_multitask_job
multitask_job()
File "C:\pytest.tmp\test_foo.py", line 11, in multitask_job
pool.map(process_task, tasks, chunksize=1)
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 608, in get
raise self._value
AssertionError
========================== 1 failed in 0.37 seconds ===========================

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1984 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA5NMOTab1XvsU8z-2_MlJCi5OQ84I0ks5q3p_fgaJpZM4KODNP
.

[nicoddemus]

@Daenyth could you please post the traceback you were getting? Just want to make sure you were also having the exact same problem as the OP.

[Daenyth]

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 96, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 131, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 152, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 79, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 135, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/hypothesis/extra/pytestplugin.py", line 69, in pytest_runtest_makereport
INTERNALERROR>     report = (yield).get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/skipping.py", line 222, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/pytest_capturelog.py", line 175, in pytest_runtest_makereport
INTERNALERROR>     report = __multicall__.execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 272, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/python.py", line 587, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/python.py", line 580, in _repr_failure_py
INTERNALERROR>     style=style)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 433, in _repr_failure_py
INTERNALERROR>     style=style, tbfilter=tbfilter)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 423, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 635, in repr_excinfo
INTERNALERROR>     excinfo = ExceptionInfo((type(e), e, e.__traceback__))
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 373, in __init__
INTERNALERROR>     self.traceback = _pytest._code.Traceback(self.tb, excinfo=self)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 270, in __init__
INTERNALERROR>     list.__init__(self, tb)
INTERNALERROR> TypeError: 'NoneType' object is not iterable

[nicoddemus]

Thanks!