test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

vstinner · 2017-07-04T15:16:51Z

BPO	30849
Nosy	@pitrou, @vstinner, @zware, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2021-05-26.22:47:39.082>
created_at = <Date 2017-07-04.15:16:51.076>
labels = ['tests']
title = 'test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x'
updated_at = <Date 2021-09-08.12:47:26.384>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-08.12:47:26.384>
actor = 'vstinner'
assignee = 'pitrou'
closed = True
closed_date = <Date 2021-05-26.22:47:39.082>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-07-04.15:16:51.076>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30849
keywords = []
message_count = 15.0
messages = ['297674', '297732', '299948', '299985', '299986', '300017', '300181', '300182', '300550', '318313', '320891', '320900', '321019', '394492', '401380']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30849'
versions = ['Python 3.6']

vstinner · 2017-07-04T15:16:51Z

Antoine: I told you that such stess-test is going to require work to polish it. So here is another example of failure. Enjoy :-)

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/534/steps/test/logs/stdio

== CPython 3.6.2rc1+ (heads/3.6:580cd5c, Jul 5 2017, 00:47:42) [GCC 4.9.2]
== Linux-3.16.0-4-amd64-x86_64-with-debian-8.7 little-endian
...
== CPU count: 1
...
Run tests in parallel using 2 child processes
0:11:40 load avg: 1.29 [383/405/1] test_signal failed -- running: test_datetime (38 sec)
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4602 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1086, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8206 != 10000 : Some signals were lost

vstinner · 2017-07-05T12:38:58Z

Another fail. The test pass when run again later.

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/538/steps/test/logs/stdio

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4073 != 10000 : Some signals were lost

vstinner · 2017-08-08T21:09:37Z

Hey Antoine, as expected, your test fails randomly on the CI :-(

Another failure:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/829/steps/test/logs/stdio

test_stress_delivery_dependent (test.test_signal.StressTest) ... Timeout (0:15:00)!
Thread 0xb74bd700 (most recent call first):
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 973 in measure_itimer_resolution
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 985 in decide_itimer_count
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 1001 in test_stress_delivery_dependent
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 615 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 663 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/runner.py", line 176 in run
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/init.py", line 1896 in _run_suite
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/init.py", line 1940 in run_unittest
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 168 in test_runner
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 169 in runtest_inner
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 137 in runtest
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 540 in _main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 510 in main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 585 in main
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/main.py", line 2 in <module>
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 85 in _run_code
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 193 in _run_module_as_main

pitrou · 2017-08-09T10:03:42Z

Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?

vstinner · 2017-08-09T10:31:37Z

Antoine: "Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?"

Contact directly the owners:

AMD64 Debian root 3.6: Chris Angelico <rosuav@gmail.com>
x86 Gentoo Non-Debug with X 3.x: Zachary Ware <zachary.ware+bb@gmail.com>

zware · 2017-08-09T15:04:36Z

Hmm. Perhaps someone can give me temporary shell access to one of those buildbots?

Check your @python.org inbox.

pitrou · 2017-08-11T22:14:59Z

Victor, Zachary, http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/830 corresponds to a very old changeset (more than one month old) before itimer() was fixed in 729780a.

vstinner · 2017-08-11T22:19:53Z

Oh, Zach's Gentoo buildbot was offline for a long time. Maybe it ran tests
on old commits? What about the first messages of this issue on the other
buildbot? Maybe it's also outdated? I am fine with closing the issue if you
fixed a bug in the meanwhile. Don't worry, I will reopen it if it comes
back :-D

vstinner · 2017-08-18T23:23:33Z

test_stress_delivery_dependent() failed on AMD64 Debian root 3.x too, it's not specific to 3.6:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1192/steps/test/logs/stdio

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1191/steps/test/logs/stdio

vstinner · 2018-05-31T14:00:52Z

Recent failure on AMD64 Debian root 3.7:

http://buildbot.python.org/all/#/builders/127/builds/361

0:03:33 load avg: 1.38 [140/415/1] test_signal failed

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3586 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1149, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8884 != 10000 : Some signals were lost

serhiy-storchaka · 2018-07-02T15:19:43Z

I can reproduce the failure on multiple OSes in VirtaulBox when run test_signal in parallel with test_regrtest:

./python -m test -j2 test_signal  test_regrtest test_regrtest test_regrtest
Run tests in parallel using 2 child processes
0:00:29 load avg: 4.24 [1/4] test_regrtest passed
0:00:58 load avg: 4.96 [2/4] test_regrtest passed -- running: test_signal (58 sec 728 ms)
0:01:00 load avg: 4.96 [3/4/1] test_signal failed
test test_signal failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5073 != 10000 : Some signals were lost

But can't reproduce on hardware.

serhiy-storchaka · 2018-07-02T16:38:25Z

After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.

vstinner · 2018-07-04T08:17:20Z

After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.

The test just failed on AMD64 Debian root 3.x and according to test.pythoninfo, this machine has a single CPU:

os.cpu_count: 1

http://buildbot.python.org/all/#/builders/27/builds/1247

...
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Using random seed 6423496
Run tests in parallel using 2 child processes
0:00:00 load avg: 0.71 [ 1/417] test_genericclass passed
0:00:00 load avg: 0.71 [ 2/417] test_copyreg passed
0:00:03 load avg: 0.71 [ 3/417] test_os passed
stty: standard input: Inappropriate ioctl for device
running: test_tokenize (32 sec 627 ms), test_signal (30 sec 11 ms)
0:00:51 load avg: 1.32 [ 4/417] test_tokenize passed (50 sec 821 ms) -- running: test_signal (48 sec 506 ms)
0:00:54 load avg: 1.32 [ 5/417/1] test_signal failed
...
test_stress_delivery_dependent (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000044 s.
FAIL
test_stress_delivery_simultaneous (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000032 s.
ok
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1156, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3617 != 10000 : Some signals were lost

vstinner · 2021-05-26T22:47:39Z

I didn't see this failure recently, I close the issue.

vstinner · 2021-09-08T12:47:26Z

We also got this error randomly on an internal s390x Red Hat build server:

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
This test uses simultaneous signal handlers.
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/lib64/python3.9/test/test_signal.py", line 1245, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5936 != 10000 : Some signals were lost

vstinner · 2023-09-20T14:45:39Z

I reopen the issue, the test still fails randomly on AMD64 Debian root 3.x: https://buildbot.python.org/all/#/builders/345/builds/5878

vstinner · 2023-09-29T17:27:48Z

See also issue gh-110083: test_signal: test_stress_modifying_handlers() crash with SIGSEGV on GHA macOS (macOS-12.7).

vstinner · 2023-11-08T14:45:13Z

Now that tests are re-run in verbose mode when they fail, the test no longer makes a whole test suite to be marked as failed. I close the issue again, so nobody seems to be eagger to fix the test or the code.

vstinner assigned pitrou Jul 4, 2017

vstinner added the tests Tests in the Lib/test dir label Jul 4, 2017

vstinner changed the title ~~test_stress_delivery_dependent() randomly fails on AMD64 Debian root 3.6~~ test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6 Jul 5, 2017

vstinner changed the title ~~test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6~~ test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x Aug 18, 2017

vstinner closed this as completed May 26, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

vstinner reopened this Sep 20, 2023

vstinner mentioned this issue Sep 29, 2023

test_signal: test_stress_modifying_handlers() crash with SIGSEGV on GHA macOS (macOS-12.7) #110083

Closed

vstinner mentioned this issue Oct 10, 2023

test_signal: test_stress_modifying_handlers() failed on s390x SLES 3.x #110647

Closed

vstinner closed this as completed Nov 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

vstinner commented Jul 4, 2017

vstinner commented Jul 4, 2017

vstinner commented Jul 5, 2017

vstinner commented Aug 8, 2017

pitrou commented Aug 9, 2017

vstinner commented Aug 9, 2017

zware commented Aug 9, 2017

pitrou commented Aug 11, 2017

vstinner commented Aug 11, 2017

vstinner commented Aug 18, 2017

vstinner commented May 31, 2018

serhiy-storchaka commented Jul 2, 2018

serhiy-storchaka commented Jul 2, 2018

vstinner commented Jul 4, 2018

vstinner commented May 26, 2021

vstinner commented Sep 8, 2021

vstinner commented Sep 20, 2023

vstinner commented Sep 29, 2023

vstinner commented Nov 8, 2023

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x #75032

Comments

vstinner commented Jul 4, 2017

vstinner commented Jul 4, 2017

vstinner commented Jul 5, 2017

vstinner commented Aug 8, 2017

pitrou commented Aug 9, 2017

vstinner commented Aug 9, 2017

zware commented Aug 9, 2017

pitrou commented Aug 11, 2017

vstinner commented Aug 11, 2017

vstinner commented Aug 18, 2017

vstinner commented May 31, 2018

serhiy-storchaka commented Jul 2, 2018

serhiy-storchaka commented Jul 2, 2018

vstinner commented Jul 4, 2018

vstinner commented May 26, 2021

vstinner commented Sep 8, 2021

vstinner commented Sep 20, 2023

vstinner commented Sep 29, 2023

vstinner commented Nov 8, 2023