Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI is failing because of Descending test id on line: 'ok 2 - should run tests with dependency repeatedly' #54535

Closed
tniessen opened this issue Aug 24, 2024 · 2 comments · Fixed by #54538
Closed

CI is failing because of Descending test id on line: 'ok 2 - should run tests with dependency repeatedly' #54535

tniessen opened this issue Aug 24, 2024 · 2 comments · Fixed by #54538
Labels
build Issues and PRs related to build files or the CI. test_runner Issues and PRs related to the test runner subsystem. test Issues and PRs related to the tests.

Comments

@tniessen
Copy link
Member

In this CI run, the only failure is test-runner-watch-mode, which fails very frequently (see #49605).

It is marked as flaky though so the CI still eventually displays: All tests passed.

The CI run still fails due to the following error:

16:57:05 tap2junit -i test.tap -o js-tests.junit.xml  
16:57:08  if errorlevel 1 exit /b 
16:57:08 ) 
16:57:08 Traceback (most recent call last):
16:57:08   File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
16:57:08     return _run_code(code, main_globals, None,
16:57:08   File "C:\Python310\lib\runpy.py", line 86, in _run_code
16:57:08     exec(code, run_globals)
16:57:08   File "C:\Python310\Scripts\tap2junit.exe\__main__.py", line 7, in <module>
16:57:08   File "C:\Python310\lib\site-packages\tap2junit\__main__.py", line 94, in main
16:57:08     convert(
16:57:08   File "C:\Python310\lib\site-packages\tap2junit\__main__.py", line 68, in convert
16:57:08     result = parse(name or input_file, data, package)
16:57:08   File "C:\Python310\lib\site-packages\tap2junit\__main__.py", line 58, in parse
16:57:08     tap_parser.parse(data)
16:57:08   File "C:\Python310\lib\site-packages\tap2junit\tap13.py", line 237, in parse
16:57:08     self._parse(io.StringIO(source))
16:57:08   File "C:\Python310\lib\site-packages\tap2junit\tap13.py", line 194, in _parse
16:57:08     raise ValueError(f"Descending test id on line: {line!r}")
16:57:08 ValueError: Descending test id on line: 'not ok 2 - should run tests with dependency repeatedly'

That line appears in the error output of test-runner-watch-mode:

00:29:32 not ok 799 parallel/test-runner-watch-mode # TODO : Fix flaky test
00:29:32   ---
00:29:32   duration_ms: 14971.46800
00:29:32   severity: flaky
00:29:32   exitcode: 1
00:29:32   stack: |-
00:29:32     TAP version 13
00:29:32     # Subtest: test runner watch mode
00:29:32         # Subtest: should run tests repeatedly
00:29:32         not ok 1 - should run tests repeatedly
00:29:32           ---
00:29:32           duration_ms: 3830.556
00:29:32           location: 'C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\parallel\\test-runner-watch-mode.mjs:136:3'
00:29:32           failureType: 'testCodeFailure'
00:29:32           error: |-
00:29:32             The input did not match the regular expression /# tests 1/. Input:
00:29:32             
00:29:32             '# Subtest: test has ran\n' +
00:29:32               'ok 1 - test has ran\n' +
00:29:32               '  ---\n' +
00:29:32               '  duration_ms: 5.2762\n' +
00:29:32               '  ...\n' +
00:29:32               '# Subtest: C:\\\\workspace\\\\node-test-binary-windows-js-suites\\\\node\\\\test\\\\.tmp.684\\\\test.js\n' +
00:29:32               'not ok 2 - C:\\\\workspace\\\\node-test-binary-windows-js-suites\\\\node\\\\test\\\\.tmp.684\\\\test.js\n' +
00:29:32               '  ---\n' +
00:29:32               '  duration_ms: 1068.813\n' +
00:29:32               "  location: 'C:\\\\workspace\\\\node-test-binary-windows-js-suites\\\\node\\\\test\\\\.tmp.684\\\\test.js:1:1'\n" +
00:29:32               "  failureType: 'testCodeFailure'\n" +
00:29:32               '  exitCode: ~\n' +
00:29:32               "  signal: 'SIGTERM'\n" +
00:29:32               "  error: 'test failed'\n" +
00:29:32               "  code: 'ERR_TEST_FAILURE'\n" +
00:29:32               '  ...\n' +
00:29:32               '# Subtest: test has ran\n' +
00:29:32               'ok 3 - test has ran\n' +
00:29:32               '  ---\n' +
00:29:32               '  duration_ms: 6.1427\n' +
00:29:32               '  ...\n' +
00:29:32               '1..3\n' +
00:29:32               '# tests 3\n' +
00:29:32               '# suites 0\n' +
00:29:32               '# pass 2\n' +
00:29:32               '# fail 1\n' +
00:29:32               '# cancelled 0\n' +
00:29:32               '# skipped 0\n' +
00:29:32               '# todo 0\n' +
00:29:32               '# duration_ms 741.4156\n'
00:29:32             
00:29:32           code: 'ERR_ASSERTION'
00:29:32           name: 'AssertionError'
00:29:32           expected:
00:29:32           actual: |-
00:29:32             # Subtest: test has ran
00:29:32             ok 1 - test has ran
00:29:32               ---
00:29:32               duration_ms: 5.2762
00:29:32               ...
00:29:32             # Subtest: C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\.tmp.684\\test.js
00:29:32             not ok 2 - C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\.tmp.684\\test.js
00:29:32               ---
00:29:32               duration_ms: 1068.813
00:29:32               location: 'C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\.tmp.684\\test.js:1:1'
00:29:32               failureType: 'testCodeFailure'
00:29:32               exitCode: ~
00:29:32               signal: 'SIGTERM'
00:29:32               error: 'test failed'
00:29:32               code: 'ERR_TEST_FAILURE'
00:29:32               ...
00:29:32             # Subtest: test has ran
00:29:32             ok 3 - test has ran
00:29:32               ---
00:29:32               duration_ms: 6.1427
00:29:32               ...
00:29:32             1..3
00:29:32             # tests 3
00:29:32             # suites 0
00:29:32             # pass 2
00:29:32             # fail 1
00:29:32             # cancelled 0
00:29:32             # skipped 0
00:29:32             # todo 0
00:29:32             # duration_ms 741.4156
00:29:32             
00:29:32           operator: 'match'
00:29:32           stack: |-
00:29:32             testUpdate (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-watch-mode.mjs:74:14)
00:29:32             process.processTicksAndRejections (node:internal/process/task_queues:105:5)
00:29:32             async testWatch (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-watch-mode.mjs:129:26)
00:29:32             async TestContext.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-watch-mode.mjs:137:5)
00:29:32             async Test.run (node:internal/test_runner/test:879:9)
00:29:32             async Promise.all (index 0)
00:29:32             async Suite.run (node:internal/test_runner/test:1239:7)
00:29:32             async Test.processPendingSubtests (node:internal/test_runner/test:590:7)
00:29:32           ...
00:29:32         # Subtest: should run tests with dependency repeatedly
00:29:32         ok 2 - should run tests with dependency repeatedly
00:29:32           ---
00:29:32           duration_ms: 2051.9995
00:29:32           ...
00:29:32         # Subtest: should run tests with ESM dependency
00:29:32         ok 3 - should run tests with ESM dependency
00:29:32           ---
00:29:32           duration_ms: 1931.9474
00:29:32           ...
00:29:32         # Subtest: should support running tests without a file
00:29:32         ok 4 - should support running tests without a file
00:29:32           ---
00:29:32           duration_ms: 1877.6795
00:29:32           ...
00:29:32         # Subtest: should support a watched test file rename
00:29:32         ok 5 - should support a watched test file rename
00:29:32           ---
00:29:32           duration_ms: 2002.1372
00:29:32           ...
00:29:32         # Subtest: should not throw when delete a watched test file
00:29:32         ok 6 - should not throw when delete a watched test file
00:29:32           ---
00:29:32           duration_ms: 2463.515
00:29:32           ...
00:29:32         1..6
00:29:32     not ok 1 - test runner watch mode
00:29:32       ---
00:29:32       duration_ms: 14164.3195
00:29:32       type: 'suite'
00:29:32       location: 'C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\parallel\\test-runner-watch-mode.mjs:134:1'
00:29:32       failureType: 'subtestsFailed'
00:29:32       error: '1 subtest failed'
00:29:32       code: 'ERR_TEST_FAILURE'
00:29:32       ...
00:29:32     1..1
00:29:32     # tests 6
00:29:32     # suites 1
00:29:32     # pass 5
00:29:32     # fail 1
00:29:32     # cancelled 0
00:29:32     # skipped 0
00:29:32     # todo 0
00:29:32     # duration_ms 14185.4253
00:29:32   ...

cc @nodejs/test_runner
@tniessen tniessen added build Issues and PRs related to build files or the CI. test_runner Issues and PRs related to the test runner subsystem. labels Aug 24, 2024
@RedYetiDev RedYetiDev added the test Issues and PRs related to the tests. label Aug 24, 2024
@RedYetiDev
Copy link
Member

(I've added the test label as well as this also involves the test.py file)

cjihrig added a commit to cjihrig/node that referenced this issue Aug 24, 2024
@cjihrig
test: force spec reporter in test-runner-watch-mode.mjs
6dbe95c 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
all.

Fixes: nodejs#54535
@cjihrig cjihrig mentioned this issue Aug 24, 2024
Merged
@cjihrig
Copy link
Contributor

cjihrig commented Aug 24, 2024

Proposed fix in #54538.

cjihrig added a commit to cjihrig/node that referenced this issue Aug 24, 2024
@cjihrig
test: force spec reporter in test-runner-watch-mode.mjs
1fc1ba8 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
from the spawned child process.

Fixes: nodejs#54535
@cjihrig cjihrig mentioned this issue Aug 24, 2024
Closed
nodejs-github-bot pushed a commit that referenced this issue Aug 26, 2024
@cjihrig
test: force spec reporter in test-runner-watch-mode.mjs
05bd3cf 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
from the spawned child process.

Fixes: #54535
PR-URL: #54538
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
RafaelGSS pushed a commit that referenced this issue Aug 30, 2024
@cjihrig @RafaelGSS
test: force spec reporter in test-runner-watch-mode.mjs
319dcc7 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
from the spawned child process.

Fixes: #54535
PR-URL: #54538
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
RafaelGSS pushed a commit that referenced this issue Aug 30, 2024
@cjihrig @RafaelGSS
test: force spec reporter in test-runner-watch-mode.mjs
b5eb24c 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
from the spawned child process.

Fixes: #54535
PR-URL: #54538
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
louwers pushed a commit to louwers/node that referenced this issue Nov 2, 2024
@cjihrig @louwers
test: force spec reporter in test-runner-watch-mode.mjs
fe87ba7 
In the CI this test generates TAP output that can confuse the
Python test runner. Avoid the problem by not outputting TAP at
from the spawned child process.

Fixes: nodejs#54535
PR-URL: nodejs#54538
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build Issues and PRs related to build files or the CI. test_runner Issues and PRs related to the test runner subsystem. test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: force spec reporter in test-runner-watch-mode.mjs cjihrig/node
3 participants
@cjihrig @tniessen @RedYetiDev