test: don't skip when common.mustCall() is pending #15421

cjihrig · 2017-09-14T23:34:06Z

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors.

Fixes: #15419

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

test

vsemozhetbyt · 2017-09-15T03:53:54Z

We can use printSkipMessage(msg) before return if needed in such cases.

mscdex · 2017-09-15T04:54:00Z

@vsemozhetbyt But does that cause any issues for non-IPv6 systems when one of the IPv4 tests fails? I don't know offhand if 'Skip' output overrides exit code checks when determining if the test file failed or anything like that...

vsemozhetbyt · 2017-09-15T05:04:12Z

@mscdex Well, I cannot foresee that, but printSkipMessage(msg) is simply a wrapper for console.log() now.

vsemozhetbyt · 2017-09-15T05:06:22Z

FWIW, refs: #14016 (comment)

Trott · 2017-09-15T05:09:03Z

I think the approach in this PR is the right one. I don't think it should use printSkipMessage() because that will print a TAP message. Our TAP messages apply on a per-file basis. So unless we're skipping the entire test file, we should not print the skip message. Even if it's harmless, it will be confusing. If something should be printed, just go with console.error('Skipping test case that requires IPv6'); or something like that.

vsemozhetbyt · 2017-09-15T09:14:20Z

@Trott So does printSkipMessage() endanger any test? What are the criteria of safe/unsafe using? It is used in ~ 10 tests now. Should we check them all? All of them used skip() previously with the same messages.

mscdex · 2017-09-15T21:18:56Z

LGTM

Trott · 2017-09-16T12:29:09Z

@Trott So does printSkipMessage() endanger any test?

I don't think so.

What are the criteria of safe/unsafe using?

It's probably always safe. If the test file exits with an error, it still counts as not ok.

It is used in ~ 10 tests now. Should we check them all? All of them used skip() previously with the same messages.

I don't know. If I see ok 23 # skip Insufficient flogiston pressure. in the results, I assume the entire test was skipped, not part of it. But (a) I'm not sure that the type of message will result with printSkipMessage() if there is other output as well and (b) I'm not sure everyone else will agree with me that a skip message like that suggests the entire file was skipped and not merely part of it.

Not sure who to ping as the most informed people about TAP and how we use it in Jenkins, so @nodjes/build @nodejs/testing

Trott · 2017-09-16T12:35:36Z

In the interest of expediting this to get CI to green, I'm totally OK with going with printSkipMessage() and figuring out if it's an issue later on. I'm also OK with landing as is and figuring it out later. Like, don't hold this up over my trivial concerns. :-D

vsemozhetbyt · 2017-09-16T12:52:26Z

@Trott FWIW, we do have tests where printSkipMessage() means partial skipping. But in these very tests we previously had skip() with the same partial skipping. For example:

test-dgram-bind-default-address.js now
test-dgram-bind-default-address.js before printSkipMessage() emerges.

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors. Fixes: nodejs#15419 PR-URL: nodejs#15421 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

cjihrig · 2017-09-17T01:25:04Z

CI: https://ci.nodejs.org/job/node-test-pull-request/10112/

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors. Fixes: nodejs/node#15419 PR-URL: nodejs/node#15421 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors. Fixes: #15419 PR-URL: #15421 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors. Fixes: nodejs/node#15419 PR-URL: nodejs/node#15421 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

The test parallel/test-dgram-multicast-set-interface.js was calling common.skip() on hosts that do not support IPv6. However, by this point, there were several outstanding common.mustCall() invocations. The process.exit() in common.skip() triggered those common.mustCall()s as errors. Fixes: #15419 PR-URL: #15421 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

nodejs-github-bot added the test Issues and PRs related to the tests. label Sep 14, 2017

cjihrig mentioned this pull request Sep 14, 2017

Investigate flaky parallel/test-dgram-multicast-set-interface on pi{1,2}-raspbian-wheezy #15419

Closed

mscdex added the dgram Issues and PRs related to the dgram subsystem / UDP. label Sep 14, 2017

jasnell approved these changes Sep 15, 2017

View reviewed changes

BridgeAR approved these changes Sep 15, 2017

View reviewed changes

Trott approved these changes Sep 15, 2017

View reviewed changes

danbev approved these changes Sep 15, 2017

View reviewed changes

lpinca approved these changes Sep 15, 2017

View reviewed changes

thefourtheye approved these changes Sep 15, 2017

View reviewed changes

cjihrig force-pushed the test branch from e81675d to 631c59b Compare September 17, 2017 01:24

cjihrig merged commit 631c59b into nodejs:master Sep 17, 2017

cjihrig deleted the test branch September 17, 2017 01:26

jasnell mentioned this pull request Sep 20, 2017

Node.js 8.6.0 Release Proposal #15509

Merged

MylesBorins added the dont-land-on-v6.x label Oct 17, 2017

joyeecheung mentioned this pull request Jan 16, 2018

Investigate flaky parallel/test-dgram-multicast-set-interface #18193

Closed

MylesBorins mentioned this pull request Jan 24, 2018

v6.13.0 proposal #18342

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: don't skip when common.mustCall() is pending #15421

test: don't skip when common.mustCall() is pending #15421

cjihrig commented Sep 14, 2017

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

mscdex commented Sep 15, 2017

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

vsemozhetbyt commented Sep 15, 2017

Trott commented Sep 15, 2017

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

mscdex commented Sep 15, 2017

Trott commented Sep 16, 2017 •

edited

Loading

Trott commented Sep 16, 2017

vsemozhetbyt commented Sep 16, 2017

cjihrig commented Sep 17, 2017

test: don't skip when common.mustCall() is pending #15421

test: don't skip when common.mustCall() is pending #15421

Conversation

cjihrig commented Sep 14, 2017

Checklist

Affected core subsystem(s)

vsemozhetbyt commented Sep 15, 2017 • edited Loading

mscdex commented Sep 15, 2017

vsemozhetbyt commented Sep 15, 2017 • edited Loading

vsemozhetbyt commented Sep 15, 2017

Trott commented Sep 15, 2017

vsemozhetbyt commented Sep 15, 2017 • edited Loading

mscdex commented Sep 15, 2017

Trott commented Sep 16, 2017 • edited Loading

Trott commented Sep 16, 2017

vsemozhetbyt commented Sep 16, 2017

cjihrig commented Sep 17, 2017

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

vsemozhetbyt commented Sep 15, 2017 •

edited

Loading

Trott commented Sep 16, 2017 •

edited

Loading