Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: run the addon tests last #12062

Closed
wants to merge 1 commit into from

Conversation

seppevs
Copy link
Contributor

@seppevs seppevs commented Mar 27, 2017

Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

See #12031

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows [commit guidelines][]
Affected core subsystem(s)

test

@nodejs-github-bot nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Mar 27, 2017
@aqrln
Copy link
Contributor

aqrln commented Mar 27, 2017

I think that "See ..." line should be formatted as a proper "Refs:" or "Fixes:" metadata field. Also, there's probably a typo in the commit message (s/lasts/last ?).

@seppevs seppevs force-pushed the test_run_addon_tests_last branch from 9eeb691 to cb369d2 Compare March 27, 2017 10:42
@seppevs
Copy link
Contributor Author

seppevs commented Mar 27, 2017

Thanks for the tips, I've changed the commit message.

@seppevs seppevs changed the title test: run the addon tests lasts test: run the addon tests last Mar 27, 2017
@vsemozhetbyt
Copy link
Contributor

vsemozhetbyt commented Mar 27, 2017

@vsemozhetbyt
Copy link
Contributor

vsemozhetbyt commented Mar 27, 2017

Does this affect Windows builds? Should the vcbuild.bat also be fixed?

@vsemozhetbyt
Copy link
Contributor

Some tests failed, trying another CI: https://ci.nodejs.org/job/node-test-pull-request/7055/

@aqrln
Copy link
Contributor

aqrln commented Mar 27, 2017

@aqrln
Copy link
Contributor

aqrln commented Mar 27, 2017

@vsemozhetbyt those tests are unrelated to this PR and I don't think another CI run will help, tbh :)
Failed test on Windows is already marked as flaky in #12054, and I've seen parallel/test-async-wrap-check-providers as well as other tests failing with timeout on CentOS 5 for a while too (just open any of the recent PRs).

@seppevs seppevs force-pushed the test_run_addon_tests_last branch from cb369d2 to dc3a770 Compare March 27, 2017 13:45
@seppevs
Copy link
Contributor Author

seppevs commented Mar 27, 2017

@aqrln Thanks for the tip. I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

@aqrln
Copy link
Contributor

aqrln commented Mar 27, 2017

@seppevs heh, neither do I, so I hope my suggestion was correct :)

/cc @nodejs/build @nodejs/platform-windows

EDIT: whoops, only members can mention teams. @vsemozhetbyt can you do that, please?

@vsemozhetbyt
Copy link
Contributor

/cc @nodejs/build @nodejs/platform-windows

Copy link
Contributor

@Fishrock123 Fishrock123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please.

cc @bnoordhuis probably?

Makefile Outdated
@@ -195,7 +195,7 @@ test: all
$(MAKE) build-addons
$(MAKE) cctest
$(PYTHON) tools/test.py --mode=release -J \
addons doctool inspector known_issues message pseudo-tty parallel sequential
doctool inspector known_issues message pseudo-tty parallel sequential addons
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing. CI runs make test-ci, not make test, so you need to change line 311 too.

Copy link
Contributor Author

@seppevs seppevs Mar 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've changed the order on line 311 as well.

@seppevs seppevs force-pushed the test_run_addon_tests_last branch from dc3a770 to 3196f56 Compare March 27, 2017 14:37
Copy link
Member

@jasnell jasnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that failures in cctest also suffer this problem.

@mscdex mscdex added addons Issues and PRs related to native addons. test Issues and PRs related to the tests. labels Mar 27, 2017
@refack
Copy link
Contributor

refack commented Mar 29, 2017

I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

So on windows, it's only a half solution since build-addons is run before the tests, and so it the build fails no tests are run 😞

@aqrln
Copy link
Contributor

aqrln commented Mar 29, 2017

@refack hmm, not only only on Windows. The same goes for Unix.

Makefile Outdated
@@ -195,7 +195,7 @@ test: all
$(MAKE) build-addons
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe split this to "all other test" -> build -> "test addons"... 🤔

vcbuild.bat Outdated
if /i "%1"=="test" set test_args=%test_args% addons doctool known_issues message parallel sequential -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
if /i "%1"=="test-ci" set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap addons doctool inspector known_issues message sequential parallel&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok
if /i "%1"=="test" set test_args=%test_args% doctool known_issues message parallel sequential addons -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
if /i "%1"=="test-ci" set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap doctool inspector known_issues message sequential parallel addons&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is you decide to split the test, you'll need it here too...

Copy link
Contributor

@refack refack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much hassle to refactor vcbuild.bat
Current LGTM (works on Windows)

@seppevs seppevs force-pushed the test_run_addon_tests_last branch from 3196f56 to 141a6d9 Compare April 4, 2017 10:23
@jasnell
Copy link
Member

jasnell commented Apr 4, 2017

This needs a rebase before it can be landed.

Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: nodejs#12031
@seppevs seppevs force-pushed the test_run_addon_tests_last branch from 141a6d9 to 9149610 Compare April 4, 2017 20:38
@seppevs
Copy link
Contributor Author

seppevs commented Apr 4, 2017

@jasnell rebase done

@mscdex
Copy link
Contributor

mscdex commented Apr 5, 2017

@aqrln
Copy link
Contributor

aqrln commented Apr 12, 2017

@seppevs can you please rebase it again?

@jasnell
Copy link
Member

jasnell commented Apr 13, 2017

This needs to be rebased before it can land! :-)

ha! posted then saw @aqrln's comment. guess I should read all the way through before posting ;-)

@refack
Copy link
Contributor

refack commented Apr 13, 2017

Poor guy already rebased one. I'm taking care of this.

refack pushed a commit to refack/node that referenced this pull request Apr 13, 2017
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: nodejs#12031
PR-URL: nodejs#12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
@refack refack closed this Apr 13, 2017
@refack
Copy link
Contributor

refack commented Apr 13, 2017

Landed in 3d2181c

@jasnell jasnell mentioned this pull request May 11, 2017
@gibfahn gibfahn mentioned this pull request Jun 15, 2017
3 tasks
gibfahn pushed a commit that referenced this pull request Jun 18, 2017
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
gibfahn pushed a commit that referenced this pull request Jun 20, 2017
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
MylesBorins pushed a commit that referenced this pull request Jul 11, 2017
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
@MylesBorins MylesBorins mentioned this pull request Jul 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addons Issues and PRs related to native addons. build Issues and PRs related to build files or the CI. test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet