test: add a place for reproducing known bugs #5528

cjihrig · 2016-03-02T17:56:46Z

Pull Request check-list

Please make sure to review and check all of these items:

Does make -j8 test (UNIX) or vcbuild test nosign (Windows) pass with
this change (including linting)?
Is the commit message formatted according to [CONTRIBUTING.md][0]?
If this change fixes a bug (or a performance problem), is a regression
test (or a benchmark) included?
Is a documentation update included (if this change modifies
existing APIs, or introduces new ones)?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Affected core subsystem(s)

test

Description of change

This PR is a proof of concept related to nodejs/testing#18. The point is to create a place to track the state of current known bugs, with a single command to verify that they still exist (the issue tracker has proven to be a difficult way to manage this).

There are currently a few hundred open issues, so I only bothered to port one bug reproduction. I also need to add make test-bugs to vcbuild for Windows. If this can get in, I'll work to create more reproductions.

jasnell · 2016-03-02T18:13:10Z

Oh nice... I have some concerns about maintainability of this but generally LGTM

geek · 2016-03-02T18:24:16Z

Makefile

@@ -192,6 +192,9 @@ test-internet: all
 test-debugger: all
 	$(PYTHON) tools/test.py debugger

+test-bugs: all


instead of bugs, maybe name the folder and make step: broken

To me, broken makes it sound like a place where we put tests that are broken, rather than a place where we have good tests that are failing because of bugs in the main code base. I prefer test-bugs or even test-known-issues.

I like test-known-issues. @geek are you cool with that?

@cjihrig agreed

geek · 2016-03-03T15:43:13Z

LGTM

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail.

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: nodejs#548

cjihrig · 2016-03-03T17:00:02Z

Added the vcbuild piece.

@Trott care to sign off on this?

EDIT: Also renamed bugs to known-issues.

Trott · 2016-03-03T17:25:24Z

LGTM if CI is green.

cjihrig · 2016-03-03T17:40:04Z

CI: https://ci.nodejs.org/job/node-test-pull-request/1835/

cjihrig · 2016-03-03T20:24:34Z

Only failure is the ARM machine that appeared to be offline for several a number of CI runs.

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

cjihrig · 2016-03-04T13:55:15Z

Thanks for the reviews. Landed in 32e1f9d and e38e2a6.

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

MylesBorins · 2016-03-10T21:47:19Z

This seems like the kind of thing useful to backport. @cjihrig do you agree?

cjihrig · 2016-03-11T02:17:55Z

I think it would be useful, but could be tricky to manage.

jasnell · 2016-03-11T02:23:20Z

hmm... not so sure about this one. @nodejs/lts ... thoughts?

Trott · 2016-03-11T04:15:28Z

I would think you would want to put this in LTS. You're going to want the bug fixes for these things in LTS and not having the tests there just increases the friction when you go to land the bug fixes. It's just a directory of tests that don't get run.

jasnell · 2016-03-11T04:31:57Z

Agreed in principle, but there will be a bit of a maintenance burden as things get fixed in one branch but not others. I do see how it's useful tho and definitely wouldn't -1 it

rvagg · 2016-03-14T02:09:13Z

I think it's a +1 from me, reflecting on the difficulty we have with the 0.10 and 0.12 test suite, what we have on master is only going to get better and having this in v4 will help it as we get toward EOL to understand more about the delta and the state of the code. I also imagine that there will be a section of users who will be interested in this kind of information on v4 when contributing because that's the only branch they care about (we see this already on 0.10 and 0.12).

MylesBorins · 2016-03-17T19:07:13Z

so this does not land cleanly on v4.x-staging @cjihrig would you be able to send a PR directly against v4.x-staging with the conflicts sorted out? They were primarily in test.py

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 PR-URL: nodejs#5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Conflicts: tools/test.py

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: nodejs#548 PR-URL: nodejs#5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

cjihrig · 2016-03-18T17:12:28Z

Backport in #5785

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Conflicts: tools/test.py

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Conflicts: tools/test.py

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail. Refs: nodejs/testing#18 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Conflicts: tools/test.py

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: #548 Backport-URL: #5785 PR-URL: #5528 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Wyatt Preul <wpreul@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

jasnell added the test Issues and PRs related to the tests. label Mar 2, 2016

geek reviewed Mar 2, 2016
View reviewed changes

cjihrig force-pushed the bugs branch from 2b7f635 to d515a4a Compare March 3, 2016 16:39

cjihrig added 2 commits March 3, 2016 11:56

tools: support testing known issues

29ba2eb

This commit adds a known_issues directory to the test directory for scripts that reproduce known bugs. Since these scripts are expected to fail, it also adds a --expect-fail flag to test.py which reports tests as successful when they fail.

test: bug repro for vm function redefinition

48e7009

This commit adds a failing test case for the vm module. Currently, if runInContext() defines a function, and a later call to runInContext() redefines the same function, the original function is not overwritten. Refs: nodejs#548

cjihrig force-pushed the bugs branch from d515a4a to 48e7009 Compare March 3, 2016 16:56

cjihrig closed this Mar 4, 2016

cjihrig deleted the bugs branch March 4, 2016 13:55

cjihrig mentioned this pull request Mar 4, 2016

Add a place for reproducing current bugs nodejs/testing#18

Closed

Trott mentioned this pull request Mar 4, 2016

doc: document directories in test directory #5557

Closed

4 tasks

Fishrock123 mentioned this pull request Mar 7, 2016

v5.8.0 proposal #5559

Merged

MylesBorins mentioned this pull request Mar 10, 2016

Audit commits not found on v4.x #5647

Closed

MylesBorins added the lts-watch-v4.x label Mar 10, 2016

cjihrig mentioned this pull request Mar 18, 2016

Backport of 5528 #5785

Closed

MylesBorins added dont-land-on-v4.x and removed lts-watch-v4.x labels Mar 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add a place for reproducing known bugs #5528

test: add a place for reproducing known bugs #5528

cjihrig commented Mar 2, 2016

jasnell commented Mar 2, 2016

geek Mar 2, 2016

Trott Mar 2, 2016

cjihrig Mar 2, 2016

geek Mar 3, 2016

geek commented Mar 3, 2016

cjihrig commented Mar 3, 2016

Trott commented Mar 3, 2016

cjihrig commented Mar 3, 2016

cjihrig commented Mar 3, 2016

cjihrig commented Mar 4, 2016

MylesBorins commented Mar 10, 2016

cjihrig commented Mar 11, 2016

jasnell commented Mar 11, 2016

Trott commented Mar 11, 2016

jasnell commented Mar 11, 2016

rvagg commented Mar 14, 2016

MylesBorins commented Mar 17, 2016

cjihrig commented Mar 18, 2016

test: add a place for reproducing known bugs #5528

test: add a place for reproducing known bugs #5528

Conversation

cjihrig commented Mar 2, 2016

Pull Request check-list

Affected core subsystem(s)

Description of change

jasnell commented Mar 2, 2016

geek Mar 2, 2016

Choose a reason for hiding this comment

Trott Mar 2, 2016

Choose a reason for hiding this comment

cjihrig Mar 2, 2016

Choose a reason for hiding this comment

geek Mar 3, 2016

Choose a reason for hiding this comment

geek commented Mar 3, 2016

cjihrig commented Mar 3, 2016

Trott commented Mar 3, 2016

cjihrig commented Mar 3, 2016

cjihrig commented Mar 3, 2016

cjihrig commented Mar 4, 2016

MylesBorins commented Mar 10, 2016

cjihrig commented Mar 11, 2016

jasnell commented Mar 11, 2016

Trott commented Mar 11, 2016

jasnell commented Mar 11, 2016

rvagg commented Mar 14, 2016

MylesBorins commented Mar 17, 2016

cjihrig commented Mar 18, 2016