Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate flaky parallel/test-performance #19197

Closed
apapirovski opened this issue Mar 7, 2018 · 5 comments
Closed

Investigate flaky parallel/test-performance #19197

apapirovski opened this issue Mar 7, 2018 · 5 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.

Comments

@apapirovski
Copy link
Member

The recently changed parallel/test-performance appears to unfortunately be flaky now. Recently failed on aix61-ppc64. More info:

assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)

/cc @TimothyGu
@apapirovski apapirovski added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Mar 7, 2018
@apapirovski
Copy link
Member Author

Failed again recently: https://ci.nodejs.org/job/node-test-commit-linux-containered/2768/nodes=ubuntu1604_sharedlibs_debug_x64/tapResults/

@Trott
Copy link
Member

Trott commented Mar 8, 2018

The timing stuff sure suggests to me that the test ought to be in sequential rather than parallel.

@Trott
Copy link
Member

Trott commented Mar 8, 2018

Oh, yes, definitely can fail due to system load.

$ tools/test.py --repeat 192 -j 96 test/parallel/test-performance.js 
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
=== release test-performance ===                    
Path: parallel/test-performance
assert.js:249
    throw err;
    ^

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(Math.abs(delta) < 1000)

    at checkNodeTiming (/Users/trott/io.js/test/parallel/test-performance.js:118:7)
    at Object.<anonymous> (/Users/trott/io.js/test/parallel/test-performance.js:125:1)
    at Module._compile (module.js:670:30)
    at Object.Module._extensions..js (module.js:681:10)
    at Module.load (module.js:581:32)
    at tryModuleLoad (module.js:521:12)
    at Function.Module._load (module.js:513:3)
    at Function.Module.runMain (module.js:711:10)
    at startup (bootstrap_node.js:223:16)
    at bootstrapNodeJSCore (bootstrap_node.js:562:3)
Command: out/Release/node /Users/trott/io.js/test/parallel/test-performance.js
[00:11|% 100|+ 189|-   3]: Done                     
$

Trott added a commit to Trott/io.js that referenced this issue Mar 8, 2018
@Trott
test: address unreliable test-performance
de2e35a 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: nodejs#19197
@Trott Trott mentioned this issue Mar 8, 2018
Closed
2 tasks
@Trott
Copy link
Member

Trott commented Mar 8, 2018

PR to address this: #19228

@jasnell jasnell closed this as completed in ed8fa81 Mar 9, 2018
MylesBorins pushed a commit that referenced this issue Mar 15, 2018
@Trott @MylesBorins
test: address unreliable test-performance
8daa6fa 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: #19197

PR-URL: #19228
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
MylesBorins pushed a commit that referenced this issue Mar 20, 2018
@Trott @MylesBorins
test: address unreliable test-performance
16ab3b5 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: #19197

PR-URL: #19228
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
MayaLekova pushed a commit to MayaLekova/node that referenced this issue May 8, 2018
@Trott @MayaLekova
test: address unreliable test-performance
f7b7c1d 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: nodejs#19197

PR-URL: nodejs#19228
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
@joyeecheung joyeecheung mentioned this issue Aug 17, 2018
Closed
@joyeecheung
Copy link
Member

This happened again in sequential: nodejs/reliability#14

Reason sequential/test-performance
Type JS_TEST_FAILURE
Failed PR 2 (#22318, #22288)
Appeared test-digitalocean-ubuntu1604_sharedlibs_container-x64-7, test-requireio_andineck-debian9-armv6l_pi1p-1
First CI https://ci.nodejs.org/job/node-test-pull-request/16459/
Last CI https://ci.nodejs.org/job/node-test-pull-request/16462/
Example 
not ok 2289 sequential/test-performance
  ---
  duration_ms: 4.558
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:338
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(Math.abs(delta) < 1000)
    
        at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/test/sequential/test-performance.js:64:7)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/test/sequential/test-performance.js:73:1)
        at Module._compile (internal/modules/cjs/loader.js:689:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
        at Module.load (internal/modules/cjs/loader.js:599:32)
        at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
        at Function.Module._load (internal/modules/cjs/loader.js:530:3)
        at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
        at startup (internal/bootstrap/node.js:266:19)
        at bootstrapNodeJSCore (internal/bootstrap/node.js:611:3)
  ...

@joyeecheung joyeecheung reopened this Aug 17, 2018
jasnell pushed a commit to jasnell/node that referenced this issue Aug 17, 2018
@Trott @jasnell
test: address unreliable test-performance
6539227 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: nodejs#19197

PR-URL: nodejs#19228
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
@BridgeAR BridgeAR mentioned this issue Aug 19, 2018
Closed
4 tasks
targos pushed a commit that referenced this issue Aug 24, 2018
@BridgeAR @targos
test: harden sequential/test-performance
b283819 
1) This adds a better error logging so we are able to address further
   failures easier.
2) It adds a extra epsilon so the test runs into less issues in case
   the machine is under heavy load.
3) The epsilon in increased if the CPU is under heavy load.
4) The total startup epsilon was reduced due to recent startup time
   improvements.

PR-URL: #22404
Fixes: #19197
Refs: nodejs/reliability#14
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
targos pushed a commit that referenced this issue Sep 3, 2018
@BridgeAR @targos
test: harden sequential/test-performance
acfb29c 
1) This adds a better error logging so we are able to address further
   failures easier.
2) It adds a extra epsilon so the test runs into less issues in case
   the machine is under heavy load.
3) The epsilon in increased if the CPU is under heavy load.
4) The total startup epsilon was reduced due to recent startup time
   improvements.

PR-URL: #22404
Fixes: #19197
Refs: nodejs/reliability#14
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
MylesBorins pushed a commit that referenced this issue Sep 6, 2018
@Trott @MylesBorins
test: address unreliable test-performance
8adb0b3 
test-performance can fail due to resource constraints. Move it from
parallel to sequential so it does not compete with other tests for
resources.

Fixes: #19197

Backport-PR-URL: #22380
PR-URL: #19228
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.
Projects
None yet
Milestone
No milestone
3 participants
@apapirovski @Trott @joyeecheung