Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix flaky sequential/test-child-process-pass-fd on fedora 24 #11041

Closed
Trott opened this issue Jan 27, 2017 · 20 comments
Closed

fix flaky sequential/test-child-process-pass-fd on fedora 24 #11041

Trott opened this issue Jan 27, 2017 · 20 comments
Labels
child_process Issues and PRs related to the child_process subsystem. test Issues and PRs related to the tests.

Comments

@Trott
Copy link
Member

Trott commented Jan 27, 2017 •
edited
Loading

Example failure:

https://ci.nodejs.org/job/node-test-commit-linux/7552/nodes=fedora24/console

  duration_ms: 1.113
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
  ...

/cc @santigimeno
@Trott Trott added child_process Issues and PRs related to the child_process subsystem. test Issues and PRs related to the tests. labels Jan 27, 2017
@Trott
Copy link
Member Author

Trott commented Jan 27, 2017 •
edited
Loading

Another example failure:

https://ci.nodejs.org/job/node-test-commit-linux/7544/nodes=fedora24/console

not ok 1318 sequential/test-child-process-pass-fd
  ---
  duration_ms: 1.112
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)

@Trott
Copy link
Member Author

Trott commented Jan 27, 2017

Small data set so far, but these failures are both on test-rackspace-fedora24-x64-1 while test-digitalocean-fedora24-x64-1 always seems to pass. @nodejs/build

@Trott
Copy link
Member Author

Trott commented Jan 27, 2017

Another one: https://ci.nodejs.org/job/node-test-commit-linux/7552/nodes=fedora24/console

And once again, it's on test-rackspace-fedora24-x64-1.

@Trott Trott mentioned this issue Jan 27, 2017
Closed
2 tasks
@Trott Trott changed the title fix flaky equential/test-child-process-pass-fd on fedora 24 fix flaky sequential/test-child-process-pass-fd on fedora 24 Jan 27, 2017
@santigimeno
Copy link
Member

santigimeno commented Jan 28, 2017 •
edited
Loading

The EAGAIN error on spawn suggests that the problem could be that a limit on number of processes running is reached. See http://man7.org/linux/man-pages/man2/fork.2.html#ERRORS. So it could be a configuration problem in the CI bot.

@Trott
Copy link
Member Author

Trott commented Jan 28, 2017

The EAGAIN error on spawn suggests that the problem could be that a limit on number of processes running is reached.

I wonder if it's something like that, but not quite exactly that.

[root@test-rackspace-fedora24-x64-1 ~]# ulimit -u
31822
[root@test-rackspace-fedora24-x64-1 ~]# ps -ax | wc -l
145
[root@test-rackspace-fedora24-x64-1 ~]#

It's hard for me to imagine that we're actually topping out over 31822 processes.

@santigimeno
Copy link
Member

@Trott have you checked the other limits defined in http://man7.org/linux/man-pages/man2/fork.2.html#ERRORS (max # of threads and max # of pids )?

@Trott
Copy link
Member Author

Trott commented Jan 29, 2017

@santigimeno

[root@test-rackspace-fedora24-x64-1 ~]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31822
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31822
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@test-rackspace-fedora24-x64-1 ~]# cat /proc/sys/kernel/threads-max
63645
[root@test-rackspace-fedora24-x64-1 ~]#

@Trott
Copy link
Member Author

Trott commented Jan 30, 2017

Also, for future reference: Still happening.

https://ci.nodejs.org/job/node-test-commit-linux/7574/nodes=fedora24/console

not ok 1326 sequential/test-child-process-pass-fd
  ---
  duration_ms: 1.112
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)

@Trott
Copy link
Member Author

Trott commented Jan 31, 2017

Still going on: https://ci.nodejs.org/job/node-test-commit-linux/7585/nodes=fedora24/console

not ok 1329 sequential/test-child-process-pass-fd
  ---
  duration_ms: 1.112
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)

@joyeecheung joyeecheung mentioned this issue Jan 31, 2017
Closed
3 tasks
@Trott Trott mentioned this issue Jan 31, 2017
Closed
2 tasks
@Trott
Copy link
Member Author

Trott commented Feb 1, 2017 •
edited
Loading

Still happening. And only on test-rackspace-fedora24-x64-1.

https://ci.nodejs.org/job/node-test-commit-linux/7632/nodes=fedora24/console

not ok 1337 sequential/test-child-process-pass-fd
  ---
  duration_ms: 1.112
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)

@Trott
Copy link
Member Author

Trott commented Feb 1, 2017

@nodejs/testing maybe? I mean, it seems more of a @nodejs/build thing but this needs more eyes, I think...

@Trott
Copy link
Member Author

Trott commented Feb 1, 2017

Maybe test-rackspace-fedora24-x64-1 just needs a reboot? It's been up for 155 days. Which yeah yeah I know it's UNIX-like it should be able to stay up forever etc etc. But if no one else has a better idea?

@Trott
Copy link
Member Author

Trott commented Feb 2, 2017

Again today:

https://ci.nodejs.org/job/node-test-commit-linux/7654/nodes=fedora24/console

not ok 1337 sequential/test-child-process-pass-fd
  ---
  duration_ms: 1.112
  severity: fail
  stack: |-
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/out/Release/node EAGAIN
        at exports._errnoException (util.js:1023:11)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
        at onErrorNT (internal/child_process.js:359:16)
        at _combinedTickCallback (internal/process/next_tick.js:74:11)
        at process._tickCallback (internal/process/next_tick.js:98:9)
        at Module.runMain (module.js:607:11)
        at run (bootstrap_node.js:418:7)
        at startup (bootstrap_node.js:139:9)
        at bootstrap_node.js:533:3
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)
    events.js:161
          throw er; // Unhandled 'error' event
          ^
    
    Error: channel closed
        at process.target.send (internal/child_process.js:553:16)
        at Socket.socketConnected (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora24/test/sequential/test-child-process-pass-fd.js:39:15)
        at Object.onceWrapper (events.js:291:19)
        at emitNone (events.js:86:13)
        at Socket.emit (events.js:186:7)
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1077:10)

@Trott
Copy link
Member Author

Trott commented Feb 2, 2017

The machine is getting a reboot now thanks to @joaocgreis.

@Trott
Copy link
Member Author

Trott commented Feb 3, 2017

Reboot didn't help. Test has failed at least twice since the reboot.

Example: https://ci.nodejs.org/job/node-test-commit-linux/7680/nodes=fedora24/console

@Trott
Copy link
Member Author

Trott commented Feb 3, 2017 •
edited
Loading

Since it's only failing on the Rackspace host and not on the Digital Ocean host, maybe comparing some settings might be useful?

test-rackspace-fedora24-x64-1 (flaky)

[root@test-rackspace-fedora24-x64-1 ~]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31812
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31812
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@test-rackspace-fedora24-x64-1 ~]# cat /proc/sys/kernel/threads-max
63625
[root@test-rackspace-fedora24-x64-1 ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5200.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 2
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5484.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 4
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5512.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 6
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5488.25
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 8
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 8
initial apicid	: 8
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5486.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 10
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 10
initial apicid	: 10
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5488.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 12
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 12
initial apicid	: 12
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5480.60
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2599.949
cache size	: 20480 KB
physical id	: 14
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 14
initial apicid	: 14
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm
bugs		:
bogomips	: 5484.36
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

[root@test-rackspace-fedora24-x64-1 ~]# cat /proc/meminfo
MemTotal:        8169820 kB
MemFree:         3944576 kB
MemAvailable:    7563672 kB
Buffers:          187556 kB
Cached:          3332088 kB
SwapCached:            0 kB
Active:          1759100 kB
Inactive:        1975948 kB
Active(anon):     215712 kB
Inactive(anon):      132 kB
Active(file):    1543388 kB
Inactive(file):  1975816 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1224 kB
Writeback:             0 kB
AnonPages:        215420 kB
Mapped:            96352 kB
Shmem:               432 kB
Slab:             444232 kB
SReclaimable:     402676 kB
SUnreclaim:        41556 kB
KernelStack:        3024 kB
PageTables:         3804 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4084908 kB
Committed_AS:     337884 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      118784 kB
DirectMap2M:     8265728 kB
[root@test-rackspace-fedora24-x64-1 ~]#

test-digitalocean-fedora24-x64-1 (not flaky)

[root@test-digitalocean-fedora24-x64-1 ~]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7926
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7926
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@test-digitalocean-fedora24-x64-1 ~]# cat /proc/sys/kernel/threads-max
15853
[root@test-digitalocean-fedora24-x64-1 ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
stepping	: 4
microcode	: 0x1
cpu MHz		: 2399.998
cache size	: 15360 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
bugs		:
bogomips	: 4799.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
stepping	: 4
microcode	: 0x1
cpu MHz		: 2399.998
cache size	: 15360 KB
physical id	: 1
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm vnmi ept fsgsbase tsc_adjust smep erms xsaveopt arat
bugs		:
bogomips	: 4799.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

[root@test-digitalocean-fedora24-x64-1 ~]# cat /proc/meminfo
MemTotal:        2048952 kB
MemFree:          172656 kB
MemAvailable:    1676484 kB
Buffers:           94648 kB
Cached:          1303308 kB
SwapCached:            0 kB
Active:           861424 kB
Inactive:         669076 kB
Active(anon):     132832 kB
Inactive(anon):      100 kB
Active(file):     728592 kB
Inactive(file):   668976 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 8 kB
Writeback:             0 kB
AnonPages:        132504 kB
Mapped:            49884 kB
Shmem:               388 kB
Slab:             326768 kB
SReclaimable:     294320 kB
SUnreclaim:        32448 kB
KernelStack:        1536 kB
PageTables:         2684 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1024476 kB
Committed_AS:     216232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      864248 kB
DirectMap2M:     1232896 kB
DirectMap1G:           0 kB
[root@test-digitalocean-fedora24-x64-1 ~]#

@Trott
Copy link
Member Author

Trott commented Feb 3, 2017

In most (all?) respects, the host showing flakiness is provisioned with more resources than the host that is not being flaky.
¯\(ツ)/¯

@Trott
Copy link
Member Author

Trott commented Feb 3, 2017 •
edited
Loading

Output of lsmod is pretty different on the two hosts. Many more modules loaded on the machine exhibiting flakiness. No idea if that's relevant. Kind of fishing for stuff right now.

test-rackspace-fedora24-x64-1 (flaky)

[root@test-rackspace-fedora24-x64-1 ~]# lsmod
Module                  Size  Used by
ip6t_rpfilter          16384  1
ip6t_REJECT            16384  2
nf_reject_ipv6         16384  1 ip6t_REJECT
xt_conntrack           16384  13
ip_set                 36864  0
nfnetlink              16384  1 ip_set
ebtable_broute         16384  1
bridge                131072  1 ebtable_broute
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
ebtable_nat            16384  1
ip6table_mangle        16384  1
ip6table_security      16384  1
ip6table_raw           16384  1
ip6table_nat           16384  1
nf_conntrack_ipv6      20480  8
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6
nf_nat_ipv6            16384  1 ip6table_nat
iptable_mangle         16384  1
iptable_security       16384  1
iptable_raw            16384  1
iptable_nat            16384  1
nf_conntrack_ipv4      16384  7
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 28672  2 nf_nat_ipv6,nf_nat_ipv4
nf_conntrack          106496  6 nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
ebtable_filter         16384  1
ebtables               36864  3 ebtable_filter,ebtable_nat,ebtable_broute
ip6table_filter        16384  1
ip6_tables             28672  5 ip6table_mangle,ip6table_filter,ip6table_security,ip6table_raw,ip6table_nat
xenfs                  16384  1
xen_privcmd            16384  1 xenfs
intel_rapl             20480  0
sb_edac                24576  0
edac_core              53248  1 sb_edac
snd_pcsp               16384  0
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
snd_pcm               118784  1 snd_pcsp
cirrus                 24576  1
ghash_clmulni_intel    16384  0
ttm                    94208  1 cirrus
ppdev                  20480  0
drm_kms_helper        151552  1 cirrus
xen_netfront           28672  0
snd_timer              32768  1 snd_pcm
joydev                 20480  0
drm                   344064  4 cirrus,ttm,drm_kms_helper
snd                    86016  3 snd_pcsp,snd_timer,snd_pcm
intel_rapl_perf        16384  0
soundcore              16384  1 snd
i2c_piix4              24576  0
parport_pc             28672  0
parport                49152  2 parport_pc,ppdev
acpi_cpufreq           20480  0
fjes                   28672  0
tpm_tis                16384  0
tpm_tis_core           20480  1 tpm_tis
tpm                    45056  2 tpm_tis,tpm_tis_core
xen_blkfront           40960  2
crc32c_intel           24576  0
serio_raw              16384  0
ata_generic            16384  0
pata_acpi              16384  0
[root@test-rackspace-fedora24-x64-1 ~]#

test-digitalocean-fedora24-x64-1 (not flaky)

[root@test-digitalocean-fedora24-x64-1 ~]# lsmod
Module                  Size  Used by
kvm_intel             188416  0
kvm                   561152  1 kvm_intel
irqbypass              16384  1 kvm
ppdev                  20480  0
crct10dif_pclmul       16384  0
parport_pc             28672  0
crc32_pclmul           16384  0
joydev                 20480  0
virtio_net             32768  0
ghash_clmulni_intel    16384  0
virtio_balloon         16384  0
parport                49152  2 ppdev,parport_pc
i2c_piix4              24576  0
acpi_cpufreq           20480  0
tpm_tis                20480  0
tpm                    40960  1 tpm_tis
cirrus                 28672  1
drm_kms_helper        143360  1 cirrus
ttm                    90112  1 cirrus
drm                   339968  4 ttm,drm_kms_helper,cirrus
virtio_scsi            20480  0
virtio_blk             20480  2
crc32c_intel           24576  0
serio_raw              16384  0
ata_generic            16384  0
virtio_pci             24576  0
pata_acpi              16384  0
virtio_ring            24576  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_scsi
virtio                 16384  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_scsi
[root@test-digitalocean-fedora24-x64-1 ~]#

@Trott
Copy link
Member Author

Trott commented Feb 3, 2017

Maybe @nodejs/diagnostics would be good to loop in here? (And if not, sorry!)

@targos targos mentioned this issue Feb 7, 2017
Merged
2 tasks
@Trott
Copy link
Member Author

Trott commented Jul 12, 2017

This stopped happening several months ago although now we're seeing a different failure (timeout) after snapshots were removed from Node.js. Will fix that separately, but closing this.

@Trott Trott closed this as completed Jul 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
child_process Issues and PRs related to the child_process subsystem. test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Trott @santigimeno