child_process: fix handleless NODE_HANDLE handling #13235

santigimeno · 2017-05-26T08:30:06Z

It can happen that recvmsg() may return an error on ancillary data
reception when receiving a NODE_HANDLE message (for example
MSG_CTRUNC). This would end up, if the handle type was net.Socket,
on a message event with a non null but invalid sendHandle. Avoid
this by checking the handle validity before performing the
handleConversion.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

child_process

santigimeno · 2017-05-26T08:31:25Z

Let's see if this can fix: #3072. /cc @cjihrig

santigimeno · 2017-05-26T08:40:26Z

BTW thanks to @cjihrig as he came up with a test case and helped debugging the issue.

santigimeno · 2017-05-26T20:44:04Z

Removed the test as it doesn't look reliable at all. If somebody has a better idea for a test, it would be great.

jasnell · 2017-05-29T21:26:36Z

lib/internal/child_process.js

+    // It can happen the handle is not received because of some error on
+    // ancillary data reception such as MSG_CTRUNC. In this case don't try to
+    // perform any conversion.
+    if (!handle)


hmm... I understand the reasoning behind this but I'm wondering if this is the right approach. If the handle is invalid at this point, should we treat it as an error condition?

It's not necessarily an error. For example, connections can be closed by the client while they wait in a queue to be sent from the master to the worker. I think the correct thing to do is advocate that people check their received handles in the worker prior to using them (#13196).

Yes, as far as I can tell there are two cases where the sending a handle ends up with no handle on reception:

As Colin said, if the handle is closed while waiting in the sending queue. This case is detected by the sending process and instead of sending a NODE_HANDLE message, it sends a normal ipc message. In this case, the sendHandle argument of the message listener callback is set to null.

When for some reason, the handle cannot be duped on the receiving process. This is the case this PR tries to fix and I think it makes sense that it behaves in the same way as the previous case (setting sendHandle to null).

I think there's another argument not to treat it as an error. We can send data (the message argument) along with the handle that can be useful on reception even if there's no handle associated.

Ok, I'm convinced :-)

Wouldn't a better approach be to send a "please resend the handle" message to the master instead of NODE_HANDLE_ACK?

It makes sense. Do you have something in mind on how would that work on the sending process?
I think resending the message automatically can lead to problems if the receiving process constantly fails. Maybe using the process.send() callback could be a good idea. So if the message sent is a NODE_HANDLE wait for a NODE_HANDLE_ACK or NODE_HANDLE_NACK to call the callback.

Right, I said "master" when I should have said "parent process."

You are right that unchecked repeated resending is no good. The cluster module could deal with it by sending the handle to another worker after a few turns but that isn't applicable to the child_process module.

I don't have a great solution in mind but as a strawman: try three times and close + log a warning after that? If it happens frequently enough and bothers users enough they will file bug reports about it and we can try to come up with something better.

We could cheat when the ChildProcess instance is managed by the cluster module and do the send-to-another-worker thing. It would be tight coupling but that doesn't bother me personally. Your call.

cjihrig · 2017-06-01T14:11:03Z

lib/internal/child_process.js

@@ -516,6 +516,12 @@ function setupChannel(target, channel) {
    // a message.
    target._send({ cmd: 'NODE_HANDLE_ACK' }, null, true);

+    // It can happen the handle is not received because of some error on


Can you replace "It can happen the handle" with "It is possible that the handle"

santigimeno · 2017-06-04T09:18:34Z

PR updated with @bnoordhuis suggestion. The new commit message would look:

child_process: fix handleless NODE_HANDLE handling

It is possible that `recvmsg()` may return an error on ancillary data
reception when receiving a `NODE_HANDLE` message (for example
`MSG_CTRUNC`). This would end up, if the handle type was `net.Socket`,
on a `message` event with a non null but invalid `sendHandle`. To
improve the situation, send a `NODE_HANDLE_NACK` that'll cause the
sending process to retransmit the message again. In case the same
message is retransmitted 3 times without success, close the handle and
print a warning.

BridgeAR · 2017-06-04T13:29:30Z

lib/internal/child_process.js

+        target._send(target._pendingMessage.message,
+                     target._pendingMessage.handle,
+                     target._pendingMessage.options,
+                     target._pendingMessage. callback);


Erroneous whitespace

BridgeAR · 2017-06-04T13:31:23Z

lib/internal/child_process.js

+            process.emitWarning('Handle didn\'t reach the receiving process ' +
+                                'correctly', 'SentHandleNotReceivedWarning');
+          }
+        }
      }


This can be simplified:

if (target._pendingMessage) { if (message.cmd === 'NODE_HANDLE_ACK') { closePendingHandle(target); } else if (target._pendingMessage.retransmissions++ === MAX_HANDLE_RETRANSMISSIONS) { // If the handle didn't reach the receiving process correctly retry up // to 3 times. If it still fails, close it and print a warning. closePendingHandle(target); process.emitWarning('Handle didn\'t reach the receiving process ' + 'correctly', 'SentHandleNotReceivedWarning'); } }

bnoordhuis · 2017-06-06T09:55:09Z

lib/internal/child_process.js

+            handle: handle,
+            options: options,
+            retransmissions: 0
+          };


You could shorten this to:

target._pendingMessage = { callback, message, handle, options, retransmissions: 0 };

bnoordhuis · 2017-06-06T09:55:47Z

lib/internal/child_process.js

-        target._pendingHandle.close();
-        target._pendingHandle = null;
+    if ((message.cmd === 'NODE_HANDLE_ACK') ||
+        (message.cmd === 'NODE_HANDLE_NACK')) {


Superfluous parens.

santigimeno · 2017-06-07T21:39:27Z

PR updated with comments from @BridgeAR and @bnoordhuis addressed. PTAL. Thanks!

santigimeno · 2017-06-23T10:41:54Z

ping @bnoordhuis @cjihrig :)

bnoordhuis

Sorry, I remember looking at this last week but I must've forgotten to actually submit my review.

LGTM and sorry for the delay!

bnoordhuis · 2017-06-23T13:37:56Z

lib/internal/child_process.js

+        target._send(target._pendingMessage.message,
+                     target._pendingMessage.handle,
+                     target._pendingMessage.options,
+                     target._pendingMessage.callback);


Can you put braces around the body for legibility?

cjihrig

Tests would be great, but LGTM.

cjihrig · 2017-06-23T14:00:36Z

lib/internal/child_process.js

+        } else if (target._pendingMessage.retransmissions++ ===
+                   MAX_HANDLE_RETRANSMISSIONS) {
+          closePendingHandle(target);
+          process.emitWarning('Handle didn\'t reach the receiving process ' +


Tiny nit: could you change didn't to did not.

santigimeno · 2017-06-23T19:53:02Z

All nits addressed.

Tests would be great

I agree, I tried something that worked for me locally but wasn't reliable at all. If anyone has suggestions would be great.

CI: https://ci.nodejs.org/job/node-test-pull-request/8813/

santigimeno · 2017-06-27T20:50:59Z

CI: https://ci.nodejs.org/job/node-test-pull-request/8845/

santigimeno · 2017-07-01T15:32:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/8918/

It is possible that `recvmsg()` may return an error on ancillary data reception when receiving a `NODE_HANDLE` message (for example `MSG_CTRUNC`). This would end up, if the handle type was `net.Socket`, on a `message` event with a non null but invalid `sendHandle`. To improve the situation, send a `NODE_HANDLE_NACK` that'll cause the sending process to retransmit the message again. In case the same message is retransmitted 3 times without success, close the handle and print a warning. PR-URL: nodejs#13235 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

santigimeno · 2017-07-01T16:16:38Z

Landed in 71ca122. Thanks!

It is possible that `recvmsg()` may return an error on ancillary data reception when receiving a `NODE_HANDLE` message (for example `MSG_CTRUNC`). This would end up, if the handle type was `net.Socket`, on a `message` event with a non null but invalid `sendHandle`. To improve the situation, send a `NODE_HANDLE_NACK` that'll cause the sending process to retransmit the message again. In case the same message is retransmitted 3 times without success, close the handle and print a warning. PR-URL: nodejs#13235 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

It is possible that `recvmsg()` may return an error on ancillary data reception when receiving a `NODE_HANDLE` message (for example `MSG_CTRUNC`). This would end up, if the handle type was `net.Socket`, on a `message` event with a non null but invalid `sendHandle`. To improve the situation, send a `NODE_HANDLE_NACK` that'll cause the sending process to retransmit the message again. In case the same message is retransmitted 3 times without success, close the handle and print a warning. PR-URL: #13235 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

MylesBorins · 2017-08-15T00:09:57Z

Landed on v6.x-staging in 010857a18f, let me know if it should be backed out

It is possible that `recvmsg()` may return an error on ancillary data reception when receiving a `NODE_HANDLE` message (for example `MSG_CTRUNC`). This would end up, if the handle type was `net.Socket`, on a `message` event with a non null but invalid `sendHandle`. To improve the situation, send a `NODE_HANDLE_NACK` that'll cause the sending process to retransmit the message again. In case the same message is retransmitted 3 times without success, close the handle and print a warning. Fixes: #9706 PR-URL: #13235 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

nodejs-github-bot added the child_process Issues and PRs related to the child_process subsystem. label May 26, 2017

santigimeno force-pushed the cluster_fix branch from 6cbad85 to ce9968c Compare May 26, 2017 08:32

santigimeno mentioned this pull request May 26, 2017

Cluster crash in main process, callback undefined when handling internal messages #3072

Closed

jasnell reviewed May 29, 2017

View reviewed changes

cjihrig approved these changes Jun 1, 2017

View reviewed changes

santigimeno force-pushed the cluster_fix branch 2 times, most recently from a884a07 to ed6e9fb Compare June 4, 2017 09:15

BridgeAR requested changes Jun 4, 2017

View reviewed changes

bnoordhuis requested changes Jun 6, 2017

View reviewed changes

santigimeno force-pushed the cluster_fix branch 2 times, most recently from b4815fb to 1dda284 Compare June 7, 2017 21:38

bnoordhuis approved these changes Jun 23, 2017

View reviewed changes

cjihrig approved these changes Jun 23, 2017

View reviewed changes

santigimeno force-pushed the cluster_fix branch from 1dda284 to 4237b7f Compare June 23, 2017 19:48

santigimeno force-pushed the cluster_fix branch from 4237b7f to 7f5518f Compare June 27, 2017 20:41

santigimeno force-pushed the cluster_fix branch from 7f5518f to 0a9529c Compare June 27, 2017 20:49

santigimeno force-pushed the cluster_fix branch from 0a9529c to 19aa4c1 Compare July 1, 2017 15:30

santigimeno force-pushed the cluster_fix branch from 19aa4c1 to 71ca122 Compare July 1, 2017 16:14

santigimeno merged commit 71ca122 into nodejs:master Jul 1, 2017

addaleax mentioned this pull request Jul 3, 2017

v8.2.0 proposal #13744

Merged

bnoordhuis mentioned this pull request Jul 17, 2017

IPC channel stops delivering messages to cluster workers #9706

Closed

MylesBorins added the land-on-v6.x label Aug 15, 2017

MylesBorins mentioned this pull request Aug 16, 2017

v6.11.3-proposal #14852

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

child_process: fix handleless NODE_HANDLE handling #13235

child_process: fix handleless NODE_HANDLE handling #13235

santigimeno commented May 26, 2017

santigimeno commented May 26, 2017

santigimeno commented May 26, 2017

santigimeno commented May 26, 2017

jasnell May 29, 2017

cjihrig May 29, 2017

santigimeno May 30, 2017

jasnell May 30, 2017

bnoordhuis Jun 1, 2017

santigimeno Jun 1, 2017 •

edited

Loading

bnoordhuis Jun 1, 2017

cjihrig Jun 1, 2017

santigimeno commented Jun 4, 2017

BridgeAR Jun 4, 2017

BridgeAR Jun 4, 2017

bnoordhuis Jun 6, 2017

bnoordhuis Jun 6, 2017

santigimeno commented Jun 7, 2017

santigimeno commented Jun 23, 2017

bnoordhuis left a comment

bnoordhuis Jun 23, 2017

cjihrig left a comment

cjihrig Jun 23, 2017

santigimeno commented Jun 23, 2017 •

edited

Loading

santigimeno commented Jun 27, 2017

santigimeno commented Jul 1, 2017

santigimeno commented Jul 1, 2017

MylesBorins commented Aug 15, 2017

child_process: fix handleless NODE_HANDLE handling #13235

child_process: fix handleless NODE_HANDLE handling #13235

Conversation

santigimeno commented May 26, 2017

Checklist

Affected core subsystem(s)

santigimeno commented May 26, 2017

santigimeno commented May 26, 2017

santigimeno commented May 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santigimeno Jun 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santigimeno commented Jun 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santigimeno commented Jun 7, 2017

santigimeno commented Jun 23, 2017

bnoordhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjihrig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santigimeno commented Jun 23, 2017 • edited Loading

santigimeno commented Jun 27, 2017

santigimeno commented Jul 1, 2017

santigimeno commented Jul 1, 2017

MylesBorins commented Aug 15, 2017

santigimeno Jun 1, 2017 •

edited

Loading

santigimeno commented Jun 23, 2017 •

edited

Loading