fix handle inheritance for shells

[ZoeyR]

Right now the launched shell with inherit handles from the process that launches it. This causes issues if stdin/stdout/stderr are named pipes. Turning off handle inheritance fixes this issue.

[Tyriar]

This causes issues if stdin/stdout/stderr are named pipes.

Can you elaborate a little on this?

[ZoeyR]

There was a long internal thread with the console team about this, but the short of it is that the PseudoConsole api doesn't handle shell applications with standard input handles that are not "console handles". If the application launching the shell has standard input handles that are named pipes, the shell application will behave oddly and in some circumstances crash.

I noticed this first when testing node-pty in Service Hub for VS 16.3. What would happen is that the stderr from cmd.exe would never get printed out. By setting STARTF_USESTDHANDLES and setting all the std handles to nullptr we fully isolate the shell from the application launching it.

[Tyriar]

@zadjii-msft a review would be appreciated 🙂

[zadjii-msft]

... I have no memory of this thread. This seems reasonable to me, and I do vaguely recall something weird happening if the shell inherits its parent's handle. IIRC this was a bug in the lead up to the original release of the conpty API, and I thought we fixed it for the actual API.

I don't think this will regress anything, so it's probably safe.

[ZoeyR]

@zadjii-msft I think the fix was that if the shell inherits console handles that you close them, but in this case the shell is inheriting named pipes instead.