Add ability to spawn Windows process with Proc Thread Attributes #88193

TyPR124 · 2021-08-20T20:54:17Z

This adds a Windows-specific extension method on Command that allows users to include proc thread attributes. The test uses the PROC_THREAD_ATTRIBUTE_PARENT_PROCESS attribute to spawn a process with a custom parent process, and then verifies the child's parent matches the expected parent. That seemed to me the easiest way to test.

Without this, it is not possible to implement anything requiring these attributes without re-implementing Command (and friends) from scratch as a separate crate.

In particular, I would like to use this to make use of the Windows ConPTY API. I've already succesfully implemented this in my own library, but don't like the thought of having to maintain my own copy of what is already in the standard library, potentially missing any future bug fixes or security updates.

This addition requires a minimum of Windows Vista or Server 2008, which as far as I understand should now be acceptable to add to the standard library since XP support is not required any longer.

The exposed API is pretty much the bare minimum to make this work, but it may be possible to improve the API in the future.

I can open a tracking issue and update the PR with that in a bit.

rust-highfive · 2021-08-20T20:54:19Z

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

ChrisDenton · 2021-08-21T12:39:46Z

I would definitely love to see more of CreateProcessW exposed to users.

The exposed API is pretty much the bare minimum to make this work, but it may be possible to improve the API in the future.

Hm, it does feel like Command should ideally take ownership of the value somehow.

Other than that it would be good to have safe (or at least safer) APIs for common operations. E.g. setting the parent process, setting exploit mitigations and restricting inherited handles.

library/std/src/process/tests.rs

ChrisDenton · 2021-08-21T15:59:11Z

library/std/src/sys/windows/process.rs

+    let count = attributes.len() as u32;
+    // First call to get required size. Error is expected.
+    unsafe { c::InitializeProcThreadAttributeList(ptr::null_mut(), count, 0, &mut buffer_size) };
+    let mut buf = Pin::new(crate::vec::from_elem(0u8, buffer_size as usize));


Does this need to be Pin? Surely the Vec itself can be moved around freely?

The Pin prevents us from calling (for example) vec.push() or doing anything that may re-allocate (and move) the underlying data. I'm not 100% sure it is necessary to not move the data, as I don't know the internal workings of these Initialize/Update/Delete functions, but it seemed the right choice to me since there is at least no good reason to move or grow the data after that point.

The Pin doesn't prevent us from moving the Vec itself (after all, we must move it out of this function to return it), just the data it points to.

[u8] is Unpin, so Pin<Vec<u8>> so both Pin::into_inner and DerefMut are implemented allowing you to move the inner bytes anyway.

https://doc.rust-lang.org/stable/std/pin/struct.Pin.html#method.into_inner

I think perhaps a boxed slice would be enough to prevent accidentally reallocating? Or it might be a good idea to fully contain the owned data. So have an API like:

struct ProcThreadAttributeList(Vec<u8>); // or Box<[T]> impl ProcThreadAttributeList { fn new(attribute_count: u32) -> io::Result<Self> { ... } unsafe fn update(&mut self, attribute: usize, value_ptr: *mut c_void, value_size: u32) -> io::Result<()> { ... } } impl Drop for ProcThreadAttributeList { ... }

That way there is no reason to touch the inner data.

@ChrisDenton I could do that, just seemed like overkill to me when I was writing it.

@bjorn3 You could copy the bytes, or overwrite them, but you can't move them. let mut v = Pin::new(vec![]); v.push(()); does not compile. Unless I'm just being really dumb right now. You are right about Pin::into_inner but calling that explicitly indicates it is now okay to move the underlying data (which it might be anyway, but, again, I'm not certain).

TyPR124 · 2021-08-21T18:40:51Z

Hm, it does feel like Command should ideally take ownership of the value somehow.

I think I agree. I am considering making the method generic over a T: Sized + Any, in which case we could grab the size ourselves and store the T in some Box<dyn Any>, making it slightly easier on the user. A couple concerns I have about that are:

Do we really need to allocate for every attribute?
Do we really need ownership of the generic type? Are there any situations where the user might need to retain ownership?
- I'm guessing that any value passed in here would always be Copy so this shouldn't be an issue, but I'm not certain of that.

That said, the more I think about it, the more I tend to think taking ownership would be the right choice. But I'd like to hear if others have any feedback here first.

Other than that it would be good to have safe (or at least safer) APIs

I'm skeptical this could ever be made fully safe, however it will certainly be possible (and I'd say, encouraged) to build safe APIs on top of this.

The reason I'm skeptical basically comes down to the variety of different attributes a user could pass in and the fact that MS could add new attributes at any time. For instance, if MS added something which allowed the user to have some of the parent process's memory mapped into the child's address space, that would immediately preclude this from ever being truly safe.

My hope is that this extension method, it will enable authors of 3rd party libraries to make safe APIs via their own extension traits, even if those safe APIs never make it into the standard library.

ChrisDenton · 2021-08-21T19:38:07Z

Yes sorry, that last part was meant as a separate thought. I was meaning it would be possible to have a safe API for common operations such as a parent_handle or exploit_mitigation method. I agree that arbitrary ones would be impossible because we can't know what future attributes will be added and a lower-level method will always be useful for that.

bjorn3 · 2021-08-22T06:52:14Z

Can you please use rebase instead of merge?

Mark-Simulacrum · 2021-08-24T19:02:49Z

r? @yaahc for T-libs-api review

I'm not familiar enough with Windows to review the impl either.

TyPR124 · 2021-08-24T22:58:24Z

I will try to fix the git history sometime this week (or this weekend) and also I still need to open a tracking issue.

TyPR124 · 2021-08-25T02:51:40Z

While considering what it might look like to make the method generic over the value type, I noticed that there is a difference in object safety between unix and windows CommandExt. Is this something to be concerned about? Windows is object safe, unix is not.

#[cfg(test)]
fn test_object_safety() {
    #[cfg(windows)]
    use std::os::windows::process::CommandExt;
    #[cfg(unix)]
    use std::os::unix::process::CommandExt;
    fn takes_dyn_command_ext(_: &dyn CommandExt) {}
    takes_dyn_command_ext(&std::process::Command::new(""));
}

Currently, if I were to add a generic method for Windows (without Self: Sized), it silently breaks object safety.

Looking at methods on the unix side, it appears to me the trait was originally object-safe in 1.0, but that was broken in 1.15 with before_exec, and then later pre_exec and arg0.

TyPR124 · 2021-08-30T15:02:20Z

Can you please use rebase instead of merge?

I think this should be good now. Let me know if there is anything else I should do.

TyPR124 · 2021-09-09T12:58:14Z

I've gone ahead and updated this to take ownership of a generic value type on each use. The T: Copy bound could be removed, but it seemed reasonable since Windows API will presumably need to make a copy of the value.

This also breaks object safety of the windows trait - I decided not to care about this since it is already broken for Unix and I don't think there is any practical use for dyn CommandExt over Command

bors · 2021-10-30T10:43:29Z

☔ The latest upstream changes (presumably #89174) made this pull request unmergeable. Please resolve the merge conflicts.

Dylan-DPC · 2022-02-22T18:33:27Z

@TyPR124 you have conflicts to be resolved

yaahc · 2022-03-05T14:22:14Z

I agree with the idea of renaming to call it the raw attribute interface to specifically leave room for a safe interface in the future, even if we don't have to design that interface as part of this PR

TyPR124 · 2022-03-06T08:00:15Z

I am having a very hard time with pulling most recent master... so just hoping it works to fix the conflicts through github. Not sure if the new commit is an issue.

Kobzol · 2022-03-06T18:42:08Z

Make sure that you don't have any pending changes in your workspace and then run this (assuming that you are on the branch of this PR):

$ git fetch origin master
$ git rebase origin/master

Then you need to fix conflicts, and run

$ git add -u
$ git rebase --continue
$ git push --force-with-lease

TyPR124 · 2022-03-07T00:18:33Z

If my remotes look like this:

❯ git remote -v
origin  git@github.com:TyPR124/rust.git (fetch)
origin  git@github.com:TyPR124/rust.git (push)
upstream        https://github.com/rust-lang/rust (fetch)
upstream        https://github.com/rust-lang/rust (push)

Then should I replaced 'origin' with 'upstream' in those commands?

I was previously trying to follow https://rustc-dev-guide.rust-lang.org/git.html which to fix merge conflicts says "First, get a local copy of the conflicting changes: Checkout your local master branch with git checkout master, then git pull upstream master" but when I do that I just get pain. Does that info need updated?

Edit: I've done the above with upstream instead of origin. I think it looks okay now. Not sure if I've messed something up in the past, or if those instructions could be updated somehow. I do know in the past I've used github's website to update my forks master branch, maybe that is incompatible with Rust's workflow?

Added doc comments, don't take too many attrs kill the cmds in test

Kobzol · 2022-03-07T07:06:07Z

Yeah if you named the rustc repo upstream you should use upstream in the commands above.

Your master branch probably got out of sync with the upstream master. In that case just switch to master (git checkout master) and forcefully set it to the upstream master branch (git reset --hard upstream/master).

Dylan-DPC · 2022-03-07T14:48:09Z

@TyPR124 git pull does a merge (git pull is effectively fetch + merge)

next time just add the -r flag to it which will do a rebase instead of a merge (i.e. git pull -r <remote>/<branch>)

Docs on --rebase

yaahc · 2022-03-07T19:01:57Z

Edit: I've done the above with upstream instead of origin. I think it looks okay now. Not sure if I've messed something up in the past, or if those instructions could be updated somehow. I do know in the past I've used github's website to update my forks master branch, maybe that is incompatible with Rust's workflow?

Looks like you got it to me, I only see your two commits, no merge commits.

Regarding the rename, what do you think about renaming the method to process_raw_thread_attribute? Also, this is gonna need a tracking issue before we can merge it.

yaahc · 2022-03-18T17:02:50Z

@rustbot label: -S-waiting-on-review +S-waiting-on-author

TyPR124 · 2022-03-23T17:02:16Z

Sorry, it's been a while since I've been able to put much time into this.

I've found that the test I wrote is racy. I'm not sure exactly why yet. If I apply this PR to nightly-2021-08-18, the test passes every time I run it (though I'm not 100% convinced that it never fails on this version). At some point between 2021-11-18 and 2021-12-18, it seems like the issue was either introduced or made more likely to cause test failure.

A workaround that seems to be 100% effective in my testing is spawn the second (child) process with sleep 1. Without this, it seems that the child process dies before the wmic command runs, so wmic fails to find the process.

Re: renaming the method. I don't think process_raw_thread_attribute makes sense here. The name PROCESS_THREAD_ATTRIBUTE is, as far as I know, is becuase these are attributes that can be applied to either process' or threads. Since the context it is being used here is specifically for process spawning, I think just raw_attribute will be better - and I can't think of any reason why we'd need a [raw-]attribute() method other than this purpose.

ChrisDenton · 2022-03-23T18:11:44Z

it seems that the child process dies before the wmic command runs, so wmic fails to find the process

Could you add the wmic command to the child process itself? Edit: Hm, getting the current process id within cmd is harder than I thought. Maybe using powershell would be simpler? I.e.:

$(Get-Process -id $PID).Parent.Id

yaahc · 2022-03-23T18:16:40Z

Re: renaming the method. I don't think process_raw_thread_attribute makes sense here. The name PROCESS_THREAD_ATTRIBUTE is, as far as I know, is becuase these are attributes that can be applied to either process' or threads. Since the context it is being used here is specifically for process spawning, I think just raw_attribute will be better - and I can't think of any reason why we'd need a [raw-]attribute() method other than this purpose.

Sounds reasonable, 👍 for naming it to raw_attribute.

TyPR124 · 2022-03-24T00:17:38Z

$(Get-Process -id $PID).Parent.Id

Good idea, but for me it only works in pwsh; not native powershell. I imagine that would be a problem.

wmic process where processid=$PID get parentprocessid seems to work in both, so I might give that a go.

lzybkr · 2022-03-24T00:36:31Z

$(Get-Process -id $PID).Parent.Id

Good idea, but for me it only works in pwsh; not native powershell. I imagine that would be a problem.

wmic process where processid=$PID get parentprocessid seems to work in both, so I might give that a go.

If you'd like a PowerShell option that works in powershell and pwsh - you can use:

(Get-CimInstance Win32_Process -Filter "ParentProcessId=$PID").ProcessId

This is still using WMI under the covers, just not invoking the wmic command line tool.

EddieIvan01 · 2022-04-20T08:11:41Z

So, is there an expected plan for merging this PR?

yaahc · 2022-04-20T18:49:52Z

So, is there an expected plan for merging this PR?

I believe it's just waiting on the author to update the PR to rename the method and create a tracking issue.

Dylan-DPC · 2022-04-28T20:47:02Z

@TyPR124 any updates?

JohnCSimon · 2022-05-22T03:04:13Z

@TyPR124
Ping from triage: I'm closing this due to inactivity, Please reopen when you are ready to continue with this.
Note: if you do please open the PR BEFORE you push to it, else you won't be able to reopen - this is a quirk of github.
Thanks for your contribution.

@rustbot label: +S-inactive

…tes, r=ChrisDenton Add ability to spawn Windows process with Proc Thread Attributes | Take 2 This is the second attempt to merge pull request rust-lang#88193 into the standard library. This PR implements the ability to add arbitrary attributes to a command on Windows targets using a new `raw_attribute` method on the [`CommandExt`](https://doc.rust-lang.org/stable/std/os/windows/process/trait.CommandExt.html) trait. `@TyPR124` and my main motivation behind adding this feature is to enable the support of pseudo terminals in the std library, but there are many more applications. A good starting point to get into this topic is to head over to the [`Win32 API documentation`](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute).

rust-highfive assigned Mark-Simulacrum Aug 20, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 20, 2021

bjorn3 reviewed Aug 21, 2021

View reviewed changes

library/std/src/process/tests.rs Outdated Show resolved Hide resolved

ChrisDenton reviewed Aug 21, 2021

View reviewed changes

rust-highfive assigned yaahc and unassigned Mark-Simulacrum Aug 24, 2021

Mark-Simulacrum added the T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. label Aug 24, 2021

TyPR124 force-pushed the spawn_with_attributes branch 2 times, most recently from 7926789 to b42573b Compare August 30, 2021 14:47

TyPR124 force-pushed the spawn_with_attributes branch from 1778cfc to 4b61cdf Compare September 9, 2021 17:28

JohnCSimon added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 27, 2021

JohnCSimon added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 19, 2021

JohnCSimon added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 5, 2021

JohnCSimon added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 13, 2022

TyPR124 added 2 commits March 6, 2022 22:16

add windows ext method for proc thread attributes

0475e2f

Added doc comments, don't take too many attrs kill the cmds in test

Take generic value type

d20fc4e

TyPR124 force-pushed the spawn_with_attributes branch from 6a59d12 to d20fc4e Compare March 7, 2022 03:17

ChrisDenton mentioned this pull request Mar 9, 2022

Consider different way of inheriting handles in child processes on Windows #38227

Open

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 18, 2022

JohnCSimon closed this May 22, 2022

rustbot added the S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. label May 22, 2022

waruqi mentioned this pull request Nov 25, 2022

Cannot move source.tmp when installing package from git xmake-io/xmake#2902

Closed

michaelvanstraten mentioned this pull request Aug 15, 2023

Add ability to spawn Windows process with Proc Thread Attributes | Take 2 #114848

Merged

michaelvanstraten mentioned this pull request Dec 31, 2023

Tracking Issue for windows_process_extensions_raw_attribute #114854

Open

3 tasks

cre4ture mentioned this pull request Jan 21, 2024

tests: Provide a way to run tests with PTY stdout uutils/coreutils#5785

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to spawn Windows process with Proc Thread Attributes #88193

Add ability to spawn Windows process with Proc Thread Attributes #88193

TyPR124 commented Aug 20, 2021

rust-highfive commented Aug 20, 2021

ChrisDenton commented Aug 21, 2021

ChrisDenton Aug 21, 2021

TyPR124 Aug 21, 2021

bjorn3 Aug 21, 2021 •

edited

Loading

ChrisDenton Aug 21, 2021

TyPR124 Aug 21, 2021

TyPR124 commented Aug 21, 2021 •

edited

Loading

ChrisDenton commented Aug 21, 2021 •

edited

Loading

bjorn3 commented Aug 22, 2021

Mark-Simulacrum commented Aug 24, 2021

TyPR124 commented Aug 24, 2021

TyPR124 commented Aug 25, 2021

TyPR124 commented Aug 30, 2021

TyPR124 commented Sep 9, 2021

bors commented Oct 30, 2021

Dylan-DPC commented Feb 22, 2022

yaahc commented Mar 5, 2022

TyPR124 commented Mar 6, 2022

Kobzol commented Mar 6, 2022

TyPR124 commented Mar 7, 2022 •

edited

Loading

Kobzol commented Mar 7, 2022

Dylan-DPC commented Mar 7, 2022

yaahc commented Mar 7, 2022 •

edited

Loading

yaahc commented Mar 18, 2022

TyPR124 commented Mar 23, 2022

ChrisDenton commented Mar 23, 2022 •

edited

Loading

yaahc commented Mar 23, 2022

TyPR124 commented Mar 24, 2022

lzybkr commented Mar 24, 2022

EddieIvan01 commented Apr 20, 2022

yaahc commented Apr 20, 2022

Dylan-DPC commented Apr 28, 2022

JohnCSimon commented May 22, 2022

Add ability to spawn Windows process with Proc Thread Attributes #88193

Add ability to spawn Windows process with Proc Thread Attributes #88193

Conversation

TyPR124 commented Aug 20, 2021

rust-highfive commented Aug 20, 2021

ChrisDenton commented Aug 21, 2021

ChrisDenton Aug 21, 2021

Choose a reason for hiding this comment

TyPR124 Aug 21, 2021

Choose a reason for hiding this comment

bjorn3 Aug 21, 2021 • edited Loading

Choose a reason for hiding this comment

ChrisDenton Aug 21, 2021

Choose a reason for hiding this comment

TyPR124 Aug 21, 2021

Choose a reason for hiding this comment

TyPR124 commented Aug 21, 2021 • edited Loading

ChrisDenton commented Aug 21, 2021 • edited Loading

bjorn3 commented Aug 22, 2021

Mark-Simulacrum commented Aug 24, 2021

TyPR124 commented Aug 24, 2021

TyPR124 commented Aug 25, 2021

TyPR124 commented Aug 30, 2021

TyPR124 commented Sep 9, 2021

bors commented Oct 30, 2021

Dylan-DPC commented Feb 22, 2022

yaahc commented Mar 5, 2022

TyPR124 commented Mar 6, 2022

Kobzol commented Mar 6, 2022

TyPR124 commented Mar 7, 2022 • edited Loading

Kobzol commented Mar 7, 2022

Dylan-DPC commented Mar 7, 2022

yaahc commented Mar 7, 2022 • edited Loading

yaahc commented Mar 18, 2022

TyPR124 commented Mar 23, 2022

ChrisDenton commented Mar 23, 2022 • edited Loading

yaahc commented Mar 23, 2022

TyPR124 commented Mar 24, 2022

lzybkr commented Mar 24, 2022

EddieIvan01 commented Apr 20, 2022

yaahc commented Apr 20, 2022

Dylan-DPC commented Apr 28, 2022

JohnCSimon commented May 22, 2022

bjorn3 Aug 21, 2021 •

edited

Loading

TyPR124 commented Aug 21, 2021 •

edited

Loading

ChrisDenton commented Aug 21, 2021 •

edited

Loading

TyPR124 commented Mar 7, 2022 •

edited

Loading

yaahc commented Mar 7, 2022 •

edited

Loading

ChrisDenton commented Mar 23, 2022 •

edited

Loading