Added From<Vec<NonZeroU8>> for CString #64069

danielhenrymantilla · 2019-09-01T15:40:23Z

Added a From<Vec<NonZeroU8>> impl for CString

Rationale

CString::from_vec_unchecked is a subtle function, that makes unsafe code harder to audit when the generated Vec's creation is non-trivial. This impl allows to write safer unsafe code thanks to the very explicit semantics of the Vec<NonZeroU8> type.
One such situation is when trying to .read() a CString, see issue Reading a CString safely without overhead from Read #59229.
- this lead to a PR: Implement CString::from_reader #59314, that was closed for being too specific / narrow (it only targetted being able to .read() a CString, when this pattern could have been generalized).
- the issue suggested another route, based on From<Vec<NonZeroU8>>, which is indeed a less general and more concise code pattern.
quoting @Shnatsel:
- For me the main thing about making this safe is simplifying auditing - people have spent like an hour looking at just this one unsafe block in libflate because it's not clear what exactly is unchecked, so you have to look it up when auditing anyway. This has distracted us from much more serious memory safety issues the library had.
  Having this trivial impl in stdlib would turn this into safe code with compiler more or less guaranteeing that it's fine, and save anyone auditing the code a whole lot of time.

rust-highfive · 2019-09-01T15:40:27Z

r? @withoutboats

(rust_highfive has picked a reviewer for you, use r? to override)

src/libstd/ffi/c_str.rs

rust-highfive · 2019-09-01T17:38:46Z

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-09-01T17:30:24.9862540Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-09-01T17:30:25.6788628Z ##[command]git config gc.auto 0
2019-09-01T17:30:25.6791906Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-09-01T17:30:25.6795190Z ##[command]git config --get-all http.proxy
2019-09-01T17:30:25.6799980Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/64069/merge:refs/remotes/pull/64069/merge
---
2019-09-01T17:36:38.0473105Z     Checking backtrace v0.3.35
2019-09-01T17:36:43.2265818Z error: impl has missing stability attribute
2019-09-01T17:36:43.2267576Z    --> src/libstd/ffi/c_str.rs:722:1
2019-09-01T17:36:43.2268155Z     |
2019-09-01T17:36:43.2268657Z 722 | / impl From<Vec<NonZeroU8>> for CString {
2019-09-01T17:36:43.2269186Z 723 | |     /// Converts a [`Vec`]`<`[`NonZeroU8`]`>` into a [`CString`] without
2019-09-01T17:36:43.2269717Z 724 | |     /// copying nor checking for inner null bytes.
2019-09-01T17:36:43.2270910Z ...   |
2019-09-01T17:36:43.2271338Z 793 | |     }
2019-09-01T17:36:43.2271800Z 794 | | }
2019-09-01T17:36:43.2272242Z     | |_^
---
2019-09-01T17:36:43.4492110Z == clock drift check ==
2019-09-01T17:36:43.4512422Z   local time: Sun Sep  1 17:36:43 UTC 2019
2019-09-01T17:36:43.6012744Z   network time: Sun, 01 Sep 2019 17:36:43 GMT
2019-09-01T17:36:43.6017002Z == end clock drift check ==
2019-09-01T17:36:50.5865241Z ##[error]Bash exited with code '1'.
2019-09-01T17:36:50.5899521Z ##[section]Starting: Checkout
2019-09-01T17:36:50.5902055Z ==============================================================================
2019-09-01T17:36:50.5902124Z Task         : Get sources
2019-09-01T17:36:50.5902167Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

clarfonthey · 2019-09-03T21:52:04Z

Perhaps there could be a method to return the nonzero bytes as a slice as well?

Shnatsel · 2019-09-03T21:57:19Z

Perhaps there could be a method to return the nonzero bytes as a slice as well?

Sounds reasonable to me. That should be its own PR though, as that can be feature-gated for some time while trait implementations are instantly stable.

JohnCSimon · 2019-09-14T01:43:01Z

Ping from triage
@Centril @withoutboats This PR is still waiting on review.
CC @danielhenrymantilla

Thanks.

Centril

Mostly just some nits...

Centril · 2019-09-14T11:46:33Z

src/libstd/ffi/c_str.rs

+        ///   - for any `cap: usize`, `Layout<[T; cap]>` needs to be equal to
+        ///     `Layout<[U; cap]>` (for the allocator)
+        #[inline]
+        unsafe


what's with the line break here?

Woops, my own Rust style leaked here (Out of topic: I see pub and unsafe, among others, as markers, hence the logic of them being on an extra line rather than on the same line, in the same vein as other #[meta] attributes)

Centril · 2019-09-14T11:48:54Z

src/libstd/ffi/c_str.rs

+        #[inline]
+        unsafe
+        fn transmute_vec<T, U>(v: Vec<T>) -> Vec<U> {
+            // necessary conditions for `Layout<[T; N]> == Layout<[U; N]>`


Why not check Layout::array::<T / U>().unwrap() against each other?

Centril · 2019-09-14T11:51:28Z

src/libstd/ffi/c_str.rs

+        ///     `[T; length]` and `[U; length]`.
+        ///
+        ///   - for any `cap: usize`, `Layout<[T; cap]>` needs to be equal to
+        ///     `Layout<[U; cap]>` (for the allocator)


Suggested change

/// `Layout<[U; cap]>` (for the allocator)

/// `Layout<[U; cap]>` (for the allocator).

src/libstd/ffi/c_str.rs

Centril · 2019-09-14T11:53:50Z

r? @RalfJung for checking the safety argument more in-depth. Also cc @rkruppe.

(Do ping T-libs once that review is done...)

hanna-kruppe

Soundness argument seems fine to me, just some nits.

hanna-kruppe · 2019-09-14T12:24:16Z

src/libstd/ffi/c_str.rs

+            //
+            //   - `v` cannot contain null bytes, given the type-level
+            //     invariant of `NonZeroU8` (this would still apply even if
+            //     this invariant was a safety invariant and not a validity


Nit: strike the hypothetical. I am reasonably sure you do need the safety invariant here. Most operations on vectors don't assert validity of any elements except perhaps the ones they move into or out of the vec. Of course, a Vec<T> is only safe if elements 0..len are safe at T, so the soundness argument can rest on that plus the safety/validity (same thing) of the NonZeroU8.

(Insert standard disclaimer about all of these details being not yet ratified here.)

You're right, I will remove the part between brackets then (I'm still glad I put it, it is good to clearly think about these things before merging).

hanna-kruppe · 2019-09-14T12:24:47Z

src/libstd/ffi/c_str.rs

+            Vec::<U>::from_raw_parts(ptr as *mut U, length, capacity)
+        }
+
+        let v = unsafe {


A type annotation won't hurt here IMO.

danielhenrymantilla · 2019-09-14T15:03:19Z

src/libstd/ffi/c_str.rs

+                crate::alloc::Layout::new::<U>(),
+            );
+            // The previous assert should imply the following one:
+            debug_assert_eq!(


The first check is an assert since it is expected to be easily optimized out (only const / compile-time information here).
On the other hand, the check involving Layout::array cannot be const, since it involves the runtime value of the vector capacity.
That being said, I expect Layout<T> == Layout<U> to imply that for all N: usize, Layout<[T; N]> == Layout<[U; N]>.
If this were not to be correct / guaranteed by the language, this debug assertion could be upgraded to a normal assert!; in this case however, the function is only used with the monomorphisation to T = NonZeroU8 and V = u8, which does not require such check at runtime.

I think it would be a serious bug in Layout::array if this is not given... its comparison test would fail to take something into account, or so.

Not sure if this check is worth the clutter it introduces.

Great! I will remove the debug_assert! involving the Layout::array check

RalfJung · 2019-09-16T18:11:56Z

I'm afraid my Rust inbox is swamped, I won't be able to review this -- I can take a glance, but don't have time to do a full review.

Any takers?

(Also is there some way to tell bors "sorry, not me", if I don't know whom else to assign?)

RalfJung · 2019-09-16T18:15:53Z

src/libstd/ffi/c_str.rs

+                // End-of-scope `mem::forget` but without aliasing problems.
+                let mut v = mem::ManuallyDrop::<Vec<T>>::new(v);
+                let v: &mut Vec<T> = &mut *v;
+                v.as_mut_ptr() // Cannot be aliased.


ManuallyDrop implements Deref; why is the intermediate let v: &mut Vec<T> = ... needed?

It's just me being (overly?) cautious with type inference / method resolution within unsafe code , so I wanted to have that Deref be very explicit.

Mark-Simulacrum · 2019-09-16T18:17:10Z

Going to re-assign to @rkruppe as they appear to have done some review here but feel free to reassign to someone else if you don't feel confident reviewing this.

hanna-kruppe · 2019-09-16T20:45:55Z

I'm fine with reviewing this. Will take another close look later for due digilence but from what I recall this should be in good shape.

I think some @rust-lang/libs sign-off is also required since this is a new insta-stable API addition.

Mark-Simulacrum · 2019-09-16T20:47:39Z

I've nominated to get someone from T-libs to kick off an FCP merge here

dtolnay · 2020-01-10T08:51:59Z

I checked Kimundi's box per rust-lang/team@bfbf48f.

rfcbot · 2020-01-10T08:52:02Z

🔔 This is now entering its final comment period, as per the review above. 🔔

rfcbot · 2020-01-20T08:55:48Z

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

KodrAus · 2020-01-27T23:15:56Z

Did we come to a decision on the multiple unsafe blocks? I don't imagine it needs to block merging, but am also of the opinion that we should combine them into one, especially now that they live alongside one another with no safe code in between. They're logically a single operation so I don't think their granularity is necessarily better here.

But thanks for working on this @danielhenrymantilla! Would you like to squash some of your commits down before merging?

@Centril

Updated tracking issue number Added safeguards for transmute_vec potentially being factored out elsewhere Clarified comment about avoiding mem::forget Removed unneeded unstable guard Added back a stability annotation for CI Minor documentation improvements Thanks to @Centril's code review Co-Authored-By: Mazdak Farrokhzad <twingoow@gmail.com> Improved layout checks, type annotations and removed unaccurate comment Removed unnecessary check on array layout Adapt the stability annotation to the new 1.41 milestone Co-Authored-By: Mazdak Farrokhzad <twingoow@gmail.com> Simplify the implementation. Use `Vec::into_raw_parts` instead of a manual implementation of `Vec::transmute`. If `Vec::into_raw_parts` uses `NonNull` instead, then the code here will need to be adjusted to take it into account (issue rust-lang#65816) Reduce the whitespace of safety comments

danielhenrymantilla · 2020-02-04T16:23:22Z

I did have a preference for the split blocks, but I seem to be at a minority here, so I have merged the unsafe blocks (and updated the stability annotation and squashed and rebased everything).

There should be nothing blocking the merge, now 🙂

Dylan-DPC-zz · 2020-02-12T00:23:22Z

r? @KodrAus

KodrAus · 2020-02-15T07:45:17Z

@bors r+

bors · 2020-02-15T07:45:18Z

📌 Commit 60274a9 has been approved by KodrAus

@Shnatsel

…_from_vec_of_nonzerou8, r=KodrAus Added From<Vec<NonZeroU8>> for CString Added a `From<Vec<NonZeroU8>>` `impl` for `CString` # Rationale - `CString::from_vec_unchecked` is a subtle function, that makes `unsafe` code harder to audit when the generated `Vec`'s creation is non-trivial. This `impl` allows to write safer `unsafe` code thanks to the very explicit semantics of the `Vec<NonZeroU8>` type. - One such situation is when trying to `.read()` a `CString`, see issue rust-lang#59229. - this lead to a PR: rust-lang#59314, that was closed for being too specific / narrow (it only targetted being able to `.read()` a `CString`, when this pattern could have been generalized). - the issue suggested another route, based on `From<Vec<NonZeroU8>>`, which is indeed a less general and more concise code pattern. - quoting @Shnatsel: - > For me the main thing about making this safe is simplifying auditing - people have spent like an hour looking at just this one unsafe block in libflate because it's not clear what exactly is unchecked, so you have to look it up when auditing anyway. This has distracted us from much more serious memory safety issues the library had. Having this trivial impl in stdlib would turn this into safe code with compiler more or less guaranteeing that it's fine, and save anyone auditing the code a whole lot of time.

@ghost

Rollup of 6 pull requests Successful merges: - #64069 (Added From<Vec<NonZeroU8>> for CString) - #66721 (implement LowerExp and UpperExp for integers) - #69106 (Fix std::fs::copy on WASI target) - #69154 (Avoid calling `fn_sig` on closures) - #69166 (Check `has_typeck_tables` before calling `typeck_tables_of`) - #69180 (Suggest a comma if a struct initializer field fails to parse) Failed merges: r? @ghost

bors · 2020-02-15T13:47:50Z

⌛ Testing commit 60274a9 with merge dbef353...

rust-highfive assigned withoutboats Sep 1, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 1, 2019

danielhenrymantilla mentioned this pull request Sep 1, 2019

Tracking issue for cstring_from_vec_of_nonzerou8 #64070

Closed

Centril reviewed Sep 1, 2019

View reviewed changes

src/libstd/ffi/c_str.rs Outdated Show resolved Hide resolved

scottmcm added the needs-fcp This change is insta-stable, so needs a completed FCP to proceed. label Sep 2, 2019

Centril added relnotes Marks issues that should be documented in the release notes of the next release. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Sep 3, 2019

Centril added this to the 1.39 milestone Sep 3, 2019

Shnatsel mentioned this pull request Sep 3, 2019

RFC ideas rust-secure-code/wg#10

Open

Centril reviewed Sep 14, 2019

View reviewed changes

rust-highfive assigned RalfJung and unassigned withoutboats Sep 14, 2019

hanna-kruppe reviewed Sep 14, 2019

View reviewed changes

danielhenrymantilla commented Sep 14, 2019

View reviewed changes

danielhenrymantilla force-pushed the feature/cstring_from_vec_of_nonzerou8 branch 2 times, most recently from 755fe39 to b32aec7 Compare September 14, 2019 15:26

RalfJung reviewed Sep 16, 2019

View reviewed changes

Mark-Simulacrum assigned hanna-kruppe and unassigned RalfJung Sep 16, 2019

Mark-Simulacrum added the I-nominated label Sep 16, 2019

Dylan-DPC-zz added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 12, 2019

Centril modified the milestones: 1.41, 1.42 Dec 20, 2019

rfcbot added final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. and removed proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. labels Jan 10, 2020

rfcbot added finished-final-comment-period The final comment period is finished for this PR / Issue. and removed final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. labels Jan 20, 2020

Centril modified the milestones: 1.42, 1.43 Jan 31, 2020

danielhenrymantilla force-pushed the feature/cstring_from_vec_of_nonzerou8 branch from 8a9725a to 60274a9 Compare February 4, 2020 16:21

rust-highfive assigned KodrAus and unassigned sfackler Feb 12, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 15, 2020

Dylan-DPC-zz mentioned this pull request Feb 15, 2020

Rollup of 6 pull requests #69182

Merged

bors merged commit 60274a9 into rust-lang:master Feb 15, 2020

Shnatsel mentioned this pull request Feb 15, 2020

Reading a CString safely without overhead from Read #59229

Closed

danielhenrymantilla deleted the feature/cstring_from_vec_of_nonzerou8 branch March 2, 2020 18:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added From<Vec<NonZeroU8>> for CString #64069

Added From<Vec<NonZeroU8>> for CString #64069

danielhenrymantilla commented Sep 1, 2019 •

edited

Loading

rust-highfive commented Sep 1, 2019

rust-highfive commented Sep 1, 2019

clarfonthey commented Sep 3, 2019

Shnatsel commented Sep 3, 2019

JohnCSimon commented Sep 14, 2019

Centril left a comment

Centril Sep 14, 2019

danielhenrymantilla Sep 14, 2019 •

edited

Loading

Centril Sep 14, 2019

Centril Sep 14, 2019

Centril commented Sep 14, 2019 •

edited

Loading

hanna-kruppe left a comment

hanna-kruppe Sep 14, 2019

danielhenrymantilla Sep 14, 2019

hanna-kruppe Sep 14, 2019

danielhenrymantilla Sep 14, 2019

danielhenrymantilla Sep 14, 2019 •

edited

Loading

RalfJung Sep 16, 2019

danielhenrymantilla Sep 16, 2019

RalfJung commented Sep 16, 2019

RalfJung Sep 16, 2019

danielhenrymantilla Sep 16, 2019

Mark-Simulacrum commented Sep 16, 2019

hanna-kruppe commented Sep 16, 2019 •

edited

Loading

Mark-Simulacrum commented Sep 16, 2019

dtolnay commented Jan 10, 2020

rfcbot commented Jan 10, 2020

rfcbot commented Jan 20, 2020

KodrAus commented Jan 27, 2020

danielhenrymantilla commented Feb 4, 2020

Dylan-DPC-zz commented Feb 12, 2020

KodrAus commented Feb 15, 2020

bors commented Feb 15, 2020

bors commented Feb 15, 2020

	/// `Layout<[U; cap]>` (for the allocator)
	/// `Layout<[U; cap]>` (for the allocator).

Added From<Vec<NonZeroU8>> for CString #64069

Added From<Vec<NonZeroU8>> for CString #64069

Conversation

danielhenrymantilla commented Sep 1, 2019 • edited Loading

Rationale

rust-highfive commented Sep 1, 2019

rust-highfive commented Sep 1, 2019

clarfonthey commented Sep 3, 2019

Shnatsel commented Sep 3, 2019

JohnCSimon commented Sep 14, 2019

Centril left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielhenrymantilla Sep 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Centril commented Sep 14, 2019 • edited Loading

hanna-kruppe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielhenrymantilla Sep 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung commented Sep 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Sep 16, 2019

hanna-kruppe commented Sep 16, 2019 • edited Loading

Mark-Simulacrum commented Sep 16, 2019

dtolnay commented Jan 10, 2020

rfcbot commented Jan 10, 2020

rfcbot commented Jan 20, 2020

KodrAus commented Jan 27, 2020

danielhenrymantilla commented Feb 4, 2020

Dylan-DPC-zz commented Feb 12, 2020

KodrAus commented Feb 15, 2020

bors commented Feb 15, 2020

bors commented Feb 15, 2020

danielhenrymantilla commented Sep 1, 2019 •

edited

Loading

danielhenrymantilla Sep 14, 2019 •

edited

Loading

Centril commented Sep 14, 2019 •

edited

Loading

danielhenrymantilla Sep 14, 2019 •

edited

Loading

hanna-kruppe commented Sep 16, 2019 •

edited

Loading