Clarify `Box<T>` representation and its use in FFI #62514

stephaneyfx · 2019-07-09T05:14:38Z

This officializes what was only shown as a code example in the unsafe code guidelines and follows the discussion in the corresponding repository.

It is also related to the issue regarding marking Box<T> #[repr(transparent)].

If the statement this PR adds is incorrect or a more in-depth discussion is warranted, I apologize. Should it be the case, the example in the unsafe code guidelines should be amended and some document should make it clear that it is not sound/supported.

rust-highfive · 2019-07-09T05:14:49Z

r? @dtolnay

(rust_highfive has picked a reviewer for you, use r? to override)

CryZe · 2019-07-09T11:45:26Z

src/liballoc/boxed.rs

@@ -63,6 +63,27 @@
 //! T` obtained from `Box::<T>::into_raw` may be deallocated using the
 //! [`Global`] allocator with `Layout::for_value(&*value)`.
 //!
+//! `Box<SomethingSized>` has the same representation as `*mut SomethingSized`.


They would always have the same representation, not just if T is Sized.

Right, sorry. I was just so focused on the FFI aspect of it that started this PR in the first place that I phrased that poorly. A fix is on the way. Thank you!

stephaneyfx

I should probably change the parameter type of bar to Option<Box<Foo>> so that the Rust code can handle C code calling this function with a null pointer, which would lead to UB. I will update the PR tonight.

src/liballoc/boxed.rs

This officializes what was only shown as a code example in [the unsafe code guidelines](https://rust-lang.github.io/unsafe-code-guidelines/layout/function-pointers.html?highlight=box#use) and follows [the discussion](rust-lang/unsafe-code-guidelines#157) in the corresponding repository. It is also related to [the issue](rust-lang#52976) regarding marking `Box<T>` `#[repr(transparent)]`.

dtolnay · 2019-07-12T04:10:04Z

I am on board with documenting this as a guarantee. Let's get a set of lang team eyes as well:
r? @joshtriplett

Alexendoo · 2019-07-24T12:15:57Z

Ping from triage @joshtriplett, any updates?

gnzlbg · 2019-07-24T16:19:25Z

This officializes what was only shown as a code example in the unsafe code guidelines and follows the discussion in the corresponding repository.

There is a PR to propose guaranteeing this as part of the UCGs RFC, but that PR has not been merged yet: rust-lang/unsafe-code-guidelines#164

gnzlbg · 2019-07-24T16:21:28Z

cc @rust-lang/lang

stephaneyfx · 2019-07-24T17:15:01Z

rust-lang/unsafe-code-guidelines#164 is more general and does not address Box in particular. Given that the field of Box is private, other fields could theoretically be added (however unlikely it is) and so this UCG RFC is not enough for users to rely on the representation of Box being the same as a C pointer. I think that this PR makes a stronger and straightforward commitment FFI writers can rely on.

joshtriplett · 2019-07-25T19:09:41Z

Sorry for the delayed review.

I have a concern here regarding semantics, rather than representation.

If you actually pass a Box<T> to an external function, that would pass ownership of it, so Rust won't know about it anymore and won't free it, so does that work for giving over ownership of freeing it to C? And conversely, if C allocates something and your Rust allocator can free C objects, can you accept a return value of Box<T> to take ownership of that heap object?

(Also, this should not work for any T with a destructor.)

This seems consistent with the language you've used here. I just want to raise the point explicitly and ask if this will actually work in all cases. If it will, then by all means let's specify this, and it seems like a very useful way to specify ownership transfer of a heap object.

joshtriplett · 2019-07-25T19:19:23Z

@rfcbot merge

Based on discussion in the lang team meeting, we're confident that this will work, and does indeed represent ownership transfer of a heap object across an FFI boundary.

rfcbot · 2019-07-25T19:19:24Z

nikomatsakis · 2019-07-25T19:20:48Z

(Tagging @pnkfelix and @withoutboats as they are on leave.)

nikomatsakis

Apart from a grammatical nit, these changes look good! However, we may want to address the point that @gnzlbg raised, where using Rust types for functions defined in C can (in subtle cases where things are compiled with PGO etc) cause UB. Do you want me to take a stab at how to word that?

nikomatsakis · 2019-12-10T22:28:38Z

src/liballoc/boxed.rs

@@ -63,6 +63,50 @@
 //! T` obtained from `Box::<T>::into_raw` may be deallocated using the
 //! [`Global`] allocator with `Layout::for_value(&*value)`.
 //!
+//! So long as `T: Sized`, a `Box<T>` is guaranteed to be represented as a
+//! single pointer and is also ABI-compatible with C pointers (i.e. the C type
+//! `T*`). This means that you have Rust code which passes ownership of a


Suggested change

//! `T*`). This means that you have Rust code which passes ownership of a

//! `T*`). This means that you can have Rust code which passes ownership of a

Done. Thank you!

nikomatsakis · 2019-12-10T22:31:14Z

src/liballoc/boxed.rs

+//! single pointer and is also ABI-compatible with C pointers (i.e. the C type
+//! `T*`). This means that you have Rust code which passes ownership of a
+//! `Box<T>` to C code by using `Box<T>` as the type on the Rust side, and
+//! `T*` as the corresponding type on the C side. As an example, consider this


So at this point, @gnzlbg has convinced me that there is one remaining issue that we may want to address here. In particular, it seems to me that it is only safe to use Rust types if all callers to the function obey the Rust restrictions (e.g., non-null, valid, etc). The easiest way to express that is to say that it is safe to use Rust in types in cases where the function is defined in Rust, but not in cases where the function is defined in C. I'm debating how much detail to go into here and where.

stephaneyfx · 2019-12-11T06:42:07Z

@nikomatsakis Yes, if you don't mind. I am sure your wording will be more accurate than mine. Thank you.

gnzlbg · 2019-12-11T07:55:52Z

@nikomatsakis I think you correctly understood the issue.

This implies to me that it is not sound to use a Rust prototype that is more narrow than the types at the definition site. So if the function is defined in C, in other words, you had best use Raw pointers and match the definition precisely.

However, I believe it is sound (if suboptimal) to use Rust types if the same types are used at the definition. So, in the example given, where the functions are defined in Rust, using types like Box, but the C headers are using raw pointers, everything is fine.

In C, the behavior of programs containing prototypes that do not exactly match the definition is undefined. When using Rust with xLTO, the only thing that should matter are the semantics of LLVM-IR, but AFAIK these semantics are unfortunately not written anywhere and LLVM-IR does not guarantee yet that declarations that are less narrow than definitions are ok. With features like xLTO it can be very hard for declarations to precisely match definitions at the LLVM-IR level, so I also think that allowing these use cases would make sense. Maybe these issues should be brought up with the LLVM community?

nikomatsakis · 2019-12-11T15:20:55Z

@gnzlbg in my opinion, this feels like an LLVM bug, I do think it'd be good to raise with them. However, for the time being we can at least advise people to only use Rust types for functions that are defined in Rust.

nikomatsakis · 2019-12-11T15:33:19Z

OK, I pushed some updates notes. I'm inclined to r+ now but I'll give it a bit to see if anyone has any suggestions on my wording.

gnzlbg

This LGTM!

nikomatsakis · 2019-12-11T17:30:55Z

@bors r+ rollup

bors · 2019-12-11T17:30:57Z

📌 Commit fafa489 has been approved by nikomatsakis

RalfJung · 2019-12-11T17:36:16Z

LGTM as well.

The concern about C prototypes also applies to &/&mut in FFI, so probably those should get the same warning?

gnzlbg · 2019-12-11T18:01:57Z

The concern about C prototypes also applies to &/&mut in FFI, so probably those should get the same warning?

Yes, I think so, and NonNull should get this warning as well.

RalfJung · 2019-12-11T18:20:57Z

@gnzlbg do you plan to write a PR, or should we open an issue to not forget about this?

gnzlbg · 2019-12-11T18:49:02Z

@RalfJung I've opened #67234

Clarify `Box<T>` representation and its use in FFI This officializes what was only shown as a code example in [the unsafe code guidelines](https://rust-lang.github.io/unsafe-code-guidelines/layout/function-pointers.html?highlight=box#use) and follows [the discussion](rust-lang/unsafe-code-guidelines#157) in the corresponding repository. It is also related to [the issue](rust-lang#52976) regarding marking `Box<T>` `#[repr(transparent)]`. If the statement this PR adds is incorrect or a more in-depth discussion is warranted, I apologize. Should it be the case, the example in the unsafe code guidelines should be amended and some document should make it clear that it is not sound/supported.

@ghost

Rollup of 8 pull requests Successful merges: - #62514 (Clarify `Box<T>` representation and its use in FFI) - #66983 (Fix `unused_parens` triggers on macro by example code) - #67215 (Fix `-Z print-type-sizes`'s handling of zero-sized fields.) - #67230 (Remove irelevant comment on `register_dtor`) - #67236 (resolve: Always resolve visibilities on impl items) - #67237 (Some small readability improvements) - #67238 (Small std::borrow::Cow improvements) - #67239 (Make TinyList::remove iterate instead of recurse) Failed merges: r? @ghost

bors · 2019-12-12T05:31:29Z

☔ The latest upstream changes (presumably #67246) made this pull request unmergeable. Please resolve the merge conflicts.

briansmith · 2022-10-20T19:53:12Z

src/liballoc/boxed.rs

+//! pub struct Foo;
+//!
+//! #[no_mangle]
+//! pub extern "C" fn foo_new() -> Box<Foo> {


Usually a function like foo_new() requires that the returned pointer be destroyed by calling the foo_delete() function. Returning a Box is almost always the wrong thing to do here since Box doesn't enforce that foo_delete is called when the Box is dropped. So this example is documenting a pattern that is usually the wrong thing to do.

@briansmith this is a long-merged PR, so new comments here are bound to get lost... please open an issue if you think this needs to be tracked.

Returning a Box is almost always the wrong thing to do here since Box doesn't enforce that foo_delete is called when the Box is dropped.

It's no different than "regular" FFI code where one returns raw pointer and requires the caller to call the corresponding _delete function with the same pointer, but returning & accepting Box better documents semantics.

If you open a new issue, please cc me, this is a pattern close to my heart :)

rust-highfive assigned dtolnay Jul 9, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 9, 2019

stephaneyfx mentioned this pull request Jul 9, 2019

Is Box FFI-safe? rust-lang/unsafe-code-guidelines#157

Closed

CryZe reviewed Jul 9, 2019

View reviewed changes

stephaneyfx changed the title ~~State that Box<SomethingSized> and *mut SomethingSized have the same representation~~ Clarify Box<T> representation and its use in FFI Jul 9, 2019

stephaneyfx force-pushed the box-ffi branch from e3671f6 to ecbca77 Compare July 9, 2019 13:43

Centril added needs-fcp This change is insta-stable, so needs a completed FCP to proceed. T-lang Relevant to the language team, which will review and decide on the PR/issue. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Jul 9, 2019

stephaneyfx commented Jul 9, 2019

View reviewed changes

src/liballoc/boxed.rs Outdated Show resolved Hide resolved

stephaneyfx force-pushed the box-ffi branch from ecbca77 to 318c5d6 Compare July 10, 2019 03:40

dtolnay approved these changes Jul 12, 2019

View reviewed changes

rust-highfive assigned joshtriplett and unassigned dtolnay Jul 12, 2019

Centril added the I-nominated label Jul 24, 2019

Centril removed I-nominated needs-fcp This change is insta-stable, so needs a completed FCP to proceed. labels Jul 25, 2019

rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels Jul 25, 2019

Remove trailing whitespace

1a26df7

nikomatsakis requested changes Dec 10, 2019

View reviewed changes

Fix description based on review

cb1cc11

clarify that Box<T> should only be used when defined *in Rust*

fafa489

nikomatsakis force-pushed the box-ffi branch from 23d7e3a to fafa489 Compare December 11, 2019 15:33

gnzlbg approved these changes Dec 11, 2019

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 11, 2019

JohnTitor mentioned this pull request Dec 12, 2019

Rollup of 8 pull requests #67246

Merged

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 12, 2019

bors merged commit fafa489 into rust-lang:master Dec 12, 2019

edsrzf mentioned this pull request Jan 12, 2020

ty: Simplify Box<T> to *T mozilla/cbindgen#451

Closed

jonas-schievink mentioned this pull request Jan 30, 2020

rust/RELEASES.md doesn't list the Box<T> FFI change for 1.41.0 #68676

Closed

jyn514 mentioned this pull request Feb 12, 2021

Remove unnecessary lint allow attrs on example #82023

Merged

briansmith reviewed Oct 20, 2022

View reviewed changes

plicease mentioned this pull request Oct 23, 2022

Examples should use now blessed Box<T> where appropriate? PerlFFI/FFI-Platypus-Lang-Rust#29

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clarify `Box<T>` representation and its use in FFI #62514

Clarify `Box<T>` representation and its use in FFI #62514

stephaneyfx commented Jul 9, 2019

rust-highfive commented Jul 9, 2019

CryZe Jul 9, 2019

stephaneyfx Jul 9, 2019

stephaneyfx left a comment

dtolnay commented Jul 12, 2019

Alexendoo commented Jul 24, 2019

gnzlbg commented Jul 24, 2019

gnzlbg commented Jul 24, 2019

stephaneyfx commented Jul 24, 2019

joshtriplett commented Jul 25, 2019 •

edited

Loading

joshtriplett commented Jul 25, 2019

rfcbot commented Jul 25, 2019 •

edited by sfackler

Loading

nikomatsakis commented Jul 25, 2019

nikomatsakis left a comment

nikomatsakis Dec 10, 2019

stephaneyfx Dec 11, 2019

nikomatsakis Dec 10, 2019

stephaneyfx commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

nikomatsakis commented Dec 11, 2019

nikomatsakis commented Dec 11, 2019

gnzlbg left a comment

nikomatsakis commented Dec 11, 2019

bors commented Dec 11, 2019

RalfJung commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

RalfJung commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

bors commented Dec 12, 2019

briansmith Oct 20, 2022

RalfJung Oct 20, 2022

RReverser Oct 20, 2022

	//! `T*`). This means that you have Rust code which passes ownership of a
	//! `T*`). This means that you can have Rust code which passes ownership of a

Clarify Box<T> representation and its use in FFI #62514

Clarify Box<T> representation and its use in FFI #62514

Conversation

stephaneyfx commented Jul 9, 2019

rust-highfive commented Jul 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephaneyfx left a comment

Choose a reason for hiding this comment

dtolnay commented Jul 12, 2019

Alexendoo commented Jul 24, 2019

gnzlbg commented Jul 24, 2019

gnzlbg commented Jul 24, 2019

stephaneyfx commented Jul 24, 2019

joshtriplett commented Jul 25, 2019 • edited Loading

joshtriplett commented Jul 25, 2019

rfcbot commented Jul 25, 2019 • edited by sfackler Loading

nikomatsakis commented Jul 25, 2019

nikomatsakis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephaneyfx commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

nikomatsakis commented Dec 11, 2019

nikomatsakis commented Dec 11, 2019

gnzlbg left a comment

Choose a reason for hiding this comment

nikomatsakis commented Dec 11, 2019

bors commented Dec 11, 2019

RalfJung commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

RalfJung commented Dec 11, 2019

gnzlbg commented Dec 11, 2019

bors commented Dec 12, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Clarify `Box<T>` representation and its use in FFI #62514

Clarify `Box<T>` representation and its use in FFI #62514

joshtriplett commented Jul 25, 2019 •

edited

Loading

rfcbot commented Jul 25, 2019 •

edited by sfackler

Loading