Prevent Vec::drain_filter from double dropping on panic #61224

aloucks · 2019-05-27T03:03:16Z

The changes in this PR prevent leaking and double-panicking in addition to double-drop.

Tracking issue: #43244

rust-highfive · 2019-05-27T03:03:19Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @cramertj (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

rust-highfive · 2019-05-27T03:24:29Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:00cb62a0:start=1558926460815491735,finish=1558926461599598202,duration=784106467
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:05:00] tidy error: /checkout/src/liballoc/vec.rs:2810: trailing whitespace
[00:05:05] some tidy checks failed
[00:05:05] 
[00:05:05] 
[00:05:05] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:05:05] 
[00:05:05] 
[00:05:05] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:05:05] Build completed unsuccessfully in 0:01:17
[00:05:05] Build completed unsuccessfully in 0:01:17
[00:05:05] Makefile:67: recipe for target 'tidy' failed
[00:05:05] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:015996d7
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Mon May 27 03:12:58 UTC 2019
---
travis_time:end:21f6eb60:start=1558926779115216981,finish=1558926779121200896,duration=5983915
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:003e6c0a
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:007a9ce8
travis_time:start:007a9ce8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:3497a48c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Fixes: rust-lang#60977

Centril · 2019-05-27T04:41:40Z

r? @RalfJung

src/liballoc/vec.rs

RalfJung · 2019-05-27T07:01:43Z

Cc @rust-lang/libs would appreciate if one of you could take over review, this is a bit outside my comfort zone.

src/liballoc/tests/vec.rs

RalfJung · 2019-05-27T07:04:57Z

src/liballoc/vec.rs

                    let del = self.del;
                    let src: *const T = &v[i];
                    let dst: *mut T = &mut v[i - del];
-                    // This is safe because self.vec has length 0
-                    // thus its elements will not have Drop::drop
-                    // called on them in the event of a panic.


Is this comment no longer true? Why is it safe instead then?

I believe the intent was to leak instead of double drop, but it didn't quite work. I've done some minor refactoring and added additional comments.

It's safe now because there are additional checks in DrainFilter::drop that perform the cleanup in the event of a panic in the filter predicate.

ebkalderon · 2019-06-25T15:47:21Z

Is this PR still being worked on? I'd love to see this unsound behavior disappear.

RalfJung · 2019-06-25T15:50:35Z

From what I can see, this is waiting for a reviewer. As mentioned above, I don't feel comfortable reviewing this. Cc @rust-lang/libs

Centril · 2019-06-25T15:56:02Z

r? @gankro

Gankra · 2019-06-25T20:36:02Z

sigh fine

aloucks · 2019-07-01T21:02:09Z

@gankro Any feedback?

Gankra · 2019-07-02T13:44:51Z

yeah sorry, i plan to review this this morning, just caught me coming off a very draining work trip and leading into a long weekend.

Gankra

The approach looks good, assuming that panicking is guaranteed to behave as assumed here. (compiler devs have always been hand-wavy to me about this topic).

Mostly some cosmetic concerns, as "the right way to write unsafe code" has changed a fair bit since this code landed, and we should take the opportunity to bring this code up to snuff. In particular, creating a slice for the whole buffer when we're de-initializing parts of it is very dubious.

Gankra · 2019-07-05T17:34:58Z

src/liballoc/vec.rs

+                    // but was not due to a panic in the filter predicate. This is
+                    // implemented via drop so that it's guaranteed to run even in
+                    // the event of a panic while consuming the remainder of the
+                    // DrainFilter.


(currently checking with the lang team that we guarantee that this works)

cc @rust-lang/lang @matthewjasper @arielb1

specifically: if we panic in a destructor and we weren't yet panicking:

we unwind the drop impl, dropping its locals

then drop_in_place self's fields as normal

then proceed into a normal panic, unwinding the stack

(the middle point I'm not sure is necessary here, but is just my understanding of what is supposed to happen.)

Cc rust-lang/nomicon#135

cc also rust-lang/reference#514

* then drop_in_place `self`'s fields as normal

This is indeed what we're currently doing:

rust/src/test/mir-opt/unusual-item-types.rs

Lines 48 to 74 in 254f201

// START rustc.ptr-real_drop_in_place.std__vec__Vec_i32_.AddMovesForPackedDrops.before.mir

// bb0: {

// goto -> bb7;

// }

// bb1: {

// return;

// }

// bb2 (cleanup): {

// resume;

// }

// bb3: {

// goto -> bb1;

// }

// bb4 (cleanup): {

// goto -> bb2;

// }

// bb5 (cleanup): {

// drop(((*_1).0: alloc::raw_vec::RawVec<i32>)) -> bb4;

// }

// bb6: {

// drop(((*_1).0: alloc::raw_vec::RawVec<i32>)) -> [return: bb3, unwind: bb4];

// }

// bb7: {

// _2 = &mut (*_1);

// _3 = const <std::vec::Vec<i32> as std::ops::Drop>::drop(move _2) -> [return: bb6, unwind: bb5];

// }

// END rustc.ptr-real_drop_in_place.std__vec__Vec_i32_.AddMovesForPackedDrops.before.mir

rust-lang/reference#348 (comment)

@gankro your understanding is correct. If drop panics:

we unwind the drop impl, dropping its locals,

then drop_in_place self's fields as normal (where normal means in field declaration order - first field is dropped before second field is dropped),

then we continue unwinding Drop::drop's caller stack

If dropping a value in Drop::drop stack while its unwinding panics, or if dropping one of the type's fields panics, then we have a double-drop, and the process is guarantee to abort for all double drops.

Note also that the panic doesn't need to originate in the Drop::impl, this one can succeed, but then the fields are dropped in declaration order, and one of them can panic. When this happens, the remaining fields are still dropped in place, so if a second one of them panics, then you have a double drop. If the Drop::impl does not panic, but dropping one of the fields panics, and you don't get a double drop, you are guaranteed that the field that panicked is the only one that wasn't properly dropped.

EDIT: for all practical purposes, if dropping a value panics, you should treat that value as dropped, since "most" of the value will have been dropped in the process (at most one field wasn't dropped).

Gankra · 2019-07-05T17:58:52Z

src/liballoc/vec.rs

+                            let dst: *mut T = &mut v[i - del];
+                            ptr::copy_nonoverlapping(src, dst, 1);
+                        }
+                    }


I don't understand why this while loop exists. Surely this should just be:

if self.drain.idx < self.drain.old_len { // It looks like `pred` panicked, so we didn't process all the elements. // This is a pretty messed up state, and there isn't really an obviously right // thing to do (and we don't want to keep trying to execute `pred`). So we // just backshift all the unprocessed elements and tell the vec that they still // exist, hoping that doesn't mess up anyone further along in the panic. let idx = self.drain.idx; let num_deleted = self.drain.del; let tail_len = self.drain.old_len - idx; let ptr = self.drain.vec.as_mut_ptr(); if num_deleted > 0 { ptr.add(idx).copy_to(ptr.add(idx - num_deleted), tail_len); } } self.drain.vec.set_len(self.drain.old_len - self.drain.del);

(Here I modernized the code a bit to use the newer raw pointer APIs, and some clearer names. It would be a nice cleanup to this code if you also did that to the Iterator's fields and its next impl as well, although not a blocker.)

I incorporated this in the latest commit. I also consolidated the the self.drain.del check into the parent if and made the src and dst pointers explicit while still using the more modern pointer APIs.

Gankra · 2019-07-05T18:01:46Z

src/liballoc/vec.rs

@@ -2751,6 +2752,7 @@ pub struct DrainFilter<'a, T, F>
    del: usize,
    old_len: usize,
    pred: F,
+    panic_flag: bool,
 }


This has grown enough fields that they should really be documented. Either individually here, or with a high-level description of what we're trying to do in the implementation of next.

I added docs to each field in the latest commit.

Gankra · 2019-07-05T18:04:13Z

src/liballoc/vec.rs

-                if (self.pred)(&mut v[i]) {
+                self.panic_flag = true;
+                let drained = (self.pred)(&mut v[i]);
+                self.panic_flag = false;


I have a terrible compulsion to try to encode this state in some magic combination of old_len/idx/del, but this is probably clearest, and easiest for llvm to evaporate when it sees pred can't unwind.

I considered that route initially but came to the conclusion that even if it's possible, the simple flag would be much easier to understand and maintain.

Centril · 2019-07-08T13:32:06Z

@bors delegate=Gankro

bors · 2019-07-08T13:32:07Z

✌️ @gankro can now approve this pull request

Gankra · 2019-07-08T13:32:25Z

@bors r+

bors · 2019-07-08T13:32:26Z

📌 Commit df5b32e has been approved by Gankro

bors · 2019-07-08T15:40:03Z

⌛ Testing commit df5b32e with merge 53cb0008f8a448df6461773fd42a8c81a77637dd...

bors · 2019-07-08T19:03:01Z

💔 Test failed - checks-azure

aloucks · 2019-07-08T21:03:38Z

@gankro @Centril Retry?

macOS dist-x86_64-apple
Started: 7/8/2019, 11:40:37 AM
Agent: Hosted Agent
3h 22m 8s

Job issues
·
1 error
The agent: Hosted Agent lost communication with the server. Verify the machine is running and has a healthy network connection. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

Centril · 2019-07-08T21:38:04Z

@bors retry

bors · 2019-07-08T22:03:35Z

⌛ Testing commit df5b32e with merge 09ab31b...

Prevent Vec::drain_filter from double dropping on panic Fixes: #60977 The changes in this PR prevent leaking and double-panicking in addition to double-drop. Tracking issue: #43244

bors · 2019-07-09T01:21:04Z

☀️ Test successful - checks-azure, checks-travis, status-appveyor
Approved by: Gankro
Pushing 09ab31b to master...

This PR reverts rust-lang#48065, which aimed to optimize `Vec::retain` by making use of `Vec::drain_filter`. Unfortunately at that time, `drain_filter` was unsound. The soundness hole in `Vec::drain_filter` was fixed in rust-lang#61224 by guaranteeing that cleanup logic runs via a nested `Drop`, even in the event of a panic. Implementing this nested drop affects codegen (apparently?) and results in slower code. Fixes rust-lang#65970

Restore original implementation of Vec::retain This PR reverts rust-lang#48065, which aimed to optimize `Vec::retain` by making use of `Vec::drain_filter`. Unfortunately at that time, `drain_filter` was unsound. The soundness hole in `Vec::drain_filter` was fixed in rust-lang#61224 by guaranteeing that cleanup logic runs via a nested `Drop`, even in the event of a panic. Implementing this nested drop affects codegen (apparently?) and results in slower code. Fixes rust-lang#65970

rust-highfive assigned cramertj May 27, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 27, 2019

aloucks force-pushed the drain_filter branch from 8b1aa3a to 9f97fb3 Compare May 27, 2019 03:06

Prevent Vec::drain_filter from double dropping on panic

84ae969

Fixes: rust-lang#60977

aloucks force-pushed the drain_filter branch from 9f97fb3 to 84ae969 Compare May 27, 2019 03:27

rust-highfive assigned RalfJung and unassigned cramertj May 27, 2019

bjorn3 requested changes May 27, 2019

View reviewed changes

src/liballoc/vec.rs Outdated Show resolved Hide resolved

RalfJung reviewed May 27, 2019

View reviewed changes

src/liballoc/tests/vec.rs Show resolved Hide resolved

RalfJung reviewed May 27, 2019

View reviewed changes

aloucks added 4 commits May 27, 2019 11:30

Fix formatting nit

17a517a

Add drain_filter_unconsumed test

53d46ae

Refactor DrainFilter::next and update comments

b13ae65

Disable drain_filter tests that require catch_unwind on miri

f5ab031

jonas-schievink added the T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. label Jun 6, 2019

rust-highfive assigned Gankra and unassigned RalfJung Jun 25, 2019

Gankra suggested changes Jul 5, 2019

View reviewed changes

Remove while loop in DrainFilter::drop and add additional docs

a4a6a67

aloucks force-pushed the drain_filter branch from 67830c2 to a4a6a67 Compare July 7, 2019 17:44

This comment has been minimized.

Sign in to view

Centril closed this Jul 8, 2019

Centril reopened this Jul 8, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 8, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 8, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 8, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 9, 2019

bors merged commit df5b32e into rust-lang:master Jul 9, 2019

niklasf mentioned this pull request Jul 17, 2019

Add ArrayVec::drain_filter() bluss/arrayvec#95

Closed

crgl mentioned this pull request Nov 1, 2019

Using retain() on a vector a lot slower in version 1.38.0 than 1.36.0 #65970

Closed

This was referenced Dec 12, 2019

Tracking issue for Vec::extract_if and LinkedList::extract_if #43244

Open

Vec::retain leaks items when predicate panics #52267

Closed

aloucks mentioned this pull request Dec 14, 2019

Restore original implementation of Vec::retain #67300

Merged

	// START rustc.ptr-real_drop_in_place.std__vec__Vec_i32_.AddMovesForPackedDrops.before.mir
	// bb0: {
	// goto -> bb7;
	// }
	// bb1: {
	// return;
	// }
	// bb2 (cleanup): {
	// resume;
	// }
	// bb3: {
	// goto -> bb1;
	// }
	// bb4 (cleanup): {
	// goto -> bb2;
	// }
	// bb5 (cleanup): {
	// drop(((*_1).0: alloc::raw_vec::RawVec<i32>)) -> bb4;
	// }
	// bb6: {
	// drop(((*_1).0: alloc::raw_vec::RawVec<i32>)) -> [return: bb3, unwind: bb4];
	// }
	// bb7: {
	// _2 = &mut (*_1);
	// _3 = const <std::vec::Vec<i32> as std::ops::Drop>::drop(move _2) -> [return: bb6, unwind: bb5];
	// }
	// END rustc.ptr-real_drop_in_place.std__vec__Vec_i32_.AddMovesForPackedDrops.before.mir

Prevent Vec::drain_filter from double dropping on panic #61224

Prevent Vec::drain_filter from double dropping on panic #61224

Conversation

aloucks commented May 27, 2019 • edited Loading

rust-highfive commented May 27, 2019

rust-highfive commented May 27, 2019

Centril commented May 27, 2019

RalfJung commented May 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebkalderon commented Jun 25, 2019

RalfJung commented Jun 25, 2019

Centril commented Jun 25, 2019

Gankra commented Jun 25, 2019

aloucks commented Jul 1, 2019

Gankra commented Jul 2, 2019

Gankra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung Jul 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnzlbg Jul 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aloucks Jul 7, 2019 • edited Loading

Choose a reason for hiding this comment

This comment has been minimized.

Centril commented Jul 8, 2019

bors commented Jul 8, 2019

Gankra commented Jul 8, 2019

bors commented Jul 8, 2019

bors commented Jul 8, 2019

bors commented Jul 8, 2019

aloucks commented Jul 8, 2019

Centril commented Jul 8, 2019

bors commented Jul 8, 2019

bors commented Jul 9, 2019

aloucks commented May 27, 2019 •

edited

Loading

RalfJung Jul 5, 2019 •

edited

Loading

gnzlbg Jul 7, 2019 •

edited

Loading

aloucks Jul 7, 2019 •

edited

Loading