Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC for Vec::append_from_within() #2714

Closed
wants to merge 8 commits into from
Closed

Conversation

Shnatsel
Copy link
Member

@Shnatsel Shnatsel commented Jun 21, 2019

Rendered

This addition is quite trivial. However, in the pre-RFC I've learned that there are many approaches to solving this, so I'm opening an RFC to provide a clear rationale and get some input on the final design.

Prototype implementation in playground

@SimonSapin
Copy link
Contributor

This feels very ad-hoc. I wonder if exposing some intermediate building block would help? Something like:

impl<T> Vec<T> {
    /// Returns the result of `self.as_mut_slice()` together with the rest of the capacity.
    ///
    /// The two slices have length `self.len()` and `self.capacity() - self.len()` respectively.
    pub fn with_uninit_capacity(&mut self) -> (&mut [T], &mut [MaybeUninit<T>]) {}
}

impl<T> MaybeUninit<T> {
    /// Panic if the two slices have different lengths.
    pub fn copy_from_slice(this: &mut [Self], other: &[T]) where T: Copy {}
    pub fn clone_from_slice(this: &mut [Self], other: &[T]) where T: Clone {}
}

Then, append_from_within could be implemented outside of the standard library with the only unsafe code being a call to Vec::set_len.

@Shnatsel
Copy link
Member Author

I do not believe the MaybeUninit solution is optimal. For one, MaybeUninit::copy_from_slice actually guarantees that the buffer is initialized after its completion, but this is not reflected in the type system. Which is why this still requires an unsafe set_len() operation on the vector. Also, performance characteristics of this solution when used in a loop are exactly the same as for append_from_within. Finally, two similar but entirely safe solutions are already discussed in detail in "Alternatives" section.

Note that the desired end result in all three motivating examples is actually repeat_tail(&mut self, elements: usize, times: usize), while append_from_within() intended as a minimum viable safe building block for implementing the former.

While a hypothetical more general solutions could serve more use cases (see the pre-RFC for a motivating example from Claxon for a more general abstraction), it would take a long time to design and implement, and it would not entirely supersede append_from_within(). This function is more discoverable and easier to apply than a generic abstraction by virtue of being much simpler and also by being the counterpart of an existing method on slice. And in this case having a low cognitive load is important: as illustrated by the motivating examples, people are repeatedly tempted to hand-roll this and keep getting it wrong.

@Ixrec
Copy link
Contributor

Ixrec commented Jun 22, 2019

Could we add a self-contained description of why this method is so difficult to implement correctly? Two of the three example links didn't seem to ever describe the bug, and the third explained only at the end of a very long post about a bunch of other security stuff that the broken code forgot to check if an argument was 0 (which imo is a perfectly good answer, if that is correct, it just needs to be easier to find).

@Shnatsel
Copy link
Member Author

The code in rust stdlib was not checking for overflow in capacity calculation. Here's the fix.

DEFLATE decoders forgot to check that one of the arguments is not 0. Here is a simplified version of the code they used that illustrates the problem:

/// Repeats `repeating_fragment_len` bytes at the end of the buffer
/// until an additional `num_bytes_to_fill` in the buffer is filled
pub fn decode_rle_vulnerable(buffer: &mut Vec<u8>, repeating_fragment_len: usize, num_bytes_to_fill: usize) {
    buffer.reserve(num_bytes_to_fill); // allocate required memory immediately, it's faster this way
    unsafe { // set length of the buffer up front so we can set elements in a slice instead of pushing, which is slow
        buffer.set_len(buffer.len() + num_bytes_to_fill);
    }
    for i in (buffer.len() - num_bytes_to_fill)..buffer.len() {
        self.buffer[i] = self.buffer[i - repeating_fragment_len];
    }
}

If you pass repeating_fragment_len set to 0 it will expose contents of uninitialized memory in the output. Both inflate and libflate crates have this bug even though they were implemented independently.

@Ixrec
Copy link
Contributor

Ixrec commented Jun 22, 2019

Thanks, that makes perfect sense now.

@burdges
Copy link

burdges commented Jun 22, 2019

At present, you'll need separate append_copies_from_within and append_clones_from_within, because the copy version require only Vec::set_len, while the Clone versions requires first Vec::as_mut_ptr and then later Vec::set_len.

In the simple versions, If Vec<(F,Arc<T>)> panics mid-way, due to F::Clone panicing, then the Arc<T> gets left with an incorrect ref count, which sounds okay but annoying. If otoh Mutex<Vec<F>> panics then PoisonError permits access while the Vec<F> contains uninitialized data. You could do only one Vec::set_len at the end in the Clone version, but doing them after each Clone works too.

If you wait for specialization, then presumably append_from_within can be specialized for Copy types, so I'd kinda suggest waiting for specialization.

Oops! Just noticed you want this restricted to Copy types throughout.

@Shnatsel
Copy link
Member Author

Shnatsel commented Jun 22, 2019

I was using slice::copy_within() as reference, which is restricted to Copy types and does not come with an equivalent function for Clone types.

I am open to adding append_clones_from_within() if there are use cases for it.

@scottmcm
Copy link
Member

scottmcm commented Jun 23, 2019

On copy-vs-clone, I think Copy is fine for now. It could be expanded compatibly to Clone in the future if desired, especially once we eventually get specialization and could thus promise that it's just a memcpy when the input is Copy.

I was using slice::copy_within() as reference

I like this plan -- when I saw it get stabilized in 1.37 I also thought of this scenario, as it implicitly set a precedent for API design for things that treat part of themselves as input.

(Now that it's simplified so much and not blazing its own trail -- something like .as_fixed_capacity() was a much bigger addition -- I'd personally consider it small enough to just be a PR instead of an RFC, though of course I'm not on libs so my opinion isn't all that important here.)

@scottmcm scottmcm added T-libs-api Relevant to the library API team, which will review and decide on the RFC. A-unsafe Unsafe related proposals & ideas A-collections Proposals about collection APIs labels Jun 23, 2019
@Shnatsel
Copy link
Member Author

Shnatsel commented Jun 29, 2019

@WanzenBug and me have ported libflate to a 100% safe RLE implementation using this abstraction. That let us remove an unsound bespoke unsafe block.

It also improved end-to-end performance by 5% 10% over the unsafe, unsound implementation, and by much more over the best safe implementation possible without append_from_within().

to guide-level explanation, as requested in a comment
@XAMPPRocky
Copy link
Member

XAMPPRocky commented Jul 11, 2019

From reading the RFC I found that Vec::repeat_part was a more clear description and API for the desired operation than append_from_within despite being more generic. That coupled with the potential to allow for eliding the checks makes it more compelling to me than append_from_within.

bikeshed I think switching the position of the parameters of repeat_part would better match the name. e.g. repeat_part(&mut self, times: usize, src: Range) as you first specify how may times you're going to repeat and then what part you want to repeat.

@comex
Copy link

comex commented Jul 11, 2019

I don't think the name repeat_part is clear at all; it sounds like it might return a repeated value without modifying the Vec, or perhaps replace the part with repeated copies of itself (as opposed to appending those copies at the end).

@XAMPPRocky
Copy link
Member

So I went through Vec's current methods to see if there was existing terminology that would make more sense, and I think the extend terminology might suit this better. E.g. append_from_within -> extend_from_within, repeat_part -> extend_from_part.

@alexcrichton
Copy link
Member

Thanks for the RFC here @Shnatsel and sorry for the delay!

FWIW the libs team tends to not require RFCs for API additions like this nowadays. This looks like a pretty reasonable feature addition to send in a PR, and we can have more discussion there if necessary (but it seems like a lot has happened here already!) and we can have another round of discussion before stabilization.

Does that sound alright with you? If so feel free to adapt the discussion here to a PR sent to libstd, and we can close this and open a tracking issue and copy over any remaining unresolved questions and such.

@Shnatsel
Copy link
Member Author

Shnatsel commented Sep 3, 2019

Sounds good to me! I'll close this as soon as I create a PR.

@Shnatsel
Copy link
Member Author

Shnatsel commented Sep 3, 2019

BTW: could we get the README in this repo updated? It states that additions to std require an RFC.

@alexcrichton
Copy link
Member

Sure! Seems reasonable to update the README

@Ixrec
Copy link
Contributor

Ixrec commented Oct 11, 2019

@Shnatsel did you ever get a chance to make a PR?

@Shnatsel
Copy link
Member Author

No, I didn't get around to it yet. I got sidetracked by other projects.

@Shnatsel
Copy link
Member Author

For future reference, this would also eliminate the only unsafe block in ruzstd, the zstd decoder in Rust:

https://github.com/KillingSpark/zstd-rs/blob/e521dfdb4005b9d6d9556a8e3df9db445a5d038d/src/decoding/decodebuffer.rs#L135

@KodrAus KodrAus added the Libs-Tracked Libs issues that are tracked on the team's project board. label Jul 29, 2020
@WaffleLapkin
Copy link
Member

WaffleLapkin commented Nov 11, 2020

@Shnatsel am I right, that feature is still unimplemented? If so, I could make a PR.

@Shnatsel
Copy link
Member Author

am I right, that feature is still unimplemented?

Yes, that is correct.

After a bunch of experimentation with this in a crate I am convinced that the API described here is the right level of abstraction to expose.

The implementation is already written and tested, see https://github.com/WanzenBug/rle-decode-helper/blob/690742a0de158d391b7bde1a0c71cccfdad33ab3/src/lib.rs#L74

All you need to do is make a PR for the standard library adding this function. I'd very much appreciate this since I keep getting sidetracked by other projects.

WaffleLapkin added a commit to WaffleLapkin/rust that referenced this pull request Jan 31, 2021
Implement <rust-lang/rfcs#2714>, changes from the RFC:
- Rename the method `append_from_within` => `extend_from_within`
- Loose :Copy bound => :Clone
- Specialize in case of :Copy

This commit also adds `Vec::split_at_spare` private method and use it to implement
`Vec::spare_capacity_mut` and `Vec::extend_from_within`. This method returns 2
slices - initialized elements (same as `&mut vec[..]`) and uninitialized but
allocated space (same as `vec.spare_capacity_mut()`).
bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 2, 2021
…r=KodrAus

add `Vec::extend_from_within` method under `vec_extend_from_within` feature gate

Implement <rust-lang/rfcs#2714>

### tl;dr

This PR adds a `extend_from_within` method to `Vec` which allows copying elements from a range to the end:

```rust
#![feature(vec_extend_from_within)]

let mut vec = vec![0, 1, 2, 3, 4];

vec.extend_from_within(2..);
assert_eq!(vec, [0, 1, 2, 3, 4, 2, 3, 4]);

vec.extend_from_within(..2);
assert_eq!(vec, [0, 1, 2, 3, 4, 2, 3, 4, 0, 1]);

vec.extend_from_within(4..8);
assert_eq!(vec, [0, 1, 2, 3, 4, 2, 3, 4, 0, 1, 4, 2, 3, 4]);
```

### Implementation notes

Originally I've copied `@Shnatsel's` [implementation](https://github.com/WanzenBug/rle-decode-helper/blob/690742a0de158d391b7bde1a0c71cccfdad33ab3/src/lib.rs#L74) with some minor changes to support other ranges:
```rust
pub fn append_from_within<R>(&mut self, src: R)
where
    T: Copy,
    R: RangeBounds<usize>,
{
    let len = self.len();
    let Range { start, end } = src.assert_len(len);;

    let count = end - start;
    self.reserve(count);
    unsafe {
        // This is safe because `reserve()` above succeeded,
        // so `self.len() + count` did not overflow usize
        ptr::copy_nonoverlapping(
            self.get_unchecked(src.start),
            self.as_mut_ptr().add(len),
            count,
        );
        self.set_len(len + count);
    }
}
```

But then I've realized that this duplicates most of the code from (private) `Vec::append_elements`, so I've used it instead.

Then I've applied `@KodrAus` suggestions from rust-lang#79015 (comment).
@WaffleLapkin
Copy link
Member

Since rust-lang/rust#79015 was merged, I guess this PR can be closed?

@Shnatsel Shnatsel closed this Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-collections Proposals about collection APIs A-unsafe Unsafe related proposals & ideas Libs-Tracked Libs issues that are tracked on the team's project board. T-libs-api Relevant to the library API team, which will review and decide on the RFC.
Projects
None yet
Development

Successfully merging this pull request may close these issues.