Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IntoIterator should be implemented for [T; N] not just references #25725

Closed
mdinger opened this issue May 22, 2015 · 37 comments
Closed

IntoIterator should be implemented for [T; N] not just references #25725

mdinger opened this issue May 22, 2015 · 37 comments
Labels
A-iterators Area: Iterators C-feature-accepted Category: A feature request that has been accepted pending implementation. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@mdinger
Copy link
Contributor

mdinger commented May 22, 2015

Currently IntoIterator is implemented for arrays of &'a mut [T; N] and &'a [T; N] but not for [T; N] making it only possible to collect() arrays of references, not arrays of values:

// These are the same in spite of `iter()` changing to `into_iter()`.
let vec_ref1: Vec<&i32> = [1, 2].iter().collect();
let vec_ref2: Vec<&i32> = [1, 2].into_iter().collect();

// This is what `into_iter()` should do but it doesn't actually work.
let vec: Vec<i32> = [1, 2].into_iter().collect();

This is something users are likely to run headlong into (probably accidentally) when testing out examples from the iterator docs. So, IntoIterator should be implemented so that last option is possible. The API stabilization RFC: rust-lang/rfcs#1105 states:

any breakage in a minor release ... it must always be possible to locally fix the problem through some kind of disambiguation that could have been done in advance

Meaning this would be allowed as a minor change in spite of [1, 2].into_iter() behavior changing because it could be changed beforehand to [1, 2].iter().

@mdinger
Copy link
Contributor Author

mdinger commented May 22, 2015

cc @bluss

@alexcrichton
Copy link
Member

This is tricky to implement for an arbitrary T as panics need to be handled where a moved-out element isn't dropped, and this also leads to a large number of iterators where the reference-based IntoIterator implementations just return a normal slice iterator.

@bluss
Copy link
Member

bluss commented May 26, 2015

For the record, crate arrayvec implements a by-value iterator using one type and a trait for a range of array sizes. I think it has solved all safety concerns, but I'll definitely look into what happens during panic again.

@mdinger
Copy link
Contributor Author

mdinger commented May 27, 2015

I take it this would/will be much easier once rust-lang/rfcs#1038 is implemented someday. Then at a minimum, this would be fixable shortly after that issue is.

@steveklabnik
Copy link
Member

The lack of this shows up in the beginnings of the book, and it's really annoying. I've shown off arrays, I've shown off loops, I want to show for over arrays. But I haven't introduced method syntax yet, so for e in a.iter() is something new. But you won't often be calling iter() explicitly when you're using Vecs, but we haven't introduced Vec yet either.

@mitchmindtree
Copy link
Contributor

mitchmindtree commented Jul 22, 2016

Just thought I'd give this a +1 and add that I come across this quite frequently. I even occasionally resort to once(a).chain(once(b)).chain(once(c)) during some calls to flat_map just so I can remain on the stack when the only other alternatives are vec![a, b, c] or having to import another crate like arrayvec.

This is tricky to implement for an arbitrary T as panics need to be handled where a moved-out element isn't dropped

At least 90% of my use cases for this are for arrays where the elements are Copy, which I think avoids this issue as a type cannot impl both Copy and Drop. Perhaps it could be useful to first provide an IntoIterator implementation that requires the Copy bound, which may be removed later down the track if we're able to come up with a solution for non-Copy types? Alternatively, perhaps a .copy_iter() that takes self where T: Copy might be useful until a more generalised .into_iter() can be implemented?

Edit: I just want to add that often, .iter().cloned() is not a possible solution when your array has a very short lifetime, i.e.

let (r, g, b) = rgb();
let rgbas: Vec<_> = alphas.into_iter()
    .flat_map(|a| [r, g, b, a].iter().cloned()) // This fails due to an insufficient lifetime
    // .flat_map(|a| [r, g, b, a]) // This would be sooo nice
    .collect();

@steveklabnik steveklabnik added T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. and removed A-libs labels Mar 24, 2017
@Mark-Simulacrum Mark-Simulacrum added the C-feature-request Category: A feature request, i.e: not implemented / a PR. label Jul 22, 2017
@haudan
Copy link

haudan commented Aug 10, 2017

+1, I also ran against this problem yesterday with this snippet (the Vec variant compiles, array does not). This is very surprising, especially for a beginner.

@kennytm
Copy link
Member

kennytm commented Aug 10, 2017

As discussed in #32871, we either need:

  1. impl Trait to be stable, so we could secretly use the impl IntoIterator for [T; 0] to [T; 32] trick (probably also need Named existentials and impl Trait variable declarations rfcs#2071 to name the associated type), or
  2. Get Const generics rfcs#2000 merged + implemented first (assuming stable code can use it without stabilizing const-generic, otherwise we need to count in stabilization time as well)

@dtolnay dtolnay added C-feature-accepted Category: A feature request that has been accepted pending implementation. and removed C-feature-request Category: A feature request, i.e: not implemented / a PR. labels Nov 15, 2017
bors added a commit that referenced this issue Mar 14, 2018
impl IntoIterator for arrays

This allows an array to move its values out through iteration.

This was attempted once before in #32871, but closed because the `IntoIter<T, [T; $N]>` type is not something we would want to stabilize.  However, RFC 2000's const generics (#44580) are now on the horizon, so we can plan on changing this to `IntoIter<T, const N: usize>` before stabilization.

Adding the `impl IntoIterator` now will allows folks to go ahead and iterate arrays in stable code.  They just won't be able to name the `array::IntoIter` type or use its inherent `as_slice`/`as_mut_slice` methods until they've stabilized.

Quite a few iterator examples were already using `.into_iter()` on arrays, getting auto-deref'ed to the slice iterator.  These were easily fixed by calling `.iter()` instead, but it shows that this might cause a lot of breaking changes in the wild, and we'll need a crater run to evaluate this.  Outside of examples, there was only one instance of in-tree code that had a problem.

Fixes #25725.

r? @alexcrichton
@orlp
Copy link
Contributor

orlp commented Oct 19, 2018

@kennytm impl Trait is now stable, so can this be implemented now?

@kennytm
Copy link
Member

kennytm commented Oct 20, 2018

@orlp I think you could give it a try. @rust-lang/libs WDYT?

(We'll also need the corresponding clippy lint rust-lang/rust-clippy#1565 to reduce the damage of inference breakage after stabilization.)

@orlp
Copy link
Contributor

orlp commented Oct 20, 2018

@kennytm Actually, IntoIterator needs member type IntoIter: Iterator, so we need existential types first, as you said :(

@orlp
Copy link
Contributor

orlp commented Oct 20, 2018

@kennytm I did come up with a way to do IntoIter<T> instead of IntoIter<[T; N]>: https://gist.github.com/orlp/2c0a704c3a30f6203fcaa33aca941f1c.

But I don't really think that helps as a temporary solution, because in the end we still want IntoIter<T, N>.

@RReverser
Copy link
Contributor

@orlp I changed your implementation slightly to avoid extra VarArray enum: https://gist.github.com/RReverser/bb9fd5250dba7e126f1df6dc6fb17662

Technically this is equivalent to your implementation, but avoids storing length twice both as an enum tag and as a separate field.

@orlp
Copy link
Contributor

orlp commented Oct 21, 2018

@RReverser I intentionally did not do that because it creates uninitialized memory, which is UB for T that have invalid bit patterns. Just creating uninitialized memory for those T is UB, even if you never access the uninitialized memory.

Here is a minimal example showing something that could happen: https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=23a761b2cda00df365dd11f128ee3409.

@RReverser
Copy link
Contributor

@orlp Hmm, these answers ("just creating uninitialized memory") don't seem to make much sense, but I haven't dug into ManuallyDrop changes they mention. Either way, you can replace uninitialized with zeroed which will avoid this issue.

@orlp
Copy link
Contributor

orlp commented Oct 21, 2018

@RReverser Did you check my minimal example? zeroed is just as bad.

@RReverser
Copy link
Contributor

@orlp That's a completely different issue and is related to layout optimisations, not zeroed or uninitialized behaviour.

@RReverser
Copy link
Contributor

RReverser commented Aug 5, 2019

@Diggsey vec.extend([...].iter().copied()) is fairly easy too and would avoid allocation, although it would be still nice to avoid doing that manually.

@rockmenjack
Copy link

Implicitly changing into_iter() to iter() for [T;N] also makes below statement inconsistent:

There are three common methods which can create iterators from a collection:
iter(), which iterates over &T.
iter_mut(), which iterates over &mut T.
into_iter(), which iterates over T.

@SimonSapin
Copy link
Contributor

@rockmenjack We’re not changing it that way. We want to add <[T; N] as IntoIterator>::into_iter, which doesn’t exist today but would make things more consistent per the pattern that you quote. But since it doesn’t exist, code that does let x: [T; N] = …; x.into_iter(); can compile through auto-ref using <&[T; N] as IntoIterator>::into_iter which gives an iterator of &T. So we’d be changing the meaning of valid code and likely break it.

@rockmenjack
Copy link

let x: [T; N] = …; x.into_iter(); can compile through auto-ref using <&[T; N] as IntoIterator>::into_iter which gives an iterator of &T. So we’d be changing the meaning of valid code and likely break it.

@SimonSapin this is what I meant inconsistent, call to [T;N]::into_iter() should be rejected by the compiler not auto-ref.
For a broader case, we shall block SomeTrait::some_fn(self) taking &T

@SimonSapin
Copy link
Contributor

Auto-ref in the general case is important and useful. Removing it would break lot of foo.bar() method calls where foo is owned and bar borrows &self, which would have to be rewritten as (&foo).bar(). It’s not only not allowed by the language’s stability promise, but I think it wouldn’t be desirable at all.

@rockmenjack
Copy link

rockmenjack commented Sep 3, 2019

Auto-ref in the general case is important and useful. Removing it would break lot of foo.bar() method calls where foo is owned and bar borrows &self, which would have to be rewritten as (&foo).bar(). It’s not only not allowed by the language’s stability promise, but I think it wouldn’t be desirable at all.

I agree with the case you mentioned, but that is bar(&self).
The function signature of into_iter() is:

fn into_iter(self) -> Self::IntoIter

user of into_iter might expect into_iter() taking the ownership of self which is not true at the moment for all types. And yeah, self can be &[T;N], just some what counterintuitive.

@SimonSapin
Copy link
Contributor

It’s important to look at what the type of self is. In impl<'a, T> IntoIterator for &'a [T; 4] for example, the Self type is &'a [T; 4], not [T; 4]. The into_iter method does take ownership of its receiver, but that receiver itself is a reference that borrows the array.

Having that impl is important so that you can write a for loop like let x: [T; 4] = …; for i in &x { … } when you want to to iterate over &T references to the array’s items.

Centril added a commit to Centril/rust that referenced this issue Oct 25, 2019
…=scottmcm

Add by-value iterator for arrays

This adds an iterator that can iterate over arrays by value, yielding all elements by value. However, **this PR does _not_ add a corresponding `IntoIterator` impl for arrays**. The `IntoIterator` impl needs some discussion about backwards-compatibility that should take place in a separate PR. With this patch, this code should work (but there is currently still a bug):

```rust
#![feature(array_value_iter)]
use std::array::IntoIter;

let arr = [1, 2, 3];
for x in IntoIter::new(arr) {
    println!("{}", x);
}
```

**TODO**:
- [x] Get initial feedback
- [x] Add tests
- [x] Figure out why stage1 produces weird bugs ([comment](rust-lang#62959 (comment)))
- [x] Add UI tests as mentioned [here](rust-lang#62959 (comment)) (will do that soon-ish)
- [x] Fix [this new bug](rust-lang#62959 (comment))

**Notes for reviewers**
- Is the use of `MaybeUninit` correct here? I think it has to be used due to the `Clone` impl which has to fill the dead array elements with something, but cannot fill it with a correct instance.
- Are the unit tests sufficient?

CC rust-lang#25725
@LukasKalbertodt
Copy link
Member

FYI: an by-value iterator for arrays was implemented in #62959. It is still unstable and missing the actual IntoIterator impl for arrays. That impl will (maybe) be added in #65819, but some discussion regarding backwards compatibility is necessary first. In case you want to chime in.

@m-ou-se
Copy link
Member

m-ou-se commented Apr 26, 2021

This trait implementation has now been added, with a small hack to avoid changing the meaning of array.into_iter() until Rust 2021: #84147

@m-ou-se m-ou-se closed this as completed Apr 26, 2021
m-ou-se added a commit to m-ou-se/rust that referenced this issue Apr 28, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 28, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 29, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 29, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 29, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 29, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
jackh726 added a commit to jackh726/rust that referenced this issue Apr 29, 2021
Point out that behavior might be switched on 2015 and 2018 too one day

Reword documentation to make it clear that behaviour can be switched on older editions too, one day in the future. It doesn't *have* to be switched, but I think it's good to have it as an option and re-evaluate it a few months/years down the line when e.g. the crates that showed up in crater were broken by different changes in the language already.

cc rust-lang#25725, rust-lang#65819, rust-lang#66145, rust-lang#84147 , and rust-lang#84133 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-iterators Area: Iterators C-feature-accepted Category: A feature request that has been accepted pending implementation. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet