Unclear lifetime error in closure producing a future

[95th]

Code:

use std::future::Future;

pub async fn bar() {
    foo(|x| baz(x)).await;
}

pub async fn baz(x: &u8) -> bool {
    if *x == 1 {
        false
    } else {
        true
    }
}

pub async fn foo<F, T>(f: F) -> bool
where
    F: Fn(&u8) -> T,
    T: Future<Output = bool>,
{
    f(&32).await
}

produces following error:

error: lifetime may not live long enough
 --> src/lib.rs:4:13
  |
4 |     foo(|x| baz(x)).await;
  |          -- ^^^^^^ returning this value requires that `'1` must outlive `'2`
  |          ||
  |          |return type of closure is impl std::future::Future
  |          has type `&'1 u8`

Is this error correct? If yes, why?

following also doesn't work:

pub async fn bar() {
    foo(baz).await;
}

error (still not clear):

error[E0271]: type mismatch resolving `for<'r> <for<'_> fn(&u8) -> impl std::future::Future {baz} as std::ops::FnOnce<(&'r u8,)>>::Output == _`
  --> src/lib.rs:4:5
   |
4  |     foo(baz).await;
   |     ^^^ expected bound lifetime parameter, found concrete lifetime
...
15 | pub async fn foo<F, T>(f: F) -> bool
   |              --- required by a bound in this
16 | where
17 |     F: Fn(&u8) -> T,
   |                   - required by this bound in `foo`

meta:

$ rustc --version --verbose
rustc 1.45.0 (5c1f21c3b 2020-07-13)
binary: rustc
commit-hash: 5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2
commit-date: 2020-07-13
host: x86_64-unknown-linux-gnu
release: 1.45.0
LLVM version: 10.0

[SNCPlay42]

This note

return type of closure is impl std::future::Future

Is supposed to define the lifetime '2 referred to by the error message by showing it as a parameter of the return type, but '2 doesn't appear in the type anywhere, possibly because of the opaque impl Future type.

cc #74072

[nikomatsakis]

This code is invalid. The closure is given a &u8, which is only valid during the closure execution. But baz(x) captures x and embeds it into a future that is returned -- this future, when awaited, will (or could) attempt to use x, even though the closure itself has returned, and hence x is no longer valid.

The error message here is obviously not great. We have specialized errors for these kinds of cases in closures normally, I'm not sure why it didn't trigger in this case.

[nikomatsakis]

There isn't a great workaround that uses stable syntax to make this compile, interestingly. We discussed it some on Zulip.

The error message for a comparable scenario without async isn't that great either, actually, I am somewhat surprised (playground):

error: lifetime may not live long enough
 --> src/lib.rs:4:13
  |
4 |     foo(|x| baz(x))
  |          -- ^^^^^^ returning this value requires that `'1` must outlive `'2`
  |          ||
  |          |return type of closure is &'2 u8
  |          has type `&'1 u8`

[nikomatsakis]

Returning a reference to a local variable is somewhat better (playground):

error[E0515]: cannot return value referencing local variable `p`
 --> src/lib.rs:4:26
  |
4 |     foo(|x| { let p = 3; baz(&p) })
  |                          ^^^^--^
  |                          |   |
  |                          |   `p` is borrowed here
  |                          returns a value referencing data owned by the current function

[WinLinux1028]

This code is invalid. The closure is given a &u8, which is only valid during the closure execution. But baz(x) captures x and embeds it into a future that is returned -- this future, when awaited, will (or could) attempt to use x, even though the closure itself has returned, and hence x is no longer valid.

The error message here is obviously not great. We have specialized errors for these kinds of cases in closures normally, I'm not sure why it didn't trigger in this case.

If the Future which returned by f(&32) awaited out of &32's lifetime, I think so too.
But if not, this code is valid.

This code will work:

pub async fn bar() {
    println!("{:?}",foo(|x| baz(x)).await);
}

pub async fn baz(x: &u8) -> bool {
    if *x == 1 {
        false
    } else {
        true
    }
}

pub async fn foo<F, T>(f: F) -> bool
where
    F: Fn(&'static u8) -> T,
    T: Future<Output = bool>,
{
    f(force_static(&32)).await
}

fn force_static<T>(a: &T) -> &'static T {
    unsafe { &*(a as *const T) }
}

[eholk]

One of the things I find surprising here is when the error messages talk about '1 and '2. It's not really clear where these lifetimes would come from, although presumably they are some implied or inferred lifetime that's not written explicitly. Would it be possible to have rustc give some more information about where these come from, such as what values are borrowed with each lifetime?

[eholk]

@WinLinux1028 - That's interesting that the force_static makes it work. I wonder if we could do this automatically, since I think &32 has a static lifetime anyway, or at least can be promoted to one?

Also, is the force_static actually needed, since f already has a 'static bound on its parameter?

[eholk]

Yeah, it looks like @WinLinux1028's latest example works without force_static: playground link.

[estebank]

Triage: the current output is the same as for sync fns:

error: lifetime may not live long enough
 --> src/lib.rs:4:13
  |
4 |     foo(|x| baz(x)).await;
  |          -- ^^^^^^ returning this value requires that `'1` must outlive `'2`
  |          ||
  |          |return type of closure `impl Future` contains a lifetime `'2`
  |          has type `&'1 u8`

[eholk]

@rustbot label +AsyncAwait-polish

[estebank]

This should point at the bounds in foo and potentially suggest

pub async fn foo<'a, F, T>(f: F) -> bool
where
    F: Fn(&'a u8) -> T,
    T: Future<Output = bool> + 'a,

[estebank]

Current output of the original test case:

error[E0308]: mismatched types
  --> src/lib.rs:4:5
   |
4  |     foo(baz).await;
   |     ^^^^^^^^ one type is more general than the other
   |
   = note: expected trait `for<'r> <for<'r> fn(&'r u8) -> impl for<'r> Future<Output = bool> {baz} as FnOnce<(&'r u8,)>>`
              found trait `for<'r> <for<'r> fn(&'r u8) -> impl for<'r> Future<Output = bool> {baz} as FnOnce<(&'r u8,)>>`
note: the lifetime requirement is introduced here
  --> src/lib.rs:17:19
   |
17 |     F: Fn(&u8) -> T,
   |                   ^

[vincenzopalazzo]

I will try to resurrect this with a deep dive into it

@rustbot claim

[vincenzopalazzo]

Okay, after spending some time on it, I've come up with a summary.

This is the complete example that includes the error, and also explains why this code will not work.

use std::future::Future;

pub async fn bar() {
    foo(|x| baz(x)).await;
}

pub async fn baz(x: &u8) -> bool {
    if *x == 1 {
        false
    } else {
        true
    }
}

/// From Nico
///
/// This code is invalid. The closure is given a &u8, which is only valid
/// during the closure execution. But baz(x) captures x and embeds it
/// into a future that is returned -- this future, when awaited, will
/// (or could) attempt to use x, even though the closure itself has
/// returned, and hence x is no longer valid.
///
/// The error message here is obviously not great. We have specialized
/// errors for these kinds of cases in closures normally, I'm not sure
/// why it didn't trigger in this case.
pub async fn foo<F, T>(f: F) -> bool
where
    F: Fn(&u8) -> T,
    T: Future<Output = bool>,
{
    f(&32).await
}

fn main() {}

But it seems that, upon further reflection, the point made by @WinLinux1028 and analyzed by @eholk is correct. I wasn't able to find a counterexample where this code doesn't work. However, I do need to say that the solution proposed by @estebank is more attractive to me because it exactly describes what the callback does (since I don't like using 'static lifetimes).

Here is the revised code:

use std::future::Future;

pub async fn bar() {
    foo(|x| baz(x)).await;
}

pub async fn baz(x: &u8) -> bool {
    if *x == 1 {
        false
    } else {
        true
    }
}

pub async fn foo<'a, F, T>(f: F) -> bool
where
    F: Fn(&'a u8) -> T,
    T: Future<Output = bool>,
{
    f(&32).await
}

fn main() {}

Furthermore, both synchronous and asynchronous code produce the same error. Therefore, I believe that this issue should be addressed for both types of code, rather than just the asynchronous function.

As a result, I intend to propose this matter for discussion at the upcoming async triage meeting, in order to receive feedback on the solution suggested by @eholk for inclusion in the compiler.

I did not check in deep, but this looks like another instance #102201

@rustbot label -AsyncAwait-Triaged +I-async-nominated

[eholk]

We revisited this in @rust-lang/wg-async triage today. We think the behavior of the compiler is correct, but there's room to improve the error message. We'll spend some more time iterating on what an improved error message would look like.

[traviscross]

Here's the minimal reproduction of this issue:

fn foo<F: Fn(&()) -> T, T>(f: F) {
    f(&());
}

// error: lifetime may not live long enough
fn test() {
    foo(|x: &()| x);
    //      -  - ^ returning this value requires that `'1` must outlive `'2`
    //      |  |
    //      |  return type of closure is &'2 ()
    //      let's call the lifetime of this reference `'1`
}

Playground link

(Thanks to @compiler-errors.)

[timvermeulen]

This should point at the bounds in foo and potentially suggest

pub async fn foo<'a, F, T>(f: F) -> bool
where
    F: Fn(&'a u8) -> T,
    T: Future<Output = bool> + 'a,

This restricts foo to only call f with references that are valid for any arbitrary lifetime 'a, would that ever actually be useful? I expect this pattern to mostly arise when the function needs to call the closure with a reference to a local variable.