Transform blocks into `async move` for more compatibility #143

SergioBenitez · 2021-02-03T06:12:37Z

This is effectively #125 with correct drop ordering. I've kept the changes minimal, hence the temporary allow(dead_code, unused_imports), which will be removed once this is approved and I sweep out the now-unused code.

This breaks #45/#97, which from a cursory reading, looks like is a result of hard-coding previous functionality into tracing, though do let me know if there's something I can do here to maintain that compatibility.

I should add: this makes quite a few more impls in Rocket compile without needing to manually expand the async-trait or other hackery. This fixes all of the cases related to #61 in Rocket. My guess, though I haven't tested, is that this fixes #126 as well.

SergioBenitez · 2021-02-03T09:19:22Z

cc @nightmared @hawkw re: #45/#97

dtolnay

Thanks, this is great. I am on board with this if we can get the regression rate low. Luckily I have a work codebase with thousands of uses of #[async_trait]. A significant fraction of them are broken by this PR, so I will get started on triaging those.

SergioBenitez · 2021-02-03T09:32:37Z

A significant fraction of them are broken by this PR, so I will get started on triaging those.

Awesome! Let me know if I can help with triaging some of those!

SergioBenitez · 2021-02-03T09:33:53Z

src/expand.rs

+//         let x = x;
+//         let (a, b) = __arg1;
+//
+//         __self + x


Suggested change

// __self + x

// __self + x + a + b

dtolnay · 2021-02-03T09:34:10Z

A lot of things that use self inside an expression macro fail with "borrow of moved value". Something like:

use async_trait::async_trait;

#[async_trait]
trait Trait {
    async fn f(self);
}

#[async_trait]
impl Trait for String {
    async fn f(self) {  // same for &mut self
        println!("{}", self);
    }
}

error[E0382]: borrow of moved value: `self`
  --> src/main.rs:11:27
   |
10 |     async fn f(self) {
   |                ---- value moved here
11 |         println!("{}", self);
   |                        ^^^^ value borrowed here after move
   |
   = note: move occurs because `self` has type `String`, which does not implement the `Copy` trait

SergioBenitez · 2021-02-03T09:37:19Z

Ah, that one's easy. So the SelfRenamer visitor only looks at Idents to rename all self to __self; the self there are arbitrary tokens in the macro call's token stream, so they're missed by the renamer.

We could visit the tokens in the token stream and rename those self's too.

Alas, I'd hope there was something more clever we could do with self. It's a real bummer to rename self to __self and then have to rename every other self in the block.

SergioBenitez · 2021-02-03T09:48:17Z

We could visit the tokens in the token stream and rename those self's too.

Now that I take a closer look at your original implementation, I see that this is exactly what you used to do.

dtolnay · 2021-02-03T09:50:52Z

Here is a "type mismatch" failure that appears to be quite common:

use async_trait::async_trait;

#[async_trait]
pub trait Trait {
    async fn f() -> Box<dyn Iterator<Item = ()>> {
        Box::new(std::iter::empty())
    }
}

error[E0271]: type mismatch resolving `<impl Future as Future>::Output == Box<(dyn Iterator<Item = ()> + 'static)>`
 --> src/main.rs:5:50
  |
5 |       async fn f() -> Box<dyn Iterator<Item = ()>> {
  |  __________________________________________________^
6 | |         Box::new(std::iter::empty())
7 | |     }
  | |_____^ expected trait object `dyn Iterator`, found struct `std::iter::Empty`
  |
  = note: expected struct `Box<(dyn Iterator<Item = ()> + 'static)>`
             found struct `Box<std::iter::Empty<_>>`
  = note: required for the cast to the object type `dyn Future<Output = Box<(dyn Iterator<Item = ()> + 'static)>> + Send`

dtolnay · 2021-02-03T10:06:33Z

I think those are the big two. Will check again after those are fixed. There is a third category which is some logging-related proc macros doing something tracing-like on the inner code but we can get that fixed to recognize Box::pin(async move {...}). As far as I can tell all the failures involve one of these three: self in a macro argument, trait object in return type, or hacky proc macro which is fine to break.

dtolnay · 2021-02-03T10:26:35Z

One more thing: this regresses #105 on compilers 1.46 and older, which is this failure in the 1.40 CI job:

error[E0425]: cannot find value `__self` in this scope
   --> tests/test.rs:599:13
    |
599 |             #[async_trait]
    |             ^^^^^^^^^^^^^^ not found in this scope

If it's easy, it would be good to get that worked around again even if new rustc fixed the bug. It was only a 1 line workaround in #105.

nightmared · 2021-02-03T15:21:46Z

I'll take a look at this. I guess the proper course of action would be to support both the old and the new behavior ?
if so, looks like we need to do some working on tracing-attributes :p

dtolnay · 2021-02-03T23:52:59Z

I tried the latest commit -- it's regressing some code like this:

use async_trait::async_trait;

#[async_trait]
pub trait Trait: Sized {
    async fn f(self) {
        struct Struct;
        impl Struct {
            fn f(self) {
                let _ = self;
            }
        }
    }
}

error[E0434]: can't capture dynamic environment in a fn item
 --> src/main.rs:9:25
  |
9 |                 let _ = self;
  |                         ^^^^
  |
  = help: use the `|| { ... }` closure form instead

dtolnay · 2021-02-03T23:53:27Z

Also this:

#![deny(warnings)]

use async_trait::async_trait;

#[async_trait]
pub trait Trait {
    async fn f() {
        unimplemented!()
    }
}

error: unreachable expression
 --> src/main.rs:7:18
  |
7 |       async fn f() {
  |  __________________^
8 | |         unimplemented!()
  | |         ---------------- any code following this expression is unreachable
9 | |     }
  | |_____^ unreachable expression

SergioBenitez · 2021-02-04T00:05:14Z

Alright, that last commit should handle all of the regressions thus far. As a side effect, some error messages have improved. Feel free to hit me with more regressions.

I've also marked the tracing test should_panic for now.

SergioBenitez · 2021-02-04T01:55:55Z

Oh, perhaps I missed the point of the unreachable example you showed. I see the deny warnings now. Makes sense. Will take a look a bit later.

Edit: handled.

dtolnay

This looks good to me.

I'll give a little time before merging to give the tracing folks a chance to get an update out.

dtolnay · 2021-02-04T03:57:24Z

tests/ui/self-span.stderr

 error[E0308]: mismatched types
-  --> $DIR/self-span.rs:25:21
+  --> $DIR/self-span.rs:24:21
   |
+24 |     async fn method(self) {
+   |                     ^^^^ expected `()`, found enum `E`
 25 |         let _: () = self;
-   |                --   ^^^^ expected `()`, found enum `E`
-   |                |
-   |                expected due to this
+   |                -- expected due to this


Do you have any insight into what's going on with this error message? The error before this PR is what I would expect and matches what you get in the non-trait case: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=f8833ba4ae1fad764bbaa11f1f6f2701. The error after seems misplaced to me.

Ah, interesting. I had read this message differently and believed it to be an improved version of the previous one, but I see now what the intention of the message is and how this is now a bit more confusing.

The span for that second self, which is renamed to __self, is set to the span of the receiver to fix #105. Besides being able to change only the resolution context, I'm not sure what heuristic to apply here to change only affected selfs.

I'm really not sure what to do here. The previous workaround seemed to "work" because we generated a new function. Now that we don't do so, we need to replace the span of all selfs, not just of the one argument to the function as before. This will result in all error messages that correspond to self to point to the receiver, which is an unfortunate compromise.

My suggestion is that we gate this workaround to rustc versions <= 1.45, or whenever the fix landed in the compiler.

Sounds good, let's do that. Thanks!

Have a preference for library to use? My go-to is versioncheck.

SergioBenitez · 2021-02-04T08:44:23Z

This looks good to me.

I'll give a little time before merging to give the tracing folks a chance to get an update out.

Awesome. I'll remove all of the unused code then.

nightmared · 2021-02-07T11:08:59Z

On some minor note, it is not so easy to handle arguments that aren't used inside the function: tokio-rs/tracing#1228 (comment).
From what I could gather, async-trait redefines the variable v with a let binding, but in the process of doing so, it introduces an unused variable.
If I'm not mistaken, the logic is the following:
a) async_trait rewrites the body of async fn call(&mut self, v: usize) {} in this fashion:

Box::pin(async move {
  let __ret: () = {
      let mut __self = self;
      let v = v;
  };
  #[allow(unreachable_code)]
  __ret
})

So far, it have the same behavior as an equivalent sync function would have, so the warning is expected, and thus not an issue.
b) instrument then rewrites the body as:

Box::pin(async move {
  XXX // something that uses the `v` variable
  tracing::Instrument::instrument(
      async move {
          {
              let __ret: () = {
                  let mut __self = self;
                  let v = v;
              };
              #[allow(unreachable_code)]
          __ret
        }
      },
      __tracing_attr_span,
    )
    .await
})

In that case, the v variable is now used by code originated from tracing, but there is still the let v = v; binding, which is still an unused variable. Because they have the same span, the compiler reports the error as originating from the function signature:

warning: unused variable: `v`
   --> tracing-attributes/tests/async_fn.rs:176:34
    |
176 |         async fn call(&mut self, v: usize) {}
    |                                  ^ help: if this is intentional, prefix it with an underscore: `_v`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: 1 warning emitted

I'm not really sure about the way to go here. Optimally, we would inject instrument code after async-trait, but I don't think we can force ordering (or this whole story of async-trait/tracing integration would never have been a real deal) of procedural macros.
Any idea?

dtolnay · 2021-02-09T08:25:48Z

@nightmared, would it help if we insert some unique distinguishing marker like let __async_trait: (); in between the let statements and the user's original function body?

Box::pin(async move {
    let __ret: #RETURN = {
        let v = v;
        let __async_trait: ();
        #BODY
    };
    #[allow(unreachable_code)]
    __ret
})

You could crawl the macro input and replace that marker with:

Box::pin(async move {
    let __ret: #RETURN = {
        let v = v;
        if false {
            let _ = &(v + 5);  // expr from #[instrument(fields(test=%v+5))]
        }
        #BODY
    };
    #[allow(unreachable_code)]
    __ret
})

nightmared · 2021-02-09T08:40:32Z

Hmm yes, i guess this would work! However that also means we would have to follow variable reassignments like self being renamed to __self. Given I'm not really the one that will bear the burden of maintaining that compatibility in the long run (not being a maintainer, nor even an active contributor :p), I guess the best is to defer that decision to e.g. @hawkw and/or @davidbarsky.
That being said, it would probably be the cleanest way to properly place the code generated by #[instrument], and solve this issue altogether. It's probably fairly easy too: we would just have to wrap all the statements from the position of __async_trait to the end of the block.

nightmared · 2021-02-11T21:08:56Z

I have started to take a look at @dtolnay's solution, but it isn't quite as trivial as I first thought because we need to keep track of variable reassignments (e.g. let __self = self). I will keep you posted on my progress.

SergioBenitez · 2021-02-12T04:37:13Z

To propose a different solution: what if nothing is done about the warning? The warning feels correct to me: the variable is declared and never used in the function body by the written code. Purposefully silencing this warning could result in incorrect code that otherwise would have been caught. It's also easy enough to add an _ should they really intend not to use the variable. Of course, this would imply emitting code in the non-async_trait case that retains the warning, but that perhaps entails an easier and less brittle solution, perhaps one that mirrors the redefinitions in this PR.

nightmared · 2021-02-12T09:36:08Z

That is a possibility yes. However with some very minimal changes in async-trait (nightmared@32b1573) and a bit more work on tracing side, I was able to get this minor nitpick working ;)
We'll see what the maintainers of tracing think of this hack.

SergioBenitez · 2021-02-13T01:21:03Z

@dtolnay I'm continuing to work to improve async-trait in my local branch. When you're ready to merge this, let me know if you'd prefer squashing into fewer commits and/or multiple PRs.

Kiiyya · 2021-02-23T22:54:28Z

I am getting rather odd build errors with a minimal project with commit 0cda89b somehow:

#[async_trait::async_trait]
pub trait Derp {
    async fn hi();
}

fails to build with internal errors, both latest stable and nightly. Note that I literally just have those 4 lines + a dummy main(): https://gist.github.com/Kiiyya/250dc864682cc1fc256f6ac6135db45e And I have no clue what is going on, but I thought I'd just drop this here in hopes it helps, somehow?

For some reason, using that exact same commit in a way bigger project of mine compiles, I am properly confused about that elusive build error.

SergioBenitez · 2021-02-23T23:42:29Z

@Kiiyya Your larger project is probably enabling features of syn that result in Debug being implemented on those types while async-trait doesn't enable such features. This is was an omission on my part ... to omit the Debug derive. I've pushed a commit that should resolve the issue! Thanks for testing.

Kiiyya · 2021-02-23T23:45:41Z

Yep works now, that was quick, thank you!

nightmared · 2021-02-24T21:42:49Z

src/expand.rs

+        Box::pin(async move {
+            let __ret: #ret_ty = {
+                #(#decls)*
+                #(#stmts)*


In order for tokio-rs/tracing#1228 to work well, could you consider adding let __async_trait: (); between the #(#decls)* and #(#stmts)* lines? That should do the job. Thanks!

I think this is a poor solution to the problem at hand, and that emitting warnings is, in fact, the desired effect. Nevertheless, I've pushed a commit that implements this.

I do agree that it's a hack, and a nasty one at that. On the other hand, I do not believe that users will understand that exhibit A will trigger a warning and not exhibit B. That doesn't appear very coherent (at least with what I think is the behavior of instrument in non-async-trait functions) either.

Exhibit A:

#[instrument(fields(test=%v+5))] async fn call(&mut self, v: usize) {}

Exhibit B:

#[instrument(fields(test=%_v+5))] async fn call(&mut self, _v: usize) {}

I disagree that users won't understand. Without the attribute, A triggers a warning and B does not because v is unused in the body of A. Even when the attribute is applied, v continues to be unused in A, and so it is my opinion that A should continue to result in a warning.

I don't mean to imply that instrument should inconsistently emit warnings based on whether async-trait is used, but instead, that it should consistently emit warnings as would be emitted without the attribute.

dtolnay

Thanks, this is great.

dtolnay · 2021-03-05T05:09:49Z

I ended up landing this without the workaround for tracing-attributes because #143 (comment) / #143 (comment) are compelling to me.

dtolnay reviewed Feb 3, 2021

View reviewed changes

SergioBenitez commented Feb 3, 2021

View reviewed changes

nightmared mentioned this pull request Feb 3, 2021

#[instrument] - Future-proofing async-trait support in tracing-attributes tokio-rs/tracing#1219

Closed

SergioBenitez force-pushed the master branch from e94ffb1 to f2c8b9a Compare February 4, 2021 02:13

dtolnay reviewed Feb 4, 2021

View reviewed changes

SergioBenitez added 5 commits February 12, 2021 17:17

Use 'async block' in block transform.

fac72b8

Properly handle pattern-based arguments.

417390f

Replace 'self' in macro calls.

f3406dc

Aid inference by explicitly emitting return type.

b49dc26

Replace 'self' span everywhere with receiver span.

3f4db8d

SergioBenitez added 8 commits February 12, 2021 17:17

Don't recurse into items in ReplaceSelf.

f81d0f9

Silence clippy in 'test_internal_items'.

c9463d5

Silence generated unreachable code.

acaf67a

Remove unused code.

003a262

Gate self span hack on rustc <= 1.46.

42a0be1

Add UI test for lifetime related spans.

c509624

Properly set spans on introduced lifetimes.

7b437f6

Ignore 'clippy::type_repetition_in_bounds'.

0cda89b

SergioBenitez force-pushed the master branch from b9c3a59 to 0cda89b Compare February 13, 2021 01:18

Remove unused 'Debug' derive for 'Context'.

45f9d24

nightmared reviewed Feb 24, 2021

View reviewed changes

Terminate decls with 'let __async_trait: ();'.

628ed73

nightmared mentioned this pull request Feb 27, 2021

[tracing-attributes] future-proof the #[instrument] integration with async-trait tokio-rs/tracing#1228

Merged

dtolnay approved these changes Mar 5, 2021

View reviewed changes

dtolnay merged commit 0fd4e33 into dtolnay:master Mar 5, 2021

This was referenced Mar 7, 2021

Unsize coercion fails if end of method is unreachable #149

Closed

Support for impl Trait in associated type (type_alias_impl_trait) #152

Closed

This was referenced Mar 11, 2021

Bump async-trait from 0.1.42 to 0.1.48 habitat-sh/habitat#8125

Merged

build(deps): bump async-trait from 0.1.42 to 0.1.48 MaterializeInc/materialize#6084

Merged

build(deps): bump async-trait from 0.1.42 to 0.1.48 apllodb/apllodb#163

Merged

taiki-e mentioned this pull request Apr 28, 2021

Generic params from impl dont have 'async_trait lifetime #8

Closed

dtolnay mentioned this pull request Mar 4, 2023

Delete replacement of elided lifetimes in impl heading #239

Merged

dtolnay mentioned this pull request May 22, 2023

Want way to detect recursive impls #247

Closed

dtolnay mentioned this pull request Dec 30, 2023

Update explanation to show async block expansion #255

Merged

taiki-e mentioned this pull request Oct 3, 2024

Improve diagnostics for tokio::main and tokio::test attributes tokio-rs/tokio#6882

Closed

Transform blocks into async move for more compatibility #143

Transform blocks into async move for more compatibility #143

Conversation

SergioBenitez commented Feb 3, 2021 • edited Loading

SergioBenitez commented Feb 3, 2021

dtolnay left a comment

Choose a reason for hiding this comment

SergioBenitez commented Feb 3, 2021

Choose a reason for hiding this comment

dtolnay commented Feb 3, 2021 • edited Loading

SergioBenitez commented Feb 3, 2021

SergioBenitez commented Feb 3, 2021

dtolnay commented Feb 3, 2021 • edited Loading

dtolnay commented Feb 3, 2021

dtolnay commented Feb 3, 2021

nightmared commented Feb 3, 2021

dtolnay commented Feb 3, 2021

dtolnay commented Feb 3, 2021

SergioBenitez commented Feb 4, 2021

SergioBenitez commented Feb 4, 2021 • edited Loading

dtolnay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SergioBenitez commented Feb 4, 2021

nightmared commented Feb 7, 2021

dtolnay commented Feb 9, 2021

nightmared commented Feb 9, 2021

nightmared commented Feb 11, 2021

SergioBenitez commented Feb 12, 2021 • edited Loading

nightmared commented Feb 12, 2021

SergioBenitez commented Feb 13, 2021

Kiiyya commented Feb 23, 2021

SergioBenitez commented Feb 23, 2021

Kiiyya commented Feb 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SergioBenitez Feb 25, 2021 • edited Loading

Choose a reason for hiding this comment

dtolnay left a comment

Choose a reason for hiding this comment

dtolnay commented Mar 5, 2021

Transform blocks into `async move` for more compatibility #143

Transform blocks into `async move` for more compatibility #143

SergioBenitez commented Feb 3, 2021 •

edited

Loading

dtolnay commented Feb 3, 2021 •

edited

Loading

dtolnay commented Feb 3, 2021 •

edited

Loading

SergioBenitez commented Feb 4, 2021 •

edited

Loading

SergioBenitez commented Feb 12, 2021 •

edited

Loading

SergioBenitez Feb 25, 2021 •

edited

Loading