Macros 2.0: #[cfg_attr] makes .to_string() and TokenStream disagree

[alexcrichton]

I ran across a very curious case today... Given this procedural macro:

#![crate_type = "proc-macro"]
#![feature(proc_macro)]

extern crate proc_macro;

use proc_macro::TokenStream;

#[proc_macro_derive(Foo, attributes(foo))]
pub fn foo(input: TokenStream) -> TokenStream {
    println!("{}", input.to_string());
    print_tts(input, 0);

    TokenStream::empty()
}

fn print_tts(input: TokenStream, tab: usize) {
    let mut t = String::new();
    for _ in 0..tab {
        t.push_str("  ");
    }
    for token in input {
        match token.kind {
            proc_macro::TokenNode::Group(d, others) => {
                println!("{}{:?}", t, d);
                print_tts(others.into(), tab + 1);
            }
            s => println!("{}{:?}", t, s),
        }
    }
}

and this expansion:

#![crate_type = "rlib"]
#![feature(proc_macro)]

#[macro_use]
extern crate foo;

#[derive(Foo)]
pub struct MyStructc {
    #[cfg_attr(my_cfg, foo)]
    _a: i32,
}

when compiled I get:

$ rustc +nightly foo.rs && rustc +nightly bar.rs -L .
pub struct MyStructc {
    _a: i32,
}
Term(Term(pub))
Term(Term(struct))
Term(Term(MyStructc))
Brace
  Op('#', Alone)
  Bracket
    Term(Term(cfg_attr))
    Parenthesis
      Term(Term(my_cfg))
      Op(',', Alone)
      Term(Term(foo))
  Term(Term(_a))
  Op(':', Alone)
  Term(Term(i32))
Op(',', Alone)

which is quite curious!

The to_string() representation has the field present (even though the --cfg isn't supplied) and the token stream has the actual #[cfg_attr] there.

I was actually expecting neither outcome to happen (how naive of me!) in terms of #[cfg] processing typically happening before custom derives, but maybe that's not possible? -- er, just wrong thinking here

cc @dtolnay
cc @jseyfried
cc @nrc

[dtolnay]

Seems similar to how mod also causes an inconsistency. #47627

[alexcrichton]

Ok I think I see what's going on here.

This has to do with this comment introduced in #43230. Basically I think this boils down to #43081, we fundamentally cannot losslessly tokenize an AST item into a list of tokens to hand to a procedural macro.

What's happening here is that all the tokens of the struct MyStructc are saved off in the parser. This is the original source token stream which produced the struct. Later on when we tokenize the struct to hand off to a procedural macro we will use this cached token stream, if available, and tokenize it out and return it. When stringifying, however, we use the actual AST node itself (the source of truth).

The bug is that cfg processing does indeed happen before the procedural macro runs. The cfg processing is modifying the AST item's internal fields but it's not invalidating the cached token stream or modifying it. Hence when the procedural macro runs the cached token stream is invalid wrt the internal state of the AST node, causing two different results to come out depending on how you look at it.

Possible solutions here are:

1. Invalidate the cached list of tokens if the AST node is modified. This seems tricky to get right, are there other locations where a modification can happen?
2. Actually implemented Compiler loses location information before calling macros (sometimes) #43081, but holy cow that's a lot of work.
3. Compare the two results of tokenization (stringify + parse and the tokens themselves) and return the string one if they differ.

None of them seem like great solutions :(