Commits on Jul 22, 2018

1. rustc: Implement tokenization of nested items …

   Ever plagued by rust-lang#43081 the compiler can return surprising spans in situations
   related to procedural macros. This is exhibited by rust-lang#47983 where whenever a
   procedural macro is invoked in a nested item context it would fail to have
   correct span information.
   
   While rust-lang#43230 provided a "hack" to cache the token stream used for each item in
   the compiler it's not a full-blown solution. This commit continues to extend
   this "hack" a bit more to work for nested items.
   
   Previously in the parser the `parse_item` method would collect the tokens for an
   item into a cache on the item itself. It turned out, however, that nested items
   were parsed through the `parse_item_` method, so they didn't receive similar
   treatment. To remedy this situation the hook for collecting tokens was moved
   into `parse_item_` instead of `parse_item`.
   
   Afterwards the token collection scheme was updated to support nested collection
   of tokens. This is implemented by tracking `TokenStream` tokens instead of
   `TokenTree` to allow for collecting items into streams at intermediate layers
   and having them interleaved in the upper layers.
   
   All in all, this...
   
   Closes rust-lang#47983

   

   alexcrichton committed Jul 22, 2018
   Configuration menu

   • View commit details
   • Copy full SHA for d760aaf
   • Browse repository at this point

   Copy the full SHA

   d760aaf View commit details

   Browse the repository at this point in the history