Speed up TokenStream concatenation
#65198
Conversation
Currently, this function creates a new empty stream, and then appends the elements from each given stream onto that stream. This can cause quadratic behaviour. This commit changes the function so that it modifies the first stream (which can be long) by extending it with the elements from the subsequent streams (which are almost always short), which avoids the quadratic behaviour.
Currently, when two tokens must be glued together, this function duplicates
large chunks of the existing streams. This can cause quadratic behaviour.
This commit changes the function so that it overwrites the last token with a
glued token, which avoids the quadratic behaviour. This removes the need for
`TokenStreamBuilder::push_all_but_{first,last}_tree`.
The commit also restructures `push` somewhat, by removing
`TokenStream::{first_tree_and_joint,last_tree_if_joint}` in favour of more
pattern matching and some comments. This makes the code shorter, and in my
opinion, more readable.
rust-highfive
added
the
S-waiting-on-review
|
In my local measurements, this provided a very small (<1%) improvement on some of the standard benchmarks. @bors try @rust-timer queue |
|
Awaiting bors try build completion |
Speed up `TokenStream` concatenation This PR fixes the quadratic behaviour identified in #65080. r? @Mark-Simulacrum
| // Get the first stream. If it's `None`, create an empty | ||
| // stream. | ||
| let mut iter = streams.drain(); | ||
| let mut first_stream_lrc = match iter.next().unwrap().0 { |
There was a problem hiding this comment.
| let mut first_stream_lrc = match iter.next().unwrap().0 { | |
| let mut first_stream_lrc = match iter.next().unwrap().0.unwrap_or_default(); |
| // space for them. | ||
| let first_vec_mut = Lrc::make_mut(&mut first_stream_lrc); | ||
| first_vec_mut.reserve(num_appends); | ||
| for stream in iter { |
There was a problem hiding this comment.
I think this would be cleaner using .filter_map(|x| x) and .flat_map(|s| s.iter().cloned()).
There was a problem hiding this comment.
Current version is easy to read, and the alternatives seem to spend more Rust functions to do the same thing, IMO one can prefer the current.
|
☀️ Try build successful - checks-azure |
|
Queued c255427 with parent d304f5c, future comparison URL. |
There was a problem hiding this comment.
Code changes look good to me. I'd like to get @petrochenkov or perhaps @matklad to sign off too though since I'm not too familiar with this code.
|
LGTM |
|
@bors r+ rollup=never |
|
📌 Commit 75e0078 has been approved by |
bors
added
S-waiting-on-bors
|
The perf run didn't work. Let's try it the old-fashioned way: @rust-timer build c255427 |
|
Queued c255427 with parent d304f5c, future comparison URL. |
|
@nnethercote it worked but still haven't finished, you can see it here https://perf.rust-lang.org/status.html That 700000% wall time regression made queue a bit long... |
|
Finished benchmarking try commit c255427, comparison URL. |
|
Lots of small (<1%) improvements, mostly in the short-running benchmarks. |
Speed up `TokenStream` concatenation This PR fixes the quadratic behaviour identified in #65080. r? @Mark-Simulacrum
|
☀️ Test successful - checks-azure |
jyn514
added
I-compiletime
This PR fixes the quadratic behaviour identified in #65080.
r? @Mark-Simulacrum