improve GetPrevious/GetNextToken performance #3244
Labels
Area-Compilers
Feature Request
Tenet-Performance
Regression in measured performance of the product from goals.
Milestone
Comments
|
GetNextToken definition in source |
heejaechang
changed the title
heejaechang
added a commit
to heejaechang/roslyn
that referenced
this issue
Jun 2, 2015
this performance improvement is particularly for devdiv bug # 1089540 this makes the file in the bug to be formatted in several seconds compared to several minutes on my machine. there were several issues. each one fixed by #1, use concurrency on gathering operations. #2, don't use too much time to split work to chunks if that requires more work than actually formatting. #3, don't blindly set beginning of a file as inseparable start point for certain formatting options. ... but these don't actually address the most impactful root cause of this perf issues. which is perf issue of GetPrevious/GetNextToken API in compiler. (dotnet#3244) formatter internally uses GetDescendantTokens to get all tokens at once and cache them which takes less than 1 second for the entire file (2M bytes size) in the bug. and use the cache internally. but certain part of formatter (Rule Provider) can't use that internal cache, so it has to use the GetPrevious/GetNextToken to move around tokens, which in this particular bug, takes more than 40 seconds on my machine. and that is not even for entire file. (less than 1/12 of tokens) I opened a bug to compiler team, hopely so that we can get better perf on those APIs. in this PR, I mitigated the issue either by making more things to run concurrently or by changing logic which requires those APIs.
|
for the same file in the bug, smart indenter on "}" in array initializer also spends most of its time on GetNextToken. (16 seconds in my machine) |
heejaechang
added a commit
that referenced
this issue
Jun 4, 2015
make formatting performance better ... this performance improvement is particularly for devdiv bug # 1089540 this makes the file in the bug to be formatted in several seconds compared to several minutes on my machine. there were several issues. each one fixed by #1, use concurrency on gathering operations. #2, don't use too much time to split work to chunks if that requires more work than actually formatting. #3, don't blindly set beginning of a file as inseparable start point for certain formatting options. ... but these don't actually address the most impactful root cause of this perf issues. which is perf issue of GetPrevious/GetNextToken API in compiler. (#3244) formatter internally uses GetDescendantTokens to get all tokens at once and cache them which takes less than 1 second for the entire file (2M bytes size) in the bug. and use the cache internally. but certain part of formatter (Rule Provider) can't use that internal cache, so it has to use the GetPrevious/GetNextToken to move around tokens, which in this particular bug, takes more than 40 seconds on my machine. and that is not even for entire file. (less than 1/12 of tokens) I opened a bug to compiler team, hopely so that we can get better perf on those APIs. in this PR, I mitigated the issue either by making more things to run concurrently or by changing logic which requires those APIs.
jaredpar
added
the
Tenet-Performance
|
Dont' currently have any issue indicating we still have a problem here. So closing out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
look devdiv # 1089540 for more detail.
GetNextToken takes 40% of time (of several minutes)
The text was updated successfully, but these errors were encountered: