Change aggregation of executor CPU and run time for Qualification tool to speed up query #2620
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@apache.org>
tgravescs
added
the
task
gerashegalov
reviewed
Jun 7, 2021
tools/src/main/scala/org/apache/spark/sql/rapids/tool/profiling/EventsProcessor.scala
Show resolved
Hide resolved
tools/src/main/scala/org/apache/spark/sql/rapids/tool/profiling/EventsProcessor.scala
Show resolved
Hide resolved
|
build |
nartal1
approved these changes
Jun 7, 2021
gerashegalov
approved these changes
Jun 7, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
…l to speed up query (NVIDIA#2620) * Aggregate task metrics for qualification Signed-off-by: Thomas Graves <tgraves@apache.org> * change to put times in the existing stages df * update default and fix query * rename columns * update tests for new calculation exec cpu time * update comment * cleanup * Update docs and default param * update readme and comment * update spark-shell for qualification
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
…l to speed up query (NVIDIA#2620) * Aggregate task metrics for qualification Signed-off-by: Thomas Graves <tgraves@apache.org> * change to put times in the existing stages df * update default and fix query * rename columns * update tests for new calculation exec cpu time * update comment * cleanup * Update docs and default param * update readme and comment * update spark-shell for qualification
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Running the qualification tool over all nds results and having it include the executor cpu time percent was taking about 45 minutes and required 30G driver memory. To address this I changed the way we aggregate those metrics just for the qualification side so that we don't need to keep all tasks and create the individual tasks table. Now it just aggregates them up per stage on the fly and we can simply look it up per stage and aggregate them based on that. That improved the run time to be about 15 minutes and I used 5G memory (I might be able to use less, this is what I set it at).
since the performance is better I turned it on by default and changed config to allow removing it if you don't want it for some reason.