Profiling tool add CSV output option and add new combined mode #3342
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
build |
|
build |
|
Note, I ran over the same 11618 apps with 50GB memory and it took 31 minutes. |
|
looks like my branch isn't up to date, will upmerge |
|
build |
nartal1
reviewed
Sep 1, 2021
tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/CollectInformation.scala
Show resolved
Hide resolved
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
nartal1
approved these changes
Sep 1, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #3047
fixes #3364
Add an option to the profiling tool to output csv files in addition to the text output.
While doing this I also got a request to run the profiling tool on 10,000+ event logs at a time and output a single table. To support that I made some changes for memory management and added a new combined mode. The combined mode is the same as compare where all applications get put into a single table, except for it doesn't output the 2 tables that actually try to compare sqlids and job ids between the applications. The main scale issue here was with the memory needed to keep all the raw application info in memory. I changed it so that it processes each application and then just outputs a summary information which is much smaller. for the new combined mode it processes each application separately in the thread pool and then at the end combines all the summary information into single tables. I processed 11618 apps in 20 minutes using the combined mode. I did require quite a bit of memory still - around 120GB. I didn't try with less. The combined mode options outputs files into a /combined/ subdirectory.
To support outputting to csv I changed most of the functions to be getters and then we print the information separately. ProfileOutputWriter was extended to know how to output text and csv format.