Rewrite Qualification tool for better performance #2822
Conversation
|
build |
|
build |
There was a problem hiding this comment.
I am not an expert on the Qualifying tool but the changes look good to me. The only thing that might be interesting to explore is putting in a thread pool, like the Spark history server uses. That way if there are more than one application to look at for the qualification tool, it can all be done in parallel (up to a set thread pool size).
|
test failure because not upmerged |
|
thanks Bobby, definitely a good idea. I'll do that in a followup |
|
build |
|
build |
|
@revans2 are you ok with this being merged (if so can you approve)? |
|
build |
tools/pom.xml
Outdated
| @@ -41,6 +41,7 @@ | |||
| <dependency> | |||
| <groupId>org.scala-lang</groupId> | |||
| <artifactId>scala-library</artifactId> | |||
| <scope>compile,runtime</scope> | |||
There was a problem hiding this comment.
I think the compile scope is a superset of the runtime scope. As written, this is equivalent to just compile
| event match { | ||
| case _: SparkListenerResourceProfileAdded => | ||
| doSparkListenerResourceProfileAdded(app, | ||
| event.asInstanceOf[SparkListenerResourceProfileAdded]) |
There was a problem hiding this comment.
nit: can avoid asInstanceOf by using a variable in the pattern instead of _
|
build |
This PR rewrites parts of the qualification tool. I kept the overall structure very similar so that code could be reused with the Profiling tool.
overall this is what this does:
This takes significantly less time and memory now to process event logs. The time it takes is really close to just the time it takes Spark to read and parse the events.