-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Followups on handling Photon eventlogs #953
Conversation
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me> Fixes NVIDIA#948 - this change makes the Qualification detect photon eventlogs fron environmentUpdate event - Adds a boolean that will be used later for applications with photon runtime in Prof/Qual tools - Moved some of the utilities to the new class DatabricksParseHelper - Fixed a bug in handling incorrect App parsing - Fixed unit tests
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/DatabricksParseHelper.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/qualification/QualificationAppInfo.scala
Show resolved
Hide resolved
core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/AppFilterSuite.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/AppFilterSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @cindyyuanjiang
I. addressed your review comments.
core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/AppFilterSuite.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/AppFilterSuite.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/DatabricksParseHelper.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/qualification/QualificationAppInfo.scala
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @amahussein for this change. Minor typo nit. Removing the hardcoded strings was a good idea.
|
||
/** | ||
* Checks if the properties indicate that the application is a Photon app. | ||
* This ca be checked by looking for keywords in one of the keys defined in PHOTON_SPARK_PROPS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: typo can
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @amahussein! LGTM.
Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me
Fixes #948