[BUG] Catch Profiler error when app info is empty

[cindyyuanjiang]

Fixes #947

The Profiler could fail when: it successfully creates an ApplicationInfo in createApp, when processing this app in processApps, the Profiler found that it is unable to collect information from this app in getAppInfo. This leads to an empty appInfo in

spark-rapids-tools/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala

Line 338 in f163fcc

val appInfo = collect.getAppInfo

which fails

spark-rapids-tools/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala

Line 410 in f163fcc

logInfo(s"Took ${endTime - startTime}ms to Process [${appInfo.head.appId}]")

This PR catches the error early and prints a more helpful message.

[amahussein]

Thanks @cindyyuanjiang !
QQ: since we are trying to reduce the gap betwen Q/P tools, How feasible do you think it is to handle PRofiling tool the same way we do in Q tool? For example, generate a file like rapids_4_spark_qualification_output_status.csv reporting the status of processing of each eventlog ?
If you find it an easy change, then it will be nice to handle it in this PR.

@parthosa do you recall if we have an exiting issue to generate status report for Profiling tool?

[parthosa]

@parthosa do you recall if we have an exiting issue to generate status report for Profiling tool?

We do not have any existing issue to generate status report for Profiling Tool.

For generating status report in Profiling tool, #464 could be used as reference.

Revisiting changes

[cindyyuanjiang]

QQ: since we are trying to reduce the gap betwen Q/P tools, How feasible do you think it is to handle PRofiling tool the same way we do in Q tool? For example, generate a file like rapids_4_spark_qualification_output_status.csv reporting the status of processing of each eventlog ? If you find it an easy change, then it will be nice to handle it in this PR.

Thanks @amahussein!
I think it is feasible to generate a status report for Profiling tool. I am taking a look into that.

[cindyyuanjiang]

Filed issue to track adding status report for profiling tool: #998, will add feature in a new PR.

[amahussein]

QQ: @cindyyuanjiang Did you test with multiple eventlog that throwing RuntimeExcpetion won't cause the entire Profiler app to Exit?
We want to make sure that runtimeExcprion won't crash the execution and cause the processing of other apps to be terminated.

[cindyyuanjiang]

QQ: @cindyyuanjiang Did you test with multiple eventlog that throwing RuntimeExcpetion won't cause the entire Profiler app to Exit? We want to make sure that runtimeExcprion won't crash the execution and cause the processing of other apps to be terminated.

Thanks @amahussein! I tested with multiple event logs and confirm that change in this PR would not crash the Profiler. The Profiler would continue to run until all apps are processed.

The eventlog which throws RuntimeException is marked as failed as in the progress bar:
Profile Tool Progress 100% [==========================================] (1 succeeded + 1 failed + 0 skipped + 0 N/A) / 2

[amahussein]

Thanks @cindyyuanjiang