Fix calculation of unsupported operators stage duration percentage

[parthosa]

Fixes #1003. This PR fixes the unsupported stage duration calculation by using SQL Stage Durations Sum as denominator instead of App Duration.

Additionally, handle the case when event logs do not have App Name.

Changes:

1. Add column SQL Stage Durations Sum in rapids_4_spark_qualification_output.csv.
2. Group apps by app id and sql stage durations sum and calculate unsupported operators stage duration.
3. Merge this dataframe with all_apps and calculated percentage as unsupported stage duration / sql stage durations sum.
4. Unit Test Changes - Add new column in unit tests expected files.

Design

1. We added Total Stage Duration in the main rapids_4_spark_qualification_output.csv instead of rapids_4_spark_qualification_output_unsupportedOperators.csv since the second file might not have entries for all apps (eg apps without any unsupported operators). To handle this we would have to add more fillna(), division by NaN etc.
2. This is a cleaner approach that avoids the above handling.
3. The con was that we had to update the expected files for unit tests with new column.

Test

CMD (need to specify dev JAR):

spark_rapids qualification <eventlogs>  --tools_jar <tools_jar> 

• Tested on multiple event logs with various cases:
  • App with no unsupported operator entries
  • App with unsupported entries that have no impact (hence would be filtered before calculation)
  • Two events logs having same app name - To test TCO behaviour.

Output

• Showing the result along with Spill Heuristics:

[amahussein]

Thanks @parthosa
I was not expecting a change in the Scala side.
Lets sync offline to better understanding the recent changes.

[amahussein]

@parthosa Just double checking.
I don't see the column title updated in the last 4 days. Did you commit your latest changes?

[parthosa]

I don't see the column title updated in the last 4 days. Did you commit your latest changes?

@amahussein Renamed the column to SQL Stage Durations Sum

[amahussein]

I guess that it turned to be more painful than it is originally thought :)
Thanks @parthosa !
LGTME.