Add cluster details in qualification summary output

[parthosa]

Fixes #918. This PR adds columns Vendor, Driver Host, Cluster ID, Cluster Name in the user tools qualification summary. This will assist the customers distinguish different jobs especially when in Databricks where all applications have the same name.

User Tools Output:

File: qualification_summary.csv

,Vendor,Driver Host,Cluster ID,Cluster Name,App Name,App ID,Speedup Based Recommendation,Estimated GPU Speedup,Estimated GPU Duration,App Duration,Unsupported Operators Stage Duration,Unsupported Operators Stage Duration Percent,Speed Up Estimation Model
0,databricks-azure,123.11.11.11,0220-012345-aaabbcc,job-name-test1,Databricks Shell,app-20240220075000-0000,Recommended,1.89,1206911.55,2282520.00,17298.00,0.76,SPEEDUPS
1,databricks-azure,123.11.11.12,0220-012345-xxxyyzz,job-name-test2,Databricks Shell,app-20240220074214-0000,Recommended,1.62,418620.75,676256.00,2576.00,0.38,SPEEDUPS
2,databricks-azure,123.11.11.13,0220-012345-pppqqrr,job-name-test3,Databricks Shell,app-20240220075434-0000,Recommended,1.56,1122375.53,1750824.00,11822.00,0.68,SPEEDUPS
3,databricks-azure,123.11.11.14,0220-012345-dddeeff,job-name-test4,Databricks Shell,app-20240220083138-0000,Recommended,1.44,209442.94,300577.00,455.00,0.15,SPEEDUPS
4,databricks-azure,123.11.11.15,0220-012345-ggghhii,job-name-test5,Databricks Shell,app-20240220065555-0000,Not Applicable,1.33,1718690.03,2282016.00,17709.00,0.78,SPEEDUPS

Note:

1. For TCO, we will still group apps but instead of App Name, we will group by [Vendor,Driver Host,Cluster ID,Cluster Name,App Name]
2. This change also ensures Top Candidates view shows meaningful results

+----+-------------------------+------------------+-------------------------+
|    | App ID                  | App Name         | Estimated GPU Speedup   |
|----+-------------------------+------------------+-------------------------|
|  0 | app-20240220075000-0000 | Databricks Shell | Small                   |
|  1 | app-20240220074214-0000 | Databricks Shell | Small                   |
|  2 | app-20240220075434-0000 | Databricks Shell | Small                   |
|  3 | app-20240220083138-0000 | Databricks Shell | Small                   |
|  4 | app-20240220065555-0000 | Databricks Shell | Small                   |
+----+-------------------------+------------------+-------------------------+

Report Summary:
------------------  -
Total applications  5
Top candidates      5
------------------  -

How To Test

1. Use latest dev jar while testing
   spark_rapids qualification --platform $PLATFORM --eventlogs $EVENTLOGS --tools_jar $SPARK_RAPIDS_TOOLS_DEV_JAR
2. Tested on Databricks, Dataproc and EMR

[parthosa]

Some Design Questions

1. Should we replace the intermediate cluster_information.json to cluster_information.csv?
   • Since we are convert the JSON to Dataframe in user tools now
   • This would simplify processing of this information in user tools
   • CSV would be more convenient for stakeholders usage.
2. How should empty columns be handled?
   • For example, in OnPrem and Dataproc only Spark Driver would be present, remaining 3 columns would be empty

[parthosa]

This is not required any more since we want to generate cluster information in every case.

[amahussein]

Thanks @parthosa
We discussed offline the design questions.
LGTME