Rename cluster shape columns to use 'worker' prefix in the output files and rename metadata file

[parthosa]

Issue: #1239

Changes

• Rename all cluster shape related columns/fields to use worker prefix instead of executor.
  • Usage of executorInstance, driverInstance has been replaced by workerNodeType and driverNodeType
  • Going forward we should ensure executor term is reserved only for spark executors to avoid ambiguity.
• Renames metadata file to app_metadata.json
  • Add platform field in the metadata file

Output

File: qual_2024xxx/app_metadata.json

Contents

[
  {
    "appId": "application_1692643187882_0001",
    "appName": "NDS - Power Run",
    "eventLog": "file:/path/application_1692643187882_0001",
    "clusterInfo": {
      "platform": "emr",
      "sourceCluster": {
        "driverNodeType": "i3.2xlarge",
        "workerNodeType": "m5d.8xlarge",
        "numWorkerNodes": 8
      },
      "recommendedCluster": {
        "driverNodeType": "i3.2xlarge",
        "workerNodeType": "g5.4xlarge",
        "numWorkerNodes": 16
      }
    },
    "estimatedGpuSpeedupCategory": "Small",
    "fullClusterConfigRecommendations": "/tool_output/qual_20240805173341_9Eb76BAa/rapids_4_spark_qualification_output/tuning/application_1692643187882_0001.conf",
    "gpuConfigRecommendationBreakdown": "/tool_output/qual_20240805173341_9Eb76BAa/rapids_4_spark_qualification_output/tuning/application_1692643187882_0001.log"
  }
]

File: qual_2024xxx/rapids_4_spark_qualification_output/rapids_4_spark_qualification_output_cluster_information.json

Contents

[ {
  "appName" : "NDS - Power Run",
  "appId" : "application_1692643187882_0001",
  "eventLogPath" : "file:/path/application_1692643187882_0001",
  "clusterInfo" : {
    "vendor" : "emr",
    "coresPerExecutor" : 16,
    "numExecsPerNode" : 2,
    "numWorkerNodes" : 8,
    "executorHeapMemory" : 16384,
    "driverHost" : "ip-xxx.us-west-2.compute.internal",
    "clusterId" : "j-2xxxx"
  },
  "recommendedClusterInfo" : {
    "vendor" : "emr",
    "coresPerExecutor" : 16,
    "numWorkerNodes" : 16,
    "numGpus" : 1,
    "numExecutors" : 16,
    "workerNodeType" : "g5.4xlarge"
  }
} ]

Follow Up PR

• Add cluster recommendation for onprem
• Ensure correct gpuPerWorker is recommended for eligible platforms (dataproc, onprem)

[cindyyuanjiang]

Thanks @parthosa!