Store Cluster Shape Recommendation in User Tools Qualification Output

[parthosa]

Fixes #1004. This PR adds functionality to store the cluster shape recommendation in a file cluster_shape_recommendation.json within the user tools intermediate output directory qual_202405xxxx_yyyy/intermediate_output

CMD:

spark_rapids qualification --platform <platform> --eventlogs <eventlogs> --tools_jar <local tools dev jar path>

Note:

• The --tools_jar <local tools dev jar path> parameter is required until the latest changes are published on Maven.

Output:

Output File Location: qual_202405xxxx_yyyy/intermediate_output/cluster_shape_recommendation.json

emr/databricks

[
  {
    "clusterName": "1234-5678-test",
    "sourceCluster": {
      "driverInstance": "m6gd.xlarge",
      "executorInstance": "m6gd.2xlarge",
      "numExecutors": 2
    },
    "targetCluster": {
      "driverInstance": "m6gd.xlarge",
      "executorInstance": "g5.2xlarge",
      "numExecutors": 2
    }
  }
]

dataproc

[
  {
    "clusterName": "default-cluster-name",
    "sourceCluster": {
      "driverInstance": "n1-standard-16",
      "executorInstance": "n1-standard-16",
      "numExecutors": 4
    },
    "targetCluster": {
      "driverInstance": "n1-standard-16",
      "executorInstance": "n1-standard-16",
      "numExecutors": 4,
      "gpuInfo": {
        "device": "nvidia-tesla-t4",
        "gpuPerWorker": 2
      },
      "additionalConfig": {
        "localSsd": 2
      }
    }
  }
]

Console Output:

    - Intermediate output generated by tools: /<path>/qual_20240522004525_xxxx/intermediate_output

Changes

1. New Method: Added get_cluster_configuration(), which returns a dictionary containing the cluster shape and other configurations. This method can be overridden by platform-specific classes to provide additional properties.
2. Introduced __generate_cluster_recommendation_report() to generate cluster shape recommendations from CPU to GPU clusters using the above New Method.
3. While generation of the user tools report, the cluster shape recommendation is written to a JSON file.
4. Additionally, cluster creation scripts can now use the above New Method to fill the rendering arguments.
   • This is done to avoid redundant code.

Testing

1. Tested on different platforms - emr, dataproc, databricks and onprem.
2. This file will not be generated if source cluster information is not available.

[amahussein]

• Tested on different platforms - emr, dataproc, databricks and onprem.

• This file will not be generated if cluster information is not available.

Interesting, I thought that we do not have cluster recommendation output section for DB, and onPrem.

[amahussein]

IIRC, we did not generate cluster recommendation in the output for Databricks/onPrem
Does this PR generates the cluster-recommendations for those platforms?

[parthosa]

Waiting for #1002 as it creates intermediate_output directory. cluster_information.json will be stored in the intermediate directory.

[tgravescs]

can you please update description to have details about when it generates this? Ie qual tool, profiling tool, python vs java, etc.

[parthosa]

can you please update description to have details about when it generates this? Ie qual tool, profiling tool, python vs java, etc.

@tgravescs Updated the PR title and description to include more details about the changes.

[amahussein]

Thanks @parthosa
LGTME