[FEA] Remove CLI dependency for EMR and Databricks-AWS platforms in user tool #1196

cindyyuanjiang · 2024-07-16T22:48:46Z

Contributes to #1191

Changes

This PR remove AWS CLI dependency for EMR and Databricks-AWS platforms, except for case when --cluster input is cluster ID/name.

Add instance_descriptions in class CMDDriverBase which loads and stores the instance catalog json files for each platform
Add is_props_file in class ClusterBase to indicate if the cluster is loaded from a properties file, if is true, there should not be any CLI calls when running the tool

Testing

For each platform, we confirm that there is no AWS CLI call, and the output is the same before and after this PR.

Platform: EMR
spark_rapids qualification -v -p emr --eventlogs <my-event-logs> --cluster <my-cluster-props-file>

Stdout

2024-07-16 15:33:36,755 INFO rapids.tools.qualification: ******* [Process-Arguments]: Starting *******
2024-07-16 15:33:36,755 DEBUG rapids.tools.qualification: Processing Output Arguments
2024-07-16 15:33:36,755 DEBUG rapids.tools.qualification: Root directory of local storage is set as: xxxxxx/spark-rapids-tools
2024-07-16 15:33:36,755 INFO rapids.tools.qualification.ctxt: Local workdir root folder is set as xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563
2024-07-16 15:33:36,755 INFO rapids.tools.qualification.ctxt: Dependencies are generated locally in local disk as: xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir
2024-07-16 15:33:36,755 INFO rapids.tools.qualification.ctxt: Local output folder is set as: xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563
2024-07-16 15:33:36,755 INFO rapids.tools.qualification: Qualification tool processing the arguments
2024-07-16 15:33:36,861 INFO rapids.tools.qualification: RAPIDS accelerator tools jar is downloaded to work_dir xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/rapids-4-spark-tools_2.12-24.06.1.jar
2024-07-16 15:33:36,861 INFO rapids.tools.qualification: Using Spark RAPIDS Accelerator Tools jar version 24.06.1
2024-07-16 15:33:36,861 INFO rapids.tools.qualification: Checking dependency Apache Spark
2024-07-16 15:33:36,862 INFO rapids.tools.qualification: Checking dependency Hadoop AWS
2024-07-16 15:33:36,862 INFO rapids.tools.qualification: Checking dependency AWS Java SDK Bundled
2024-07-16 15:33:36,894 INFO rapids.tools.qualification: Completed downloading of dependency [Hadoop AWS] => 0.032 seconds
2024-07-16 15:33:37,579 INFO rapids.tools.qualification: Completed downloading of dependency [AWS Java SDK Bundled] => 0.716 seconds
2024-07-16 15:33:40,371 INFO rapids.tools.qualification: Completed downloading of dependency [Apache Spark] => 3.509 seconds
2024-07-16 15:33:40,372 INFO rapids.tools.qualification: Dependencies are processed as: xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/hadoop-aws-3.3.4.jar; xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/aws-java-sdk-bundle-1.12.262.jar; xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/spark-3.5.0-bin-hadoop3/jars/*
2024-07-16 15:33:40,372 INFO rapids.tools.qualification: Total Execution Time: Downloading dependencies for local Mode => 3.511 seconds
2024-07-16 15:33:40,372 DEBUG rapids.tools.qualification: Processing Rapids plugin Arguments {}
2024-07-16 15:33:40,373 INFO rapids.tools.qualification: Loading CPU cluster properties from file xxxxxx/emr_cpu_cluster_props.json
2024-07-16 15:33:40,374 INFO rapids.tools.qualification: Creating GPU cluster by converting the CPU cluster instances to GPU supported types
2024-07-16 15:33:40,374 DEBUG rapids.tools.cmd_driver: Skip converting Master nodes
2024-07-16 15:33:40,374 INFO rapids.tools.qualification: Generating input file for Auto-tuner
2024-07-16 15:33:40,374 DEBUG rapids.tools.qualification: Opening file xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/worker_info.yaml to write worker info
2024-07-16 15:33:40,376 INFO rapids.tools.qualification: Generated autotuner worker info: xxxxxx/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/worker_info.yaml
2024-07-16 15:33:40,376 INFO rapids.tools.qualification: WorkerInfo successfully processed into workDir [xxxxxx/Desktop/spark-rapids-tools/qual_20240716223336_7a717563/work_dir/worker_info.yaml]
2024-07-16 15:33:40,376 INFO rapids.tools.qualification: No remote output folder specified.
2024-07-16 15:33:40,376 INFO rapids.tools.qualification: ======= [Process-Arguments]: Finished =======

Platform: Databricks-AWS
spark_rapids qualification -v -p databricks-aws --eventlogs <my-event-logs> --cluster <my-cluster-props-file>

Stdout

2024-07-16 15:40:41,173 INFO rapids.tools.qualification: ******* [Process-Arguments]: Starting *******
2024-07-16 15:40:41,173 DEBUG rapids.tools.qualification: Processing Output Arguments
2024-07-16 15:40:41,173 DEBUG rapids.tools.qualification: Root directory of local storage is set as: xxxxxx/spark-rapids-tools
2024-07-16 15:40:41,173 INFO rapids.tools.qualification.ctxt: Local workdir root folder is set as xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0
2024-07-16 15:40:41,174 INFO rapids.tools.qualification.ctxt: Dependencies are generated locally in local disk as: xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir
2024-07-16 15:40:41,174 INFO rapids.tools.qualification.ctxt: Local output folder is set as: xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0
2024-07-16 15:40:41,174 INFO rapids.tools.qualification: Qualification tool processing the arguments
2024-07-16 15:40:41,248 INFO rapids.tools.qualification: RAPIDS accelerator tools jar is downloaded to work_dir xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/rapids-4-spark-tools_2.12-24.06.1.jar
2024-07-16 15:40:41,248 INFO rapids.tools.qualification: Using Spark RAPIDS Accelerator Tools jar version 24.06.1
2024-07-16 15:40:41,249 INFO rapids.tools.qualification: Checking dependency Apache Spark
2024-07-16 15:40:41,249 INFO rapids.tools.qualification: Checking dependency Hadoop AWS
2024-07-16 15:40:41,250 INFO rapids.tools.qualification: Checking dependency AWS Java SDK Bundled
2024-07-16 15:40:41,277 INFO rapids.tools.qualification: Completed downloading of dependency [Hadoop AWS] => 0.028 seconds
2024-07-16 15:40:41,957 INFO rapids.tools.qualification: Completed downloading of dependency [AWS Java SDK Bundled] => 0.707 seconds
2024-07-16 15:40:44,738 INFO rapids.tools.qualification: Completed downloading of dependency [Apache Spark] => 3.489 seconds
2024-07-16 15:40:44,739 INFO rapids.tools.qualification: Dependencies are processed as: xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/hadoop-aws-3.3.4.jar; xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/aws-java-sdk-bundle-1.12.262.jar; xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/spark-3.5.0-bin-hadoop3/jars/*
2024-07-16 15:40:44,740 INFO rapids.tools.qualification: Total Execution Time: Downloading dependencies for local Mode => 3.491 seconds
2024-07-16 15:40:44,740 DEBUG rapids.tools.qualification: Processing Rapids plugin Arguments {}
2024-07-16 15:40:44,740 INFO rapids.tools.qualification: Loading CPU cluster properties from file xxxxxx/db_aws_cpu_cluster_props.json
2024-07-16 15:40:44,741 WARNING rapids.tools.cluster: Cluster configuration: `executors` count 0 does not match the `num_workers` value 8. Using generated names.
2024-07-16 15:40:44,741 INFO rapids.tools.qualification: Creating GPU cluster by converting the CPU cluster instances to GPU supported types
2024-07-16 15:40:44,742 DEBUG rapids.tools.cmd_driver: Skip converting Master nodes
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.cluster: Node with g4dn.4xlarge supports GPU devices.
2024-07-16 15:40:44,742 INFO rapids.tools.qualification: Generating input file for Auto-tuner
2024-07-16 15:40:44,742 DEBUG rapids.tools.qualification: Opening file xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/worker_info.yaml to write worker info
2024-07-16 15:40:44,744 INFO rapids.tools.qualification: Generated autotuner worker info: xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/worker_info.yaml
2024-07-16 15:40:44,744 INFO rapids.tools.qualification: WorkerInfo successfully processed into workDir [xxxxxx/spark-rapids-tools/qual_20240716224041_2a5ecdc0/work_dir/worker_info.yaml]
2024-07-16 15:40:44,745 INFO rapids.tools.qualification: No remote output folder specified.
2024-07-16 15:40:44,745 INFO rapids.tools.qualification: ======= [Process-Arguments]: Finished =======

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

cindyyuanjiang · 2024-07-18T23:23:38Z

I put some comments (# To deprecate) for code that I think can be deprecated after removing CLI dependency for all platforms. I am planning to remove the unused code in a separate PR after we have finished every platform (in addition to some final refactoring).

amahussein · 2024-07-19T14:51:49Z

I put some comments (# To deprecate) for code that I think can be deprecated after removing CLI dependency for all platforms. I am planning to remove the unused code in a separate PR after we have finished every platform (in addition to some final refactoring).

Thanks @cindyyuanjiang
That's a great idea to mark things to be removed.
Can you please change that to a # TODO: tobe deprecated description? TODO is more like a standard and can be picked up by almost all the IDEs

amahussein · 2024-07-19T16:18:41Z

@nartal1 I remember you contributed to the usability tasks and you have knowledge in this area.
Can you please take a closer look at Cindy's PR and test it?

nartal1 · 2024-07-19T18:36:35Z

Thanks @cindyyuanjiang ! Tested your PR and it looks good. As per offline discussion it would be good if we could add log message that it is using the cached instance catalog file.

Another thing is, we see this log in Initialization phase which can be confusing:

2024-07-19 10:29:15,579 INFO rapids.tools.qualification: ******* [Initialization]: Starting *******
2024-07-19 10:29:15,602 INFO rapids.tools.qualification.ctxt: Start connecting to the platform
2024-07-19 10:29:15,602 WARNING rapids.tools.csp: Property profile is not set. Setting default value default from environment variable
2024-07-19 10:29:15,602 WARNING rapids.tools.csp: Property credentialFile is not set. Setting default value /home/nartal/.aws/credentials from environment variable
2024-07-19 10:29:15,602 WARNING rapids.tools.csp: Property cliConfigFile is not set. Setting default value /home/nartal/.aws/config from environment variable
2024-07-19 10:29:15,602 WARNING rapids.tools.csp: Property region is not set. Setting default value us-east-1 from environment variable
2024-07-19 10:29:15,602 WARNING rapids.tools.csp: Property output is not set. Setting default value json from environment variable
2024-07-19 10:29:15,603 WARNING rapids.tools.cmd_driver: Environment report: Private key file path is not set. It is required to SSH on driver node. Set RAPIDS_USER_TOOLS_KEY_PAIR_PATH

Can we remove it if property file is used for cluster argument. It can be in this PR or a follow on.
@amahussein - Wanted to know your thoughts on removing above log messages.

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

amahussein · 2024-07-19T19:23:17Z

Can we remove it if property file is used for cluster argument. It can be in this PR or a follow on.
@amahussein - Wanted to know your thoughts on removing above log messages.

Thanks @nartal1 that's a good point.
We have other issues to improve logging which can be addressed then.
@cindyyuanjiang you may want to rerun a e-to-e after upmerging. I wonder if PR #1188 would require you to handle some new code of blocks.

cindyyuanjiang · 2024-07-19T21:01:33Z

@cindyyuanjiang you may want to rerun a e-to-e after upmerging. I wonder if PR #1188 would require you to handle some new code of blocks.

Thanks @amahussein! I upmerged with dev branch and did another round of e-to-e testing. Results look good to me.

amahussein

Thanks @cindyyuanjiang !
I don't have specific comments on the changes.
If @nartal1 and @parthosa are okay with the changes, then we can move fwd with it.

user_tools/src/spark_rapids_pytools/cloud_api/databricks_aws.py

user_tools/src/spark_rapids_pytools/cloud_api/emr.py

nartal1

Thanks @cindyyuanjiang ! LGTM.

cindyyuanjiang · 2024-07-19T23:29:40Z

Follow up issue for logging: #1207

parthosa

Thanks @cindyyuanjiang. This looks great.

remvoe cli dependency for emr and databricks-aws platforms

b7fcc68

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

cindyyuanjiang mentioned this pull request Jul 16, 2024

[FEA] Add support to use local cached instance description files #1191

Closed

cindyyuanjiang self-assigned this Jul 16, 2024

cindyyuanjiang marked this pull request as draft July 16, 2024 22:52

cindyyuanjiang added 5 commits July 17, 2024 21:34

resolved tox errors

0add0f2

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

add back accidentially removed word in pyproject.toml

8d0fe0e

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

fixed issue with non exist catalog files

0df5759

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

fixed python unit test

0102c12

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

fix extracting platform name

4bc7c1e

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

cindyyuanjiang marked this pull request as ready for review July 18, 2024 23:21

cindyyuanjiang requested review from parthosa, amahussein, tgravescs and nartal1 and removed request for parthosa July 18, 2024 23:21

cindyyuanjiang added bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python) usability track issues related to the Tools's user experience labels Jul 18, 2024

cindyyuanjiang added 2 commits July 19, 2024 12:04

updated comments and add logging when loading instance desription files

c6aa922

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

fix python style

76e300b

Signed-off-by: cindyyuanjiang <cindyj@nvidia.com>

amahussein approved these changes Jul 19, 2024

View reviewed changes

nartal1 reviewed Jul 19, 2024

View reviewed changes

user_tools/src/spark_rapids_pytools/cloud_api/databricks_aws.py Outdated Show resolved Hide resolved

nartal1 reviewed Jul 19, 2024

View reviewed changes

user_tools/src/spark_rapids_pytools/cloud_api/emr.py Outdated Show resolved Hide resolved

nartal1 approved these changes Jul 19, 2024

View reviewed changes

parthosa approved these changes Jul 22, 2024

View reviewed changes

cindyyuanjiang merged commit 56e6962 into NVIDIA:dev Jul 22, 2024
14 checks passed

cindyyuanjiang deleted the spark-rapids-tools-1191-emr branch July 22, 2024 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEA] Remove CLI dependency for EMR and Databricks-AWS platforms in user tool #1196

[FEA] Remove CLI dependency for EMR and Databricks-AWS platforms in user tool #1196

cindyyuanjiang commented Jul 16, 2024 •

edited

Loading

cindyyuanjiang commented Jul 18, 2024

amahussein commented Jul 19, 2024

amahussein commented Jul 19, 2024

nartal1 commented Jul 19, 2024

amahussein commented Jul 19, 2024

cindyyuanjiang commented Jul 19, 2024

amahussein left a comment

nartal1 left a comment

cindyyuanjiang commented Jul 19, 2024

parthosa left a comment •

edited

Loading

[FEA] Remove CLI dependency for EMR and Databricks-AWS platforms in user tool #1196

[FEA] Remove CLI dependency for EMR and Databricks-AWS platforms in user tool #1196

Conversation

cindyyuanjiang commented Jul 16, 2024 • edited Loading

cindyyuanjiang commented Jul 18, 2024

amahussein commented Jul 19, 2024

amahussein commented Jul 19, 2024

nartal1 commented Jul 19, 2024

amahussein commented Jul 19, 2024

cindyyuanjiang commented Jul 19, 2024

amahussein left a comment

Choose a reason for hiding this comment

nartal1 left a comment

Choose a reason for hiding this comment

cindyyuanjiang commented Jul 19, 2024

parthosa left a comment • edited Loading

Choose a reason for hiding this comment

cindyyuanjiang commented Jul 16, 2024 •

edited

Loading

parthosa left a comment •

edited

Loading