Include number of executors per node in cluster information

[parthosa]

Fixes #1117. Currently for cluster information, we calculate the number of nodes correctly but do not track the number of executors per node. This can generate wrong GPU cluster recommendations because there can be multiple executors per node.

This PR adds numExecsPerNode in the cluster information output file.

Changes:

Core/Java:

• Calculate numExecsPerNode as maximum number of executors in any host
• Update ClusterInfo and related methods.
• Add Num Executor Per Node as new field in cluster information output CSV file.
• Update unit test to include case with multiple executors on single node.

Output

Cluster Information Generated from Core:

File: rapids_4_spark_qualification_output_cluster_information.json

Previously:

[ {
  "appName" : "NDS - Power Run",
  "appId" : "application_169685947xxxxx",
  "eventLogPath" : "file:/Users/psarthi/Work/event-logs/xxxxxx",
  "clusterInfo" : {
    "vendor" : "dataproc",
    "coresPerExecutor" : 16,
    "numExecutorNodes" : 4,
    "driverHost" : "xxxx-dataproc-cpu-m.c.xxxx.internal"
  }
} ]

After this fix:

[ {
  "appName" : "NDS - Power Run",
  "appId" : "application_169685947xxxxx",
  "eventLogPath" : "file:/Users/psarthi/Work/event-logs/xxxxxx",
  "clusterInfo" : {
    "vendor" : "dataproc",
    "coresPerExecutor" : 16,
    "numExecsPerNode" : 6,
    "numExecutorNodes" : 4,
    "driverHost" : "xxxx-dataproc-cpu-m.c.xxxx.internal"
  }
} ]

Follow Up

• Need to investigate heuristics for calculating total CPU cores of the node.
• We cannot perform coresPerNode = numExecsPerNode * coresPerExecutor since a node maybe oversubscribed.
• Original issue: [FEA] Qualification tool should recommend the cluster shape based on the best TCO according to our internal benchmark #1109

[tgravescs]

so we don't want to just use activeHosts, we want to use all hosts to determine execs per node. That will be more likely right in case as an application finishes, if it has dynamic allocation on, it could release executors and thus the final active count could be inaccurate.

Put the number of executor nodes back to the active hosts for now but really we should try to do some timelining of this to see what was the max number in use at any time, but filed #1121 to followup with this. Put it back to what it was for now.

[tgravescs]

also it would be good to get a event log where dynamic allocation is disabled and the number of execs/hosts change over time. This is relatively easy in interactive, where you just run something then wait for the execs to idle timeout. then maybe run something again, maybe smaller.

[parthosa]

we don't want to just use activeHosts, we want to use all hosts to determine execs per node.

Updated the code to use all hosts.

Put the number of executor nodes back to the active hosts for now

Reverted to the original logic.

[parthosa]

Created a scenario for this (I think this meant dynamic allocation to be enabled):

get a event log where dynamic allocation is disabled and the number of execs/hosts change over time

Steps:

1. Created a dataproc cluster with 4 n1-standard-16 worker nodes.
2. Start a spark shell with --conf spark.dynamicAllocation.enabled=true --conf spark.executor.instances=8 --conf spark.executor.cores=8
3. Run a large spark application. UI shows all 8 executors are active (2 per worker node).
4. Wait for 30min. 7 executors are dead.
5. Run a small spark application. UI shows it 1 executor is active.

Now since we are calculating execs from all hosts, we will get correct num of exec/hosts as 2 instead of 1.

[tgravescs]

do we have any tests with the dynamic allocation test you had in the description?

[parthosa]

do we have any tests with the dynamic allocation test you had in the description?

Added unit test for dynamic allocation with comment

[tgravescs]

We cannot perform coresPerNode = numExecsPerNode * coresPerExecutor since a node maybe oversubscribed.

There is no way for us to know this on some platforms like yarn where they select what it schedules by. We will for now just have to make an assumption that it isn't but document and warn user