[BUG] Profiler takes 60 minutes to process and generate output on integration tests eventlog

[tgravescs]

Describe the bug
The ci/cd integration tests can take up to 60 minutes to run the profiler on a single eventlog. The eventlog is generated from the spark-rapids integration tests. We should investigate why it takes so long for that eventlog.

See our CI/CD logs on how to get that file.

[amahussein]

I executed the profiler locally. This is relativelly large eventlog compared to other ones we have in our testing enviornments.
The problem is triggered by GC optimization issue.
The allocation rate of the core tools is considered high compared to the amount of processing. This is a very corner case when it comes to GC heuristics. This is well known problem in GC heap heuristics ("automatically decide the beast heap size")
By default, we assume this JDK8 and we want to use G1GC to improve performance. However, G1GC by default assigns 5% of the heap size to the new generation. Thgerefore, with a Xmx=100G, the new generation is as little as 12G.
During rumtime, the heap won't exceed the limit of ~16GB. Everytime the Heap reaches its local maxima, it frees enough memory thinking that there is no need to increase the heap size. The entire runtime, the JVM is using only 16GB out of 100GB causing as many GCs as would be triggered if we set the max heap size to 16GB.

I explored some G1GC tuning, but I don't think this is going to be optimum.
For example:

-Xms32G -Xmx64G -XX:+UseG1GC
# took 18 minutes -> heap can grow bigger reaching up to 20 GB and it performed faster than 100GB max heap

-Xms10G -Xmx100G -XX:+UseG1GC
# took 28 minutes

# Adjusted the new object sizes to grow to 60%
-Xms10G -Xmx64g -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=60
# took 31 minutes
# heap did not exceed 16 GB

-Xms32G -Xmx64g -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=60
# took 31 minutes

-Xms16G -Xmx24g -XX:+UseG1GC
# took 21 minutes -> again faster than -Xmx 64g

Then I experimented without G1GC. My conclusion that the Xms could be a factor in the performance to work around the corner case.

-Xmx64g
# 31 minutes -> heap was constantly around 8GB

-Xms16G -Xmx64g
# 16 minutes -> achieved the same execution time as 100GB

-Xms32G -Xmx64g
# 16 minutes -> almost no improvement. from xms 16GB

-Xms16G -Xmx24g
# 21 minutes -> better than just setting Xmxx without Xms

[amahussein]

This issue needs two parts:

• user-tools wrapper should set the -Xms to be 0.5 of the -XmX
• Fix the internal CI/CD scripts to set both Xmx and Xms