[FEA] Adding Benchmarking classes to evaluate core tools performance

[bilalbari]

Fixes #1120

This PR solves for adding benchmarking for tools.

Classes -

1. Benchmark - The utility class for Benchmarking
2. BenchmarkBase - The base class for extending and writing a benchmarker
3. BenchmarkArgs - Argument parsing class for benchmarking
4. SingleThreadedQualToolBenchmark - A template benchmark class for gathering metrics for qual tool

Writing a benchmark -

1. Extend BenchmarkBase and override the runBenchmarkSuite function
2. Write logically similar benchmarks inside the runBenchmark which will add a header to the output with the name provided.
3. Now for each case to be tested, use the Benchmark class addCase to write various cases of the same Benchmark. Included example benchmarks using the --per-sql tag with the qualification tool

Running the benchmark -

1. Use the java command to run the created Benchmark class with the following supported params -

• -i : total number of iterations to run to calculate average metrics
• -w : total number of warmup iterations to run before calculating the final metrics ( warmup is relavant so that final results are not skewed by the initial java classloading times )
• -o : output directory where to store the final result file. Default to the directory rapids-tools-benchmark in the root directory
• -f : output format of the stored result. Currently supports text. Json to be added in future iterations
• -a : input arguments to pass the underlying benchmark classes

Eg-

java -cp $CLASSPATH \
  com.nvidia.spark.rapids.tool.benchmarks.SingleThreadedQualToolBenchmark \
  -i 3 -w 3 -a " --output-directory output eventlogs"

Sample Output -

================================================================================================
Benchmark_Per_SQL_Arg_Qualification
================================================================================================

JVM Name                   :   OpenJDK 64-Bit Server VM 
Java Version               :   1.8.0_412 
OS Name                    :   Linux 
OS Version                 :   6.5.0-41-generic 
MaxHeapMemory              :   14219 MB 
Total Warm Up Iterations   :   0 
Total Runtime Iterations   :   1 
 
Benchmarker:                              Best Time(ms)   Avg Time(ms)   Stdev(ms)      Avg GC Time(ms)       Avg GC Count     Stdev GC Count    Max GC Time(ms)       Max GC Count   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Enable_Per_SQL_Arg_Qualification                  22963          22963           0                812.0               28.0                  0                812                 28      1.00X
Disable_Per_SQL_Arg_Qualification                 20886          20886           0                434.0                9.0                  0                434                  9      1.10X

Output metrics -

• Best Time(ms) - best runtime over i iterations
• Avg Time(ms) - Avg runtime calculated over i iterations
• Stdev(ms) - standard deviation of the runtime to realise variance in runtime
• Avg GC Time - Average time spend doing garbage collection over i iterations
• Avg GC Count - average time garbage collection is triggered
• Max GC Time - max time spend doing GC during various iterations
• Max GC Count - maximum GC triggered during any one of the runs
• Stdec GC Count - standard deviation of GC counts over GC iterations
• Relative - this takes the first case as a baseline and gives a metric of how much faster the next cases are ( 1.2x, 1.5x etc. )

These changes are subject to be extended in future iterations

[parthosa]

Thanks @bilalbari for the PR. Made some initial comments.

[parthosa]

Thanks @bilalbari. The refactoring has simplified a lot. Made some minor comments.

[cindyyuanjiang]

Are we planning to include any documentation on how to run the benchmarks?

[cindyyuanjiang]

Thanks @bilalbari for this feature! Some comments on the README.

[parthosa]

Thank you @bilalbari for this. This feature will be really helpful for tools 🎉

[amahussein]

LGTME!
Thanks @bilalbari for great effort working on that feature.

[amahussein]

Thanks @bilalbari !
LGTME

[cindyyuanjiang]

Thank you @bilalbari! LGTM.