Automatic conversion to shimplified directory structure [databricks]

[gerashegalov]

Command executed:

mvn generate-sources -Dshimplify=true -Dshimplify.move=true

Verified with:

1. Build a multi-shim jar

mvn clean
echo 314 321 331 | xargs -t -n 1 bash -c 'mvn package -pl aggregator -am -Dbuildver=$1  -DskipTests -Dskip -Dmaven.jadadoc.skip' '{}'
mvn package -pl dist -Dincluded_buildvers=314,321,331 -Ddist.jar.compress=false 

2. Run a smoke test for all shims

SPARK_HOME=~/gits/apache/spark NUM_LOCAL_EXECS=2 PYSP_TEST_spark_rapids_shuffle_mode=MULTITHREADED PYSP_TEST_spark_rapids_shuffle_multiThreaded_writer_threads=2 PYSP_TEST_spark_rapids_shuffle_multiThreaded_reader_threads=2 PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.spark314.RapidsShuffleManager PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0 PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.1 PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.1 ./integration_tests/run_pyspark_from_build.sh -k test_hash_grpby_sum

SPARK_HOME=~/dist/spark-3.2.1-bin-hadoop3.2 NUM_LOCAL_EXECS=2 PYSP_TEST_spark_rapids_shuffle_mode=MULTITHREADED PYSP_TEST_spark_rapids_shuffle_multiThreaded_writer_threads=2 PYSP_TEST_spark_rapids_shuffle_multiThreaded_reader_threads=2 PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.spark321.RapidsShuffleManager PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0 PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.1 PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.1 ./integration_tests/run_pyspark_from_build.sh -k test_hash_grpby_sum

SPARK_HOME=~/dist/spark-3.3.1-bin-hadoop3 NUM_LOCAL_EXECS=2 PYSP_TEST_spark_rapids_shuffle_mode=MULTITHREADED PYSP_TEST_spark_rapids_shuffle_multiThreaded_writer_threads=2 PYSP_TEST_spark_rapids_shuffle_multiThreaded_reader_threads=2 PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.spark331.RapidsShuffleManager PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0 PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.1 PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.1 ./integration_tests/run_pyspark_from_build.sh -k test_hash_grpby_sum

Separate PR will be posted removing old sparkXYZ.sources definitions for the previous build. We'll merge it once we are confident in the transition robustness.

Signed-off-by: Gera Shegalov gera@apache.org

[gerashegalov]

build

[firestarman]

How do we add a new shim according to this new comment way ?

[GaryShen2008]

To add a new spark version support, it'll require us to update the comments in all the necessary files one by one?

[res-life]

After selecting a Spark3xx profile in IDEA Maven panel, seems IDEA can't automatically add the codes in target/src as source code.
There are some compile errors in the IDEA GUI although actually they are not and the whole project can be compiled by "mvn clean install".

[abellina]

I do not understand how this patch makes the codebase easier to work with. Now we have encoded into a comment the information on what version of Spark should each file go to? This seems like a burden to maintain and I do not know what makes the current method so bad that we need to change things this drastically.

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[jlowe]

Apologies for the long delay in reviewing this again. I wasn't able to review every single file, but what I did review looks correct to me and tests passing is a huge part of verifying this change. Thanks @gerashegalov for all the hard work on this!