Support _ in spark conf of integration tests

[gerashegalov]

Fixes #6351

• Allow escaping _ in Spark conf by duplicating _ to allow running Iceberg/ pytests with xdist

Additional refactoring and fixes

• Remove a stale comment from spark-tests.sh
• Allow overriding PYSP_TEST_spark_rapids_memory_gpu_allocSize to be able to run multiple Spark apps with multiple executors per pytest session with TEST_PARALLEL >= 2 on a small GPU
• Add support for --packages and --jars in xdist using findspark
  • Fix xdist failing to init findspark when pyspark already on PYTHONPATH
  • Enforce findspark usage with xdist
    • don't start Spark and allocate GPU resources for xdist master
    • remove direct imports of pyspark from conftest to make sure it happens after session start

Will file a follow-up issue to convert non-xdist tests relying on SPARK_SUBMIT_ARGS for external sources such as Delta and Iceberg to xdist . The following works locally:

export TEST_PARALLEL=2 
export SPARK_HOME=~/dist/spark-3.2.2-bin-hadoop3.2
export PYSP_TEST_spark_master="spark://$(hostname):7077"
export PYSP_TEST_spark_executor_cores="2"
export PYSP_TEST_spark_cores_max="4"
export PYSP_TEST_spark_jars_packages="org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.2"
export PYSP_TEST_spark_sql_extensions="org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
export PYSP_TEST_spark_sql_catalog_spark__catalog="org.apache.iceberg.spark.SparkSessionCatalog"
export PYSP_TEST_spark_sql_catalog_spark__catalog_type="hadoop"
export PYSP_TEST_spark_sql_catalog_spark__catalog_warehouse="$(mktemp -d -p /tmp spark-warehouse-XXXXXX)"
export PYSP_TEST_spark_rapids_memory_gpu_allocSize='512m'
./integration_tests/run_pyspark_from_build.sh -s -m iceberg --iceberg

Signed-off-by: Gera Shegalov gera@apache.org

[gerashegalov]

build

[revans2]

Most of the issues I have are nits. Combining a lot of small things together into a single PR is not ideal for me, but I am okay with it. Just confused because it was not reflected in the name of the PR at all. I had to read the description to know what was happening.

The one I really care about is setting the memory size for the test.

[gerashegalov]

Can't repro the CI failure locally, retrying

[gerashegalov]

build

[gerashegalov]

Since it's a very basic ClassNofFound for SQLPlugin class it may be related to a potential findspark version discrepancy.

[gerashegalov]

build

[gerashegalov]

The reason there is a discrepancy with the CI is that CI explicitly puts pyspark on the PYTHONPATH

2022-08-18T12:53:56.6273334Z [2022-08-18T12:46:09.218Z] + export PYTHONPATH=/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-5402-ci-2/.download/spark-3.1.1-bin-hadoop3.2/python:/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-5402-ci-2/.download/spark-3.1.1-bin-hadoop3.2/python/pyspark/:/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-5402-ci-2/.download/spark-3.1.1-bin-hadoop3.2/python/lib/py4j-0.10.9-src.zip

Thus the code that relies on pyspark not being found to fallback on findspark initialization does not kick in

On the other hand in the wrapper script we explicitly avoid using xdist if findspark is not found.

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[revans2]

Thanks for explaining the changes. This is fine. Just would like to see a comment on the one thing that is still left, but it is not critical.

[gerashegalov]

build