Add parallel support to nightly spark standalone tests

[pxLi]

Signed-off-by: Peixin Li pxli@nyu.edu

fix #2802

To speed our nightly spark standalone integrations tests,
spark 3.0.x total time: ~3h 40m to ~1h 15m
spark 3.1.x total time: ~4h to ~1h 35m (include extra ParquetCachedBatchSerializer cache_test)

I am still doing more verification on other scenarios, submit first to collect feedbacks, thanks!

This requires to be enabled in nightly pipelines' Jenkinsfile separately

[jlowe]

I thought we were tagging the source to indicate they were slow as in #3241. Curious why this isn't leveraging that, e.g.: time_consuming_tests=$(grep -rl pytest.mark.slow_test "$SCRIPT_PATH"/src/main/python)

[pxLi]

slow_tag there was for pre-merge parallel split (either time-consuming or high memory usage), mostly to balance test time of two parallel test stages. The tag naming looks ambiguous, I talked to Alex, he will help to rename this tag
In nightly cases, I want pick time-consuming ones only, and per my tests join_test (3000s) + generate_expr_test (1800s) were the only 2 tests that spent consistently over 15 mins, so I manually put them here to avoid randomly putting them in the middle or tail of task queue.

[jlowe]

Should we split the test cases in join_test and generate_expr_test so they spread more evenly even without using the parallel hack?

[pxLi]

Yes, split in test-cases level that would be better. I have tested w/ a few non-serious cases-level split but not getting much benefits from it, and this could make it harder for developer to manage the test scenarios. I would like to get back to this if current setup does not fulfill our efficiency requirement

[pxLi]

build

[jlowe]

Looks OK to me other than we may want a followup to split some of these expensive test cases to make it easier for pytest to make better decisions about running them in parallel on its own.

[pxLi]

Looks OK to me other than we may want a followup to split some of these expensive test cases to make it easier for pytest to make better decisions about running them in parallel on its own.

Thanks! I filed an issue #3279 to track the followup