[BUG] Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly intermittently

[pxLi]

Describe the bug

Intermittently seeing

[2021-12-07T06:58:00.874Z]     from spark_init_internal import get_spark_i_know_what_i_am_doing
[2021-12-07T06:58:00.874Z] ../../src/main/python/spark_init_internal.py:21: in <module>
[2021-12-07T06:58:00.874Z]     findspark.init()
[2021-12-07T06:58:00.874Z] /usr/local/lib/python3.8/dist-packages/findspark.py:145: in init
[2021-12-07T06:58:00.874Z]     raise Exception(
[2021-12-07T06:58:00.874Z] E   Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly

in pre_merge and nightly CI

Hard to reproduce, looks like PYTHONPAYH somehow got changed during pytest
Open an issue to track

[pxLi]

will keep monitoring this after #4349

[pxLi]

closed. this did not appear again after #4349

[pxLi]

seeing this again in one build of #6355
As this still seems ephemeral, I will spend some time try dig out the root cause

[pxLi]

did some investigation, the path is correct and all the required files are there.

Seems the issue was due to some NFS glitches, I will try figure out a way to remediate this later

[gerashegalov]

I noticed in #6358 that we don't properly call findspark.init when pyspark is manually added to PYTHONPATH. Maybe it is a fix for py4j as well.

UPDATE: Unlikely, since PYTHONPATH was added as part of a fix to this issue to begin with