[FEA] jdk17 support

[tgravescs]

Is your feature request related to a problem? Please describe.
Spark 3.3 is adding support for jdk17 - https://issues.apache.org/jira/browse/SPARK-33772

Filing this just to track to see if we want to support the same versions.

Tasks

Preview Give feedback

1. Update documentation to build with JDK9+
2. Add JDK17 to mvn-verify-check build matrix - JDK17 build support in CI #8234
3. Create a Unit Test setup for JDK17: extraTestArgs etc - Partial JDK 17 support #8228
4. Create a JDK17 CI pipeline

[gerashegalov]

After recent improvements like #5646 this may be largely working, and just need updates in the doc and continuous testing.

Verified with:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export SPARK_HOME=~/dist/spark-3.3.1-bin-hadoop3
export TEST_PARALLEL=0
export NUM_LOCAL_EXECS=2 
mvn clean package -pl tests,integration_tests -am -Dbuildver=331 -Dskip
./integration_tests/run_pyspark_from_build.sh -k 'cast' -s

Unit tests will need extra options for JVM

  java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x3b192d32) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x3b192d32
  at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
  at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
  at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)

[pxLi]

I just simply duplicated an internal jdk17 build pipeline from existing jdk11-nightly one (https://github.com/NVIDIA/spark-rapids/blob/branch-23.06/CONTRIBUTING.md#building-and-testing-with-jdk9)
with ENV

+ update-java-alternatives --set /usr/lib/jvm/java-1.17.0-openjdk-amd64
+ java --version
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10-Ubuntu-0ubuntu120.04.1)
OpenJDK 64-Bit Server VM (build 17.0.6+10-Ubuntu-0ubuntu120.04.1, mixed mode, sharing)
+ mvn -version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
JAVA_HOME='/usr/lib/jvm/java-17-openjdk-amd64

mvn -B clean package -s jenkins/settings.xml -P mirror-apache-to-urm \
-Dbuildver=330 -Dmaven.compiler.release=11 -Dmaven.compiler.source=11 \
-Dmaven.compiler.target=11 -Dscala.plugin.version=4.6.1 \
-DallowConventionalDistJar=true

it threw me

09:30:26  [INFO] compiling 378 Scala sources and 59 Java sources to /home/jenkins/agent/workspace/jenkins-JDK17-nightly-1/sql-plugin/target/spark330/classes ...
09:30:41  [ERROR] /home/jenkins/agent/workspace/jenkins-JDK17-nightly-1/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleTransport.scala:536: object nio is not a member of package sun
09:30:42  [ERROR] one error found
09:30:42  [INFO] ------------------------------------------------------------------------
09:30:42  [INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 23.06.0-SNAPSHOT:
09:30:42  [INFO] 
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [ 31.022 s]
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... FAILURE [01:30 min]
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Delta Lake 2.1.x Support SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Delta Lake 2.2.x Support SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Aggregator ..... SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Distribution ... SKIPPED
09:30:42  [INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
09:30:42  [INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
09:30:42  [INFO] ------------------------------------------------------------------------
09:30:42  [INFO] BUILD FAILURE
09:30:42  [INFO] ------------------------------------------------------------------------
09:30:42  [INFO] Total time:  02:01 min
09:30:42  [INFO] Finished at: 2023-04-25T01:30:42Z
09:30:42  [INFO] ------------------------------------------------------------------------
09:30:42  [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.6.1:compile (scala-compile-first) on project rapids-4-spark-sql_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.6.1:compile failed: Compilation failed: InterfaceCompileFailed -> [Help 1]
09:30:42  [ERROR] 
09:30:42  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
09:30:42  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
09:30:42  [ERROR] 
09:30:42  [ERROR] For more information about the errors and possible solutions, please read the following articles:
09:30:42  [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
09:30:42  [ERROR] 
09:30:42  [ERROR] After correcting the problems, you can resume the build with the command
09:30:42  [ERROR]   mvn <args> -rf :rapids-4-spark-sql_2.12

Looks like we may also need to add jvm options to allow access internal package for our build in jdk17 ENV
https://github.com/apache/spark/blob/master/pom.xml#L308-L325

please help update the instruction if we require building plugin using jdk17, thanks!

[gerashegalov]

[ERROR] /home/jenkins/agent/workspace/jenkins-JDK17-nightly-1/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleTransport.scala:536: object nio is not a member of package sun

This looks like a manifestation of the bug scala/bug#11814 (comment)

However, we don't really yet have the requirement to build for a lower-version JDK using higher-version JDK, The same command for the 17 source, target, release is working

JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 mvn -B clean package -Dbuildver=330 -Dmaven.compiler.release=17 -Dmaven.compiler.source=17 -Dmaven.compiler.target=17 -Dscala.plugin.version=4.6.1 -DallowConventionalDistJar=true

[pxLi]

09:07:22  [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ rapids-4-spark-udf_2.12 ---
09:07:22  [INFO] Tests are skipped.
09:07:22  [INFO] 
09:07:22  [INFO] --- scalatest-maven-plugin:2.0.2:test (test) @ rapids-4-spark-udf_2.12 ---
09:07:22  Discovery starting.
09:07:22  Discovery completed in 2 seconds, 28 milliseconds.
09:07:22  Run starting. Expected test count is: 0
09:07:22  OpcodeSuite:
09:07:22  com.nvidia.spark.OpcodeSuite *** ABORTED ***
09:07:22    java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: com.nvidia.spark.OpcodeSuite
09:07:22    at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
09:07:22    at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
09:07:22    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
09:07:22    at scala.collection.Iterator.foreach(Iterator.scala:943)
09:07:22    at scala.collection.Iterator.foreach$(Iterator.scala:943)
09:07:22    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
09:07:22    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
09:07:22    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
09:07:22    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
09:07:22    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
09:07:22    ...
09:07:22    Cause: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x368239c8) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x368239c8
09:07:22    at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
09:07:22    at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
09:07:22    at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)
09:07:22    at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
09:07:22    at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
09:07:22    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
09:07:22    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
09:07:22    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
09:07:22    at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
09:07:22    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
09:07:22    ...
09:07:22  *** RUN ABORTED ***
09:07:22    java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x368239c8) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x368239c8
09:07:22    at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
09:07:22    at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
09:07:22    at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)
09:07:22    at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
09:07:22    at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
09:07:22    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
09:07:22    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
09:07:22    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
09:07:22    at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
09:07:22    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
09:07:22    ...
09:07:22  [INFO] ------------------------------------------------------------------------

Use new maven compiler (17) passed compiling part but failed UT, we may still require to add the extra args for UT running https://github.com/apache/spark/blob/master/pom.xml#L308-L325

Sorry, unassigned myself for now. I do not have time to work on the new instruction, it would be nice to have some developer to help make sure all commands work and update the doc first. Then I will setup pipeline to continually verify the recommended way to build and test w/ new java runtime, thanks!

[gerashegalov]

agree regarding https://github.com/apache/spark/blob/master/pom.xml#L308-L325

#8056 (comment)

[pxLi]

I will enable the regular run of JDK17 pipeline after #8234

[pxLi]

Merged change, and enable regular nightly CI for jdk17 build and test (UT+IT)
Closing this ticket. create ticket #8275 to follow up maven-verify simplify work