Java 11 support

[ruebot]

From the Apache Spark mailing list:

Hi, All.

Thanks to your many many contributions,
Apache Spark master branch starts to pass on JDK11 as of today.
(with `hadoop-3.2` profile: Apache Hadoop 3.2 and Hive 2.3.6)

    https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/326/
    (JDK11 is used for building and testing.)

We already verified all UTs (including PySpark/SparkR) before.

Please feel free to use JDK11 in order to build/test/run `master` branch and
share your experience including any issues. It will help Apache Spark 3.0.0 release.

For the follow-ups, please follow https://issues.apache.org/jira/browse/SPARK-24417 .
The next step is `how to support JDK8/JDK11 together in a single artifact`.

Bests,
Dongjoon.

We'll align with Apache Spark here on Java 11 support. Once we have a Spark release with Java 11, I'll pivot to getting aut stable with Java 11.

[ruebot]

Getting closer to Spark 3.0.0!

Hi all,

To enable wide-scale community testing of the upcoming Spark 3.0 release, the Apache Spark community has posted a preview release of Spark 3.0. This preview is not a stable release in terms of either API or functionality, but it is meant to give the community early access to try the code that will become Spark 3.0. If you would like to test the release, please download it, and send feedback using either the mailing lists or JIRA.

There are a lot of exciting new features added to Spark 3.0, including Dynamic Partition Pruning, Adaptive Query Execution, Accelerator-aware Scheduling, Data Source API with Catalog Supports, Vectorization in SparkR, support of Hadoop 3/JDK 11/Scala 2.12, and many more. For a full list of major features and changes in Spark 3.0.0-preview, please check the thread(http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-3-0-preview-release-feature-list-and-major-changes-td28050.html).

We'd like to thank our contributors and users for their contributions and early feedback to this release. This release would not have been possible without you.

To download Spark 3.0.0-preview, head over to the download page: https://archive.apache.org/dist/spark/spark-3.0.0-preview

Thanks,

Xingbo

[ruebot]

Successful build:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:58 min
[INFO] Finished at: 2019-11-09T18:19:56-05:00
[INFO] ------------------------------------------------------------------------

Successful Spark 3.0.0-preview load with ~/bin/spark-3.0.0-preview-bin-hadoop2.7/bin/spark-shell --master local\[10\] --driver-memory 35g --packages io.archivesunleashed:aut:0.18.1-SNAPSHOT:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-preview
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 11.0.4)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

BUT! We're broken somewhere:

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._

RecordLoader.loadArchives("/home/nruest/Projects/au/aut/src/test/resources/arc/example.arc.gz", sc).keepValidPages()
  .map(r => ExtractDomain(r.getUrl))
  .countItems()
  .take(10)

// Exiting paste mode, now interpreting.

java.lang.NoSuchMethodError: 'scala.collection.mutable.ArrayOps scala.Predef$.refArrayOps(java.lang.Object[])'
  at io.archivesunleashed.package$RecordLoader$.getFiles(package.scala:54)
  at io.archivesunleashed.package$RecordLoader$.loadArchives(package.scala:68)
  ... 47 elided

I'll have to dig in more later.

[ruebot]

Successful run here: 06764f7!!

[ruebot]

Hi all,

To enable wide-scale community testing of the upcoming Spark 3.0 release, the Apache Spark community has posted a new preview release of Spark 3.0. This preview is not a stable release in terms of either API or functionality, but it is meant to give the community early access to try the code that will become Spark 3.0. If you would like to test the release, please download it, and send feedback using either the mailing lists or JIRA.

There are a lot of exciting new features added to Spark 3.0, including Dynamic Partition Pruning, Adaptive Query Execution, Accelerator-aware Scheduling, Data Source API with Catalog Supports, Vectorization in SparkR, support of Hadoop 3/JDK 11/Scala 2.12, and many more. For a full list of major features and changes in Spark 3.0.0-preview2, please check the thread(http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-3-0-preview-release-feature-list-and-major-changes-td28050.html and http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-3-0-preview-release-2-td28491.html).

We'd like to thank our contributors and users for their contributions and early feedback to this release. This release would not have been possible without you.

To download Spark 3.0.0-preview2, head over to the download page: https://archive.apache.org/dist/spark/spark-3.0.0-preview2

Happy Holidays.

Yuming

I'll get the branch updated shortly.

[ruebot]

Resolved with: 59b1d4e

[manyas1]

Spark team, is 3.0 official to use with Java11 then ?

[davidghiurco]

Spark team, is 3.0 official to use with Java11 then ?

According to documentation, yes, but I am unable to build a Spark 3.1.2 project with Maven 3.8.1/JDK 11:

`
[ERROR] WARNING: An illegal reflective access operation has occurred

[ERROR] WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:m2/repository/org/apache/spark/spark-unsafe_2.12/3.1.2/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
`

According to https://spark.apache.org/docs/3.1.2/:
For Java 11, -Dio.netty.tryReflectionSetAccessible=true is required additionally for Apache Arrow library. This prevents java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available when Apache Arrow uses Netty internally.

But even after adding the following to my environment
`
export MAVEN_OPTS="-Dio.netty.tryReflectionSetAccessible=true"

export JAVA_OPTS="-Dio.netty.tryReflectionSetAccessible=true"
`

I am still unable to build the Spark 3.1.2 project with JDK 11 ......