[jvm-packages] Fix deterministic partitioning with dataset containing Double.NaN

[Totoketchup]

The functions featureValueOfSparseVector or featureValueOfDenseVector could return a Float.NaN if the input vector was containing any missing values. This would make fail the partition key computation and most of the rows would end up in the same partition and therefore leading to memory issues on one of the executor. We fixed this by avoid returning a NaN and simply use the row HashCode in this case.
We added a test to ensure that the repartition is indeed now uniform on input dataset containing missing values by checking that the partitions size variance is below a certain threshold.

Another remark about the computation of the key:

• If the values contained in your DataFrame are only below let's say 1e-4 / 1e-5, then this computation:
  rowHashCode.toLong + featureValue is equivalent to rowHashCode.toLong, wouldn't it better to make it more random by computing featureValue.toString().hashCode() ? It would even fix what I did here too. WDYT ?

[codecov-commenter]

Codecov Report

Merging #5996 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5996   +/-   ##
=======================================
  Coverage   78.52%   78.52%           
=======================================
  Files          12       12           
  Lines        3013     3013           
=======================================
  Hits         2366     2366           
  Misses        647      647           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b2a26f...8dd1d5c. Read the comment docs.

[Totoketchup]

Hello @CodingCat, I see that you are the one that designed the deterministic repartitioning for XGboost looking at #4786.

I have a questiion about the computation of the partition key:

Why math.abs(row.hashCode) % numWorkers would not be enough to compute the key ? Looking at the implementation of row.hashCode (https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala) we can see that it already uses the values inside the row to compute this hash. What does it bring to get a feature from the Row and add it to the computation of the key ?

[CodingCat]

LGTM thanks! the failed test seems irrelevant, restarted the test

[hcho3]

I'm looking at the failing tests now: #6014

[hcho3]

Restarting the tests.