1.4.0 Release Candidate

[hcho3]

Roadmap: #6500

We are about to release version 1.4.0 of XGBoost. In the next two weeks, we invite everyone to try out the release candidate (RC).

Feedback period: until the end of April 10, 2021. No new feature will be added to the release; only critical bug fixes will be added.

@dmlc/xgboost-committer

Now available

Python package. RC1 available on PyPI. Try it out with the command

python -m pip install xgboost==1.4.0rc1

JVM packages. RC1 available from our Maven repository. Add XGBoost4J as dependency to your Java application.

Show instructions (Maven/SBT)

Maven

<dependencies>
  ...
  <dependency>
      <groupId>ml.dmlc</groupId>
      <artifactId>xgboost4j_2.12</artifactId>
      <version>1.4.0-RC1</version>
  </dependency>
  <dependency>
      <groupId>ml.dmlc</groupId>
      <artifactId>xgboost4j-spark_2.12</artifactId>
      <version>1.4.0-RC1</version>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>XGBoost4J Release Repo</id>
    <name>XGBoost4J Release Repo</name>
    <url>https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/</url>
  </repository>
</repositories>

SBT

libraryDependencies ++= Seq(
  "ml.dmlc" %% "xgboost4j" % "1.4.0-RC1",
  "ml.dmlc" %% "xgboost4j-spark" % "1.4.0-RC1"
)
resolvers += ("XGBoost4J Release Repo"
              at "https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/")

Starting from 1.2.0, XGBoost4J-Spark supports training with NVIDIA GPUs. To enable this capability, download artifacts suffixed with -gpu, as follows:

Show instructions (Maven/SBT)

Maven

<dependencies>
  ...
  <dependency>
      <groupId>ml.dmlc</groupId>
      <artifactId>xgboost4j-gpu_2.12</artifactId>
      <version>1.4.0-RC1</version>
  </dependency>
  <dependency>
      <groupId>ml.dmlc</groupId>
      <artifactId>xgboost4j-spark-gpu_2.12</artifactId>
      <version>1.4.0-RC1</version>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>XGBoost4J Release Repo</id>
    <name>XGBoost4J Release Repo</name>
    <url>https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/</url>
  </repository>
</repositories>

SBT

libraryDependencies ++= Seq(
  "ml.dmlc" %% "xgboost4j-gpu" % "1.4.0-RC1",
  "ml.dmlc" %% "xgboost4j-spark-gpu" % "1.4.0-RC1"
)
resolvers += ("XGBoost4J Release Repo"
              at "https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/")

PRs to be backported:

• Use CPU input for test_boost_from_prediction. #6818
• Optimize dart inplace predict perf. #6804
• Don't estimate sketch batch size when rmm is used. #6807
• Remove unnecessary calls to iota. #6797
• Fix approximated predict contribution. #6811
• More general predict proba. #6817
• Use batched copy if. #6826
• [CI] Pack R package tarball with pre-built xgboost.so (with GPU support) #6827

[hcho3]

Deprecation:

• This release will be the last release to support CUDA 10.0. ([RFC] Drop CUDA 10.0 support #6642)
• Starting in the next release, the Python package will require Pip 19.3+ due to the use of manylinux2014 tag. Also, CentOS 6, RHEL 6 and other old distributions will not be supported.

Known issue:

• MacOS build of the JVM packages doesn't support multi-threading out of the box. To enable multi-threading with JVM packages, MacOS users will need to build the JVM packages from the source. See https://xgboost.readthedocs.io/en/latest/jvm/index.html#installation-from-source

[Craigacp]

Is there an RC of XGBoost4J I can test out? Also will the 1.4.0 release include Windows binaries for XGBoost4J or is that going to be in a later release?

[trivialfis]

cc @hcho3

[hcho3]

Also will the 1.4.0 release include Windows binaries for XGBoost4J or is that going to be in a later release?

Yes.

Is there an RC of XGBoost4J I can test out?

I updated the post to show instructions for downloading RC1. The RC1 JAR has been generated by our CI server and doesn't yet contain the Windows DLL (it contains the Linux binary only).

[Craigacp]

Also will the 1.4.0 release include Windows binaries for XGBoost4J or is that going to be in a later release?

Yes.

Awesome.

Is there an RC of XGBoost4J I can test out?

I updated the post to show instructions for downloading RC1. The RC1 JAR has been generated by our CI server and doesn't yet contain the Windows DLL (it contains the Linux binary only).

Thanks. I tested it out and all the tests I have in Tribuo pass, though those are mostly functional rather than measuring runtime or statistical performance.

[trivialfis]

Draft for release note is here: #6843 .

[hcho3]

I will now make the final release and add the v1.4.0 tag. Thanks!

[trivialfis]

@CodingCat @hetong007 @hcho3 Hi I believe we can start submitting 1.4 to maven/cran/pip now.

[hcho3]

I just submitted XGBoost 1.4.0 to PyPI.

[trivialfis]

@hcho3 Thanks! I was just starting to check out the wheels from CI

[hetong007]

Seems the release_1.4.0 tag fails on win-builder: https://win-builder.r-project.org/6EiRDtHXdu43/00install.out

[trivialfis]

Hmm, something is changed in dmlc core ..

[trivialfis]

I will try that on R hub.

[trivialfis]

Building on release: https://builder.r-hub.io/status/xgboost_1.4.0.1.tar.gz-15c2d55160be5ecc324a6e209d34aea1
Failing on oldrel due to c++ requirement: https://builder.r-hub.io/status/xgboost_1.4.0.1.tar.gz-95a76e407d42fb216c2a8fa5455b0f80
But failing on dmlc core: https://ci.appveyor.com/project/tqchen/dmlc-core/builds/38684309/job/bw9dcj6rww3rg7o1

[hcho3]

@trivialfis I pushed a commit to dmlc/dmlc-core#647 fix the build for MSVC.

[hcho3]

@CodingCat This time, let's include the Windows binary xgboost4j.dll. You can locate it at https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/list.html?prefix=release_1.4.0/

[trivialfis]

@hetong007 Hi have you seen errors like this:

Pandoc (>= 1.12.3) not available. Falling back to R Markdown v1.

https://github.com/dmlc/xgboost/pull/6862/checks?check_run_id=2352185232

This pops up recently, googling returns lots cran check results that look similar but I can't find a concrete solution.

[hetong007]

Hmm, this looks new to me. May I ask:

1. Do we have this error before?
2. Do we indeed have pandoc (>= 1.12.3) configured on the CI server?
3. Based on the error messages below, seems we could also try adding markdown into the Suggest field of R-package/DESCRIPTION, could we try it?

[hetong007]

I read the related issue yihui/knitr#1864 and some other records referenced this one, probably that markdown into Suggest trick is the fix.

[trivialfis]

Thanks for the reply.

Do we have this error before?

No, this should be new within a week.

Do we indeed have pandoc (>= 1.12.3) configured on the CI server?

I need to check since it works fine before (and still works fine on r-hub). Maybe it's the rmarkdown updating dependency requirements?

Based on the error messages below, seems we could also try adding markdown into the Suggest field of R-package/DESCRIPTION, could we try it?

Yes. Then we will have both rmarkdown and markdown as suggested dependencies so I'm not entirely sure. But I will try that too.

[trivialfis]

Do we indeed have pandoc (>= 1.12.3) configured on the CI server?

I thought cran already has it instead of requiring a system-level installation?

[trivialfis]

Pushed to install pandoc: #6862 Let's see how it works. ;-)

[trivialfis]

@CodingCat Hi, if you prefer we @hcho3 can probably help to share some responsibility on uploading/managing the maven artifacts. @hcho3 has worked on lots of build automation

[trivialfis]

@hetong007 I got it working by installing pandoc on github action. #6862 .

[trivialfis]

It's probably a good idea to skip 1.4 for R and JVM and use 1.4.1

[Craigacp]

What's wrong with the JVM 1.4.0?

[trivialfis]

If you don't use GPU then nothing.

[trivialfis]

#6866

[Craigacp]

Ok. Is 1.4.0 still going to be pushed to Maven Central, or is that going to be skipped in favour of 1.4.1?

[CodingCat]

Hi, @trivialfis , is the fix for 1.4.1 a regression?

[trivialfis]

Yes, it's a regression. I will make a patch rel today.

[trivialfis]

@hetong007 Can we try to re-submit to cran, I think the needed dependencies should be available there since we have tested it on r-hub. We made a patch release on: #6867 .

[hetong007]

I checked out v1.4.1 and updated the submodules.

From the win-builder pre-check results, I have got this warning from the current R-current [1] and R-devel [2]:

* checking whether package 'xgboost' can be installed ... WARNING
Found the following significant warnings:
  amalgamation/../src/c_api/c_api.cc:738:74: warning: narrowing conversion of 'cols' from 'xgboost::bst_ulong' {aka 'long long unsigned int'} to 'size_t' {aka 'unsigned int'} inside { } [-Wnarrowing]
See 'd:/RCompile/CRANguest/R-release/xgboost.Rcheck/00install.out' for details.

[1] https://win-builder.r-project.org/9ist6LqnSY1f/00check.log
[2] https://win-builder.r-project.org/1CKtxuSKi640/00check.log

[hcho3]

FYI, @trivialfis and I have gained deployer access to the Maven Central repository (https://issues.sonatype.org/browse/OSSRH-67724). We will publish the 1.4.1 version to Maven Central once all pending issues are resolved.

cc @Craigacp

[hetong007]

R 1.4.1.1 is on its way to CRAN now. Thanks for the help @trivialfis !

[trivialfis]

@hetong007 Awesome! Thanks for managing the release.

[hcho3]

Checks for the R package are all passing: https://cran.r-project.org/web/checks/check_results_xgboost.html

Also, I've just released 1.4.1 to Maven Central. The artifacts will become available momentarily. Thanks everyone!

[hcho3]

@Craigacp XGBoost4J 1.4.1 is now on Maven Central: https://repo1.maven.org/maven2/ml/dmlc/xgboost4j_2.12/1.4.1/. The JAR file contains three native binaries, one each for three OSes (Linux, Windows, MacOS):

$ unzip -l xgboost4j_2.12-1.4.1.jar | grep lib/
        0  04-27-2021 03:28   lib/
        0  04-27-2021 03:28   lib/windows/
        0  04-27-2021 03:28   lib/windows/x86_64/
        0  04-27-2021 03:28   lib/linux/
        0  04-27-2021 03:28   lib/linux/x86_64/
        0  04-27-2021 03:28   lib/macos/
        0  04-27-2021 03:28   lib/macos/x86_64/
  2491392  04-27-2021 03:28   lib/windows/x86_64/xgboost4j.dll
  3412608  04-27-2021 03:28   lib/macos/x86_64/libxgboost4j.dylib
  5785304  04-27-2021 03:28   lib/linux/x86_64/libxgboost4j.so

[Craigacp]

That's great. I've made the PR to update my library, re-enable the Windows tests, and everything passed. Thanks for sorting out the Windows binary, I know it was a pain to get integrated.

[hcho3]

@Craigacp Now let's see if we can tackle #6767. In particular, we want to catch UnsatisfiedLinkException and display a good error message to direct users to install libomp.

[Craigacp]

Sure, that's easy. I'll work up a PR later this week or at the weekend. Basically we put a try catch around this line (https://github.com/dmlc/xgboost/blob/master/jvm-packages/xgboost4j/src/main/java/ml/dmlc/xgboost4j/java/NativeLibLoader.java#L115) and then we can pretty it up by restructuring the OS detection and rethrowing the exception with an OS based error message.

[Craigacp]

I opened PR #6926 which adds better error messages when library loading fails on the JVM.