Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[jvm-packages] [CI] Create a Maven repository to host SNAPSHOT JARs #5533

Merged
merged 1 commit into from
Apr 15, 2020
Merged

[jvm-packages] [CI] Create a Maven repository to host SNAPSHOT JARs #5533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,17 @@ pipeline {
milestone ordinal: 4
}
}
stage('Jenkins Linux: Deploy') {
agent none
steps {
script {
parallel ([
'deploy-jvm-packages': { DeployJVMPackages(spark_version: '2.4.3') }
])
}
milestone ordinal: 5
}
}
}
}

Expand Down Expand Up @@ -383,3 +394,18 @@ def TestR(args) {
deleteDir()
}
}

def DeployJVMPackages(args) {
node('linux && cpu') {
unstash name: 'srcs'
if (env.BRANCH_NAME == 'master' || env.BRANCH_NAME.startsWith('release')) {
echo 'Deploying to xgboost-maven-repo S3 repo...'
def container_type = "jvm"
def docker_binary = "docker"
sh """
${dockerRun} ${container_type} ${docker_binary} tests/ci_build/deploy_jvm_packages.sh ${args.spark_version}
"""
}
deleteDir()
}
}
191 changes: 126 additions & 65 deletions doc/jvm/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,128 +8,189 @@ XGBoost JVM Package
<img alt="Build Status" src="https://travis-ci.org/dmlc/xgboost.svg?branch=master">
</a>
<a href="https://github.com/dmlc/xgboost/blob/master/LICENSE">
<img alt="GitHub license" src="http://dmlc.github.io/img/apache2.svg">
<img alt="GitHub license" src="https://dmlc.github.io/img/apache2.svg">
</a>

You have found the XGBoost JVM Package!

.. _install_jvm_packages:

************
Installation
************

Installation from source
========================

Building XGBoost4J using Maven requires Maven 3 or newer, Java 7+ and CMake 3.3+ for compiling the JNI bindings.

Before you install XGBoost4J, you need to define environment variable ``JAVA_HOME`` as your JDK directory to ensure that your compiler can find ``jni.h`` correctly, since XGBoost4J relies on JNI to implement the interaction between the JVM and native libraries.
.. contents::
:local:
:backlinks: none

After your ``JAVA_HOME`` is defined correctly, it is as simple as run ``mvn package`` under jvm-packages directory to install XGBoost4J. You can also skip the tests by running ``mvn -DskipTests=true package``, if you are sure about the correctness of your local setup.
Installation from Maven repository
==================================

To publish the artifacts to your local maven repository, run
Access release version
----------------------
You can use XGBoost4J in your Java/Scala application by adding XGBoost4J as a dependency:

.. code-block:: bash
.. code-block:: xml
:caption: Maven

<properties>
...
<!-- Specify Scala version in package name -->
<scala.binary.version>2.12</scala.binary.version>
</properties>

<dependencies>
...
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j_${scala.binary.version}</artifactId>
<version>latest_version_num</version>
</dependency>
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j-spark_${scala.binary.version}</artifactId>
<version>latest_version_num</version>
</dependency>
</dependencies>

mvn install
.. code-block:: scala
:caption: sbt

Or, if you would like to skip tests, run
libraryDependencies ++= Seq(
"ml.dmlc" %% "xgboost4j" % "latest_version_num",
"ml.dmlc" %% "xgboost4j-spark" % "latest_version_num"
)

.. code-block:: bash
This will check out the latest stable version from the Maven Central.

mvn -DskipTests install
For the latest release version number, please check `here <https://github.com/dmlc/xgboost/releases>`_.

This command will publish the xgboost binaries, the compiled java classes as well as the java sources to your local repository. Then you can use XGBoost4J in your Java projects by including the following dependency in ``pom.xml``:
.. note:: Using Maven repository hosted by the XGBoost project

.. code-block:: xml
There may be some delay until a new release becomes available to Maven Central. If you would like to access the latest release immediately, add the Maven repository hosted by the XGBoost project:

<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j</artifactId>
<version>latest_source_version_num</version>
</dependency>
.. code-block:: xml
:caption: Maven

For sbt, please add the repository and dependency in build.sbt as following:
<repository>
<id>XGBoost4J Release Repo</id>
<name>XGBoost4J Release Repo</name>
<url>https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/</url>
</repository>

.. code-block:: scala
.. code-block:: scala
:caption: sbt

resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"
resolvers += "XGBoost4J Release Repo" at "https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/release/"

"ml.dmlc" % "xgboost4j" % "latest_source_version_num"
Access SNAPSHOT version
-----------------------

If you want to use XGBoost4J-Spark, replace ``xgboost4j`` with ``xgboost4j-spark``.
First add the following Maven repository hosted by the XGBoost project:

.. note:: XGBoost4J-Spark requires Apache Spark 2.3+
.. code-block:: xml
:caption: Maven

XGBoost4J-Spark now requires **Apache Spark 2.3+**. Latest versions of XGBoost4J-Spark uses facilities of `org.apache.spark.ml.param.shared` extensively to provide for a tight integration with Spark MLLIB framework, and these facilities are not fully available on earlier versions of Spark.
<repository>
<id>XGBoost4J Snapshot Repo</id>
<name>XGBoost4J Snapshot Repo</name>
<url>https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/</url>
</repository>

Also, make sure to install Spark directly from `Apache website <https://spark.apache.org/>`_. **Upstream XGBoost is not guaranteed to work with third-party distributions of Spark, such as Cloudera Spark.** Consult appropriate third parties to obtain their distribution of XGBoost.
.. code-block:: scala
:caption: sbt

Installation from maven repo
============================
resolvers += "XGBoost4J Snapshot Repo" at "https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/"

Access release version
----------------------
Then add XGBoost4J as a dependency:

.. code-block:: xml
:caption: maven

<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j</artifactId>
<version>latest_version_num</version>
</dependency>
<properties>
...
<!-- Specify Scala version in package name -->
<scala.binary.version>2.12</scala.binary.version>
</properties>

<dependencies>
...
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j_${scala.binary.version}</artifactId>
<version>latest_version_num-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j-spark_${scala.binary.version}</artifactId>
<version>latest_version_num-SNAPSHOT</version>
</dependency>
</dependencies>

.. code-block:: scala
:caption: sbt

"ml.dmlc" % "xgboost4j" % "latest_version_num"
libraryDependencies ++= Seq(
"ml.dmlc" %% "xgboost4j" % "latest_version_num-SNAPSHOT",
"ml.dmlc" %% "xgboost4j-spark" % "latest_version_num-SNAPSHOT"
)

This will checkout the latest stable version from the Maven Central.
Look up the ``version`` field in `pom.xml <https://github.com/dmlc/xgboost/blob/master/jvm-packages/pom.xml>`_ to get the correct version number.

For the latest release version number, please check `here <https://github.com/dmlc/xgboost/releases>`_.
The SNAPSHOT JARs are hosted by the XGBoost project. Every commit in the ``master`` branch will automatically trigger generation of a new SNAPSHOT JAR. You can control how often Maven should upgrade your SNAPSHOT installation by specifying ``updatePolicy``. See `here <http://maven.apache.org/pom.html#Repositories>`_ for details.

if you want to use XGBoost4J-Spark, replace ``xgboost4j`` with ``xgboost4j-spark``.
You can browse the file listing of the Maven repository at https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/list.html.

Access SNAPSHOT version
-----------------------
.. note:: Windows not supported by published JARs

You need to add GitHub as repo:
The published JARs from the Maven Central and GitHub currently only supports Linux and MacOS. Windows users should consider building XGBoost4J / XGBoost4J-Spark from the source. Alternatively, checkout pre-built JARs from `criteo-forks/xgboost-jars <https://github.com/criteo-forks/xgboost-jars>`_.

.. code-block:: xml
:caption: maven
Installation from source
========================

<repository>
<id>GitHub Repo</id>
<name>GitHub Repo</name>
<url>https://raw.githubusercontent.com/CodingCat/xgboost/maven-repo/</url>
</repository>
Building XGBoost4J using Maven requires Maven 3 or newer, Java 7+ and CMake 3.3+ for compiling the JNI bindings.

.. code-block:: scala
:caption: sbt
Before you install XGBoost4J, you need to define environment variable ``JAVA_HOME`` as your JDK directory to ensure that your compiler can find ``jni.h`` correctly, since XGBoost4J relies on JNI to implement the interaction between the JVM and native libraries.

After your ``JAVA_HOME`` is defined correctly, it is as simple as run ``mvn package`` under jvm-packages directory to install XGBoost4J. You can also skip the tests by running ``mvn -DskipTests=true package``, if you are sure about the correctness of your local setup.

To publish the artifacts to your local maven repository, run

.. code-block:: bash

mvn install

resolvers += "GitHub Repo" at "https://raw.githubusercontent.com/CodingCat/xgboost/maven-repo/"
Or, if you would like to skip tests, run

.. code-block:: bash

Then add dependency as following:
mvn -DskipTests install

This command will publish the xgboost binaries, the compiled java classes as well as the java sources to your local repository. Then you can use XGBoost4J in your Java projects by including the following dependency in ``pom.xml``:

.. code-block:: xml
:caption: maven

<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j</artifactId>
<version>latest_version_num</version>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j</artifactId>
<version>latest_source_version_num</version>
</dependency>

For sbt, please add the repository and dependency in build.sbt as following:

.. code-block:: scala
:caption: sbt

"ml.dmlc" % "xgboost4j" % "latest_version_num"
resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"

For the latest release version number, please check `here <https://github.com/CodingCat/xgboost/tree/maven-repo/ml/dmlc/xgboost4j>`_.
"ml.dmlc" % "xgboost4j" % "latest_source_version_num"

.. note:: Windows not supported by published JARs
If you want to use XGBoost4J-Spark, replace ``xgboost4j`` with ``xgboost4j-spark``.

The published JARs from the Maven Central and GitHub currently only supports Linux and MacOS. Windows users should consider building XGBoost4J / XGBoost4J-Spark from the source. Alternatively, checkout pre-built JARs from `criteo-forks/xgboost-jars <https://github.com/criteo-forks/xgboost-jars>`_.
.. note:: XGBoost4J-Spark requires Apache Spark 2.3+

XGBoost4J-Spark now requires **Apache Spark 2.3+**. Latest versions of XGBoost4J-Spark uses facilities of `org.apache.spark.ml.param.shared` extensively to provide for a tight integration with Spark MLLIB framework, and these facilities are not fully available on earlier versions of Spark.

Also, make sure to install Spark directly from `Apache website <https://spark.apache.org/>`_. **Upstream XGBoost is not guaranteed to work with third-party distributions of Spark, such as Cloudera Spark.** Consult appropriate third parties to obtain their distribution of XGBoost.

Enabling OpenMP for Mac OS
--------------------------
Expand Down
34 changes: 1 addition & 33 deletions doc/jvm/xgboost4j_spark_tutorial.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,39 +27,7 @@ Build an ML Application with XGBoost4J-Spark
Refer to XGBoost4J-Spark Dependency
===================================

Before we go into the tour of how to use XGBoost4J-Spark, we would bring a brief introduction about how to build a machine learning application with XGBoost4J-Spark. The first thing you need to do is to refer to the dependency in Maven Central.

You can add the following dependency in your ``pom.xml``.

.. code-block:: xml

<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j-spark</artifactId>
<version>latest_version_num</version>
</dependency>

For the latest release version number, please check `here <https://github.com/dmlc/xgboost/releases>`_.

We also publish some functionalities which would be included in the coming release in the form of snapshot version. To access these functionalities, you can add dependency to the snapshot artifacts. We publish snapshot version in github-based repo, so you can add the following repo in ``pom.xml``:

.. code-block:: xml

<repository>
<id>XGBoost4J-Spark Snapshot Repo</id>
<name>XGBoost4J-Spark Snapshot Repo</name>
<url>https://raw.githubusercontent.com/CodingCat/xgboost/maven-repo/</url>
</repository>

and then refer to the snapshot dependency by adding:

.. code-block:: xml

<dependency>
<groupId>ml.dmlc</groupId>
<artifactId>xgboost4j-spark</artifactId>
<version>next_version_num-SNAPSHOT</version>
</dependency>
Before we go into the tour of how to use XGBoost4J-Spark, you should first consult :ref:`Installation from Maven repository <install_jvm_packages>` in order to add XGBoost4J-Spark as a dependency for your project. We provide both stable releases and snapshots.

.. note:: XGBoost4J-Spark requires Apache Spark 2.4+

Expand Down
30 changes: 30 additions & 0 deletions jvm-packages/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,29 @@
</plugins>
</build>
</profile>
<profile>
<id>release-to-s3</id>
<distributionManagement>
<snapshotRepository>
<id>maven-s3-snapshot-repo</id>
<url>s3://xgboost-maven-repo/snapshot</url>
</snapshotRepository>
<repository>
<id>maven-s3-release-repo</id>
<url>s3://xgboost-maven-repo/release</url>
</repository>
</distributionManagement>
<repositories>
<repository>
<id>maven-s3-snapshot-repo</id>
<url>https://s3.amazonaws.com/xgboost-maven-repo/snapshot</url>
</repository>
<repository>
<id>maven-s3-release-repo</id>
<url>https://s3.amazonaws.com/xgboost-maven-repo/release</url>
</repository>
</repositories>
</profile>
</profiles>
<distributionManagement>
<snapshotRepository>
Expand Down Expand Up @@ -324,6 +347,13 @@
</executions>
</plugin>
</plugins>
<extensions>
<extension>
<groupId>org.kuali.maven.wagons</groupId>
<artifactId>maven-s3-wagon</artifactId>
<version>1.2.1</version>
</extension>
</extensions>
</build>
<reporting>
<plugins>
Expand Down
Loading