[WIP][PySpark] Add XGBoost PySpark API support

[wbo4958]

This PR is to close #7578.

The XGBoost PySpark API is a wrapper of xgboost4j-spark and xgboost4j-spark-gpu. It will be packaged into the existing xgboost python-package.

[candalfigomoro]

@wbo4958 Is there a way to set the Scala tracker instead of the Python tracker?

In Java/Scala I set:

xgbClassifier.set("trackerConf", new TrackerConf(0, "scala"));

[wbo4958]

@wbo4958 Is there a way to set the Scala tracker instead of the Python tracker?

In Java/Scala I set:

xgbClassifier.set("trackerConf", new TrackerConf(0, "scala"));

Yeah, looks like we need to add it support in scala.

[candalfigomoro]

This is a typical snippet I can have in Java:

        XGBoostClassifier xgbClassifier = new XGBoostClassifier();
        xgbClassifier.set("trackerConf", new TrackerConf(0, "scala"));
        xgbClassifier.setNumWorkers(8);
        xgbClassifier.setNthread(1);
        xgbClassifier.setEta(0.1);
        xgbClassifier.setMaxDepth(6);
        xgbClassifier.setNumRound(1000);
        xgbClassifier.setNumEarlyStoppingRounds(10);
        xgbClassifier.setObjective("binary:logistic");
        xgbClassifier.setScalePosWeight(1.0);
        xgbClassifier.setSubsample(0.5);
        xgbClassifier.setColsampleBytree(0.5);
        xgbClassifier.setTreeMethod("hist");
        xgbClassifier.setFeaturesCol("features");
        xgbClassifier.setLabelCol("label");
        xgbClassifier.setEvalMetric("auc");
        xgbClassifier.setMaximizeEvaluationMetrics(true);  // true for "auc"
        xgbClassifier.setEvalSets(new Map.Map1<>("validation", validation));

It looks like many parameters are missing (is there a way to set eval sets for early stopping?)

[wbo4958]

@candalfigomoro Yeah, I have not ported all the scala params yet, for now, I am focusing on adding integration tests. we can have following PRs to add more params.

[trivialfis]

Thank you for revising the original PR! This looks much more aligned with Python and Pyspark ml.

Could you please write a tutorial in doc/tutorials/ for newbies like me to get started?

[trivialfis]

I think it's more appropriate to use xgboost.pyspark.PySparkXGBClassifier instead of this java style module.

[wbo4958]

From the convention of PySpark, we should keep the same API name for PySpark and JVM.
See https://github.com/apache/spark/blob/branch-3.2/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala#L58 and https://github.com/apache/spark/blob/branch-3.2/python/pyspark/ml/classification.py#L1952

[trivialfis]

That's not the pyspark convention. It's not pyspark.ml.apache.org.xxxx.

[trivialfis]

So, what's the specific case that we should use this instead of with cpu session? Could you please document it?

[wbo4958]

the API was copied from spark-rapids, and now I've changed it according to need of xgboost.

[trivialfis]

@wbo4958 Okay .... but I'm not sure how's that relevant to the question?

[trivialfis]

Is this necessary?

[trivialfis]

I'm not entirely sure what's the result of combining this decorator with a custom doc string. Have you checked?

[trivialfis]

Please double-check copied code.