[Breaking][jvm-packages] add barrier execution mode #7836
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With introducing barrier execution mode. we don't need to kill SparkContext when some xgboost tasks failed, instead, Spark will do it for us. So in this PR, killSparkContextOnWorkerFailure parameter is deleted.
This was referenced Apr 24, 2022
|
@trivialfis could you help to review it? |
trivialfis
reviewed
Apr 25, 2022
There was a problem hiding this comment.
Thank you for working on this, being able to remove code and improve robustness is exciting!
- Could you please share more details on what's the breaking change and how users should adapt to it?
- Could you please keep some RABIT mock tests here to show that exceptions are indeed being well handled?
Comment on lines
375
to
386
| test("throw exception for empty partition in trainingset") { | ||
| val paramMap = Map("eta" -> "0.1", "max_depth" -> "6", "silent" -> "1", | ||
| "objective" -> "multi:softmax", "num_class" -> "2", "num_round" -> 5, | ||
| "num_workers" -> numWorkers, "tree_method" -> "auto") | ||
| "objective" -> "binary:logistic", "num_class" -> "2", "num_round" -> 5, | ||
| "num_workers" -> numWorkers, "tree_method" -> "auto", "allow_non_zero_for_missing" -> true) | ||
| // The Dmatrix will be empty | ||
| val trainingDF = buildDataFrame(Seq(XGBLabeledPoint(1.0f, 1, Array(), Array()))) | ||
| val trainingDF = buildDataFrame(Seq(XGBLabeledPoint(1.0f, 4, | ||
| Array(0, 1, 2, 3), Array(0, 1, 2, 3)))) | ||
| val xgb = new XGBoostClassifier(paramMap) | ||
| intercept[XGBoostError] { | ||
| val model = xgb.fit(trainingDF) | ||
| intercept[SparkException] { | ||
| xgb.fit(trainingDF) | ||
| } | ||
| } |
There was a problem hiding this comment.
Are these changes strictly related to the support of barrier mode?
There was a problem hiding this comment.
Completely not. I just make this test more like the test description.
killSparkContextOnWorkerFailure parameter is deleted. Users are supposed not to use this parameter.
Not exactly. the deleted case test("test SparkContext should not be killed ") is only for |
trivialfis
approved these changes
Apr 25, 2022
This was referenced Apr 25, 2022
trivialfis
pushed a commit
to trivialfis/xgboost
that referenced
this pull request
Apr 29, 2022
With the introduction of the barrier execution mode. we don't need to kill SparkContext when some xgboost tasks failed. Instead, Spark will handle the errors for us. So in this PR, `killSparkContextOnWorkerFailure` parameter is deleted.
trivialfis
added a commit
that referenced
this pull request
Apr 29, 2022
* [jvm-packages] move the dmatrix building into rabit context (#7823) This fixes the QuantileDeviceDMatrix in distributed environment. * [doc] update the jvm tutorial to 1.6.1 [skip ci] (#7834) * [Breaking][jvm-packages] Use barrier execution mode (#7836) With the introduction of the barrier execution mode. we don't need to kill SparkContext when some xgboost tasks failed. Instead, Spark will handle the errors for us. So in this PR, `killSparkContextOnWorkerFailure` parameter is deleted. * [doc] remove the doc about killing SparkContext [skip ci] (#7840) * [jvm-package] remove the coalesce in barrier mode (#7846) * [jvm-packages] Fix model compatibility (#7845) * Ignore all Java exceptions when looking for Linux musl support (#7844) Co-authored-by: Bobby Wang <wbo4958@gmail.com> Co-authored-by: Michael Allman <msa@allman.ms>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To fix #7835. Credit to @WeichenXu123
By introducing the barrier execution mode, we don't need to kill SparkContext when some xgboost tasks failed, instead, Spark will abort the whole barrier stage which does not depend on SparkListener, we will never encounter the xgboost task hang issue. So in this PR, the killSparkContextOnWorkerFailure parameter is deleted.
Since this PR has deleted Spark SparkParallelismTracker, the timeoutRequestWorkers parameter is not needed anymore. But one test
cross-version model loading (0.82)will fail if we just delete timeoutRequestWorkers, we just keep this parameter in this PR. Will file a following up PR to delete timeoutRequestWorkers parameters.