Same param settings, differ in the length of eval_set, get different training result

[dtss316]

Sample1:
input_param = {'booster': 'dart', 'colsample_bylevel': 0.4, 'colsample_bytree': 0.3, 'gamma': 0.9000000000000001, 'learning_rate': 0.1, 'max_delta_step': 2, 'max_depth': 2, 'min_child_weight': 2, 'n_estimators': 480, 'reg_alpha': 10, 'reg_lambda': 1, 'scale_pos_weight': 0.9, 'subsample': 0.6000000000000001}
model = xgbboost.sklearn.XGBClassifier(n_jobs=5, verbosity=0, **input_param)
model.fit(x_train, y_train, early_stopping_rounds=24, eval_metric=['auc'], eval_set=[(x_train, y_train), (x_train, y_train), (x_val, y_val)])
Sample2：
input_param = {'booster': 'dart', 'colsample_bylevel': 0.4, 'colsample_bytree': 0.3, 'gamma': 0.9000000000000001, 'learning_rate': 0.1, 'max_delta_step': 2, 'max_depth': 2, 'min_child_weight': 2, 'n_estimators': 480, 'reg_alpha': 10, 'reg_lambda': 1, 'scale_pos_weight': 0.9, 'subsample': 0.6000000000000001}
model = xgbboost.sklearn.XGBClassifier(n_jobs=5, verbosity=0, **input_param)
model.fit(x_train, y_train, early_stopping_rounds=24, eval_metric=['auc'], eval_set=[(x_train, y_train), (x_val, y_val)])

The only difference between sample1 and sample2 is eval_set.

[hcho3]

Early stopping may be broken. Try disabling early stopping and see if the results are consistent. If so, we will need to fix early stopping. Can you post your script and data so that we can try to fix it?

[hcho3]

@dtss316 Any updates?

[hcho3]

See #4638, which addresses a bug in early stopping. I'm going to close this for now. Should you experience the same issue after #4638 is merged, feel free to re-open.