custom evaluation function is not used if name is lexicographically smaller than default

[JaakobKind]

Dear xgboost developers,
I have found the following issue.
When I try to use a custom evaluation function, it is only working when I either use parameter disable_default_eval_metric or choose a name for my evaluation metric that is lexicographically larger than the name of the default metric.
If you know these workarounds, they are acceptable, but it took me some time to find out how to overcome the issue. Hence, it would be nice if this bug could be fixed.
I have attached a jupyter notebook that can be used to reproduce the bug. I needed to zip it to be able to upload it -:)

xgboost_bug.zip

[JaakobKind]

Additional info: I use xgboost 0.82.
I checked whether I get the same issue in R and was not able to reproduce it there. Hence, it seems only to occur in the python version.

[trivialfis]

@JaakobKind Sorry for late reply, will look into this.

[trivialfis]

@hcho3 Can I borrow some help here? What's the convention for early stopping when multiple evaluation metrics are used? Monitor one of them or all of them?

[hcho3]

@trivialfis The convention is to always use the first metric for early stopping

[JaakobKind]

What is the meaning of "first" here? I hope that it is not lexicographic order of the metric name.

[hcho3]

@JaakobKind It turns out that there was a bug that caused the metrics to be ordered in lexicographical order. #4638 recovers the originally intended semantics, where the last validation set and last evaluation metric are used for early stopping. (Here "last" means the exact order the user specifies the list of validation sets and metrics.)

Also, #4638 clarifies how early stopping works when multiple validation sets and metrics are given.
Preview: https://hcho3-xgboost.readthedocs.io/en/fix_early_stopping_python/python/python_api.html#xgboost.train

@trivialfis Sorry I was mistaken earlier. It was actually the last metric that should be used for early stopping. I clarified the exact behavior of early stopping in the Python API doc (#4638).

[hcho3]

@JaakobKind And if a customized metric function is added, it will always be added after built-in metrics, so early stopping will always use the customized metric.