first_metric_only only apply to default metric?

[rightx2]

Environment info

C++/Python/R version: Python 3.7

LightGBM version or commit hash: 2.3.0

Error message

I'm using LGBMRegressor and set metric to None and pass custom metrics to eval_metric of fit() (I refered to #2182):

model.fit(
    ...
    ...
    eval_metric=lambda y_true, y_pred: [my_custom_metric(y_true, y_pred)]
    ...
)

But I don't want to use DEFAULT metric(l2) for early_stopping but first_metric_only=True flag is ALWAYS applied to this default metric!

I'd like to use my_custom_metric instead of default one for early_stopping.

Moreover, I don't want to track default metric, l2.

How can I do that?

Reproducible examples

model = lightgbm.LGBMRegressor(
    n_estimators=best_epoch,
    objective="mse",
    metric=None,
    first_metric_only=True,     
)

model = model.fit(
    _train_x, _train_y,
    eval_metric=lambda y_true, y_pred: [my_custom_metric(y_true, y_pred)]
    eval_set=[
        (_train_x, _train_y,),
        (_valid_x, _valid_y,),
    ],
    verbose=1,
    early_stopping_rounds=100,
)

[StrikerRUS]

You should set metric to "None", not None.

"None" (string, not a None value) means that no metric will be registered, aliases: na, null, custom
https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters

from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

import lightgbm as lgb


def constant_metric(y_true, y_pred):
    return 'error', 0, False

def mse(y_true, y_pred):
    return 'custom MSE', mean_squared_error(y_true, y_pred), False


X, y = load_boston(True)
_train_x, _valid_x, _train_y, _valid_y = train_test_split(X, y, test_size=0.2)

model = lgb.LGBMRegressor(
    n_estimators=10,
    objective="mse",
    metric='None',
    first_metric_only=True 
)

model.fit(
    _train_x, _train_y,
    eval_metric=lambda y_true, y_pred: [mse(y_true, y_pred), constant_metric(y_true, y_pred)],
    eval_set=[
        (_train_x, _train_y,),
        (_valid_x, _valid_y,),
    ],
    verbose=1,
    early_stopping_rounds=3,
)