Support for multiple custom eval metrics

[candalfigomoro]

Hi,

it seems like LightGBM does not currently support multiple custom eval metrics.

E.g. f1-score, precision and recall are not available as eval metrics. I can add them as custom eval metrics, but I can't use all of them at the same time. Currently, it seems like LightGBM only supports 1 custom metric at a time.

LightGBM version: 2.2.3

[StrikerRUS]

Hi @candalfigomoro !
It does. Please see examples:

LightGBM/examples/python-guide/advanced_example.py

Lines 175 to 185 in a7885b6

	gbm = lgb.train(params,
	lgb_train,
	num_boost_round=10,
	init_model=gbm,
	fobj=loglikelihood,
	feval=lambda preds, train_data: [binary_error(preds, train_data),
	accuracy(preds, train_data)],
	valid_sets=lgb_eval)
	print('Finished 50 - 60 rounds with self-defined objective function '
	'and multiple self-defined eval metrics...')

LightGBM/examples/python-guide/sklearn_example.py

Lines 62 to 67 in a7885b6

	print('Starting training with multiple custom eval functions...')
	# train
	gbm.fit(X_train, y_train,
	eval_set=[(X_test, y_test)],
	eval_metric=lambda y_true, y_pred: [rmsle(y_true, y_pred), rae(y_true, y_pred)],
	early_stopping_rounds=5)

[candalfigomoro]

@StrikerRUS
Thank you. Can we mix custom and built-in metrics (in the sklearn interface)?
Maybe I want to use AUC (built-in) + F1 (custom)

[StrikerRUS]

@candalfigomoro Sure! For built-in metrics use metrics param and eval_metric for custom ones.

import numpy as np
import lightgbm as lgb

from sklearn.datasets import load_boston

def rae(y_true, y_pred):
    return 'RAE', np.sum(np.abs(y_pred - y_true)) / np.sum(np.abs(np.mean(y_true) - y_true)), False

def rmsle(y_true, y_pred):
    return 'RMSLE', np.sqrt(np.mean(np.power(np.log1p(y_pred) - np.log1p(y_true), 2))), False

X, y = load_boston(True)
clf = lgb.LGBMRegressor(n_estimators=10, metric=['AUC', 'MSE'])

clf.fit(X, y, eval_set=[(X, y)], eval_metric=lambda y_true, y_pred: [rmsle(y_true, y_pred), rae(y_true, y_pred)])

[1]	training's auc: 1	training's l2: 71.212	training's RMSLE: 0.363416	training's RAE: 0.915593
[2]	training's auc: 1	training's l2: 60.4348	training's RMSLE: 0.337014	training's RAE: 0.841258
[3]	training's auc: 1	training's l2: 51.6226	training's RMSLE: 0.313682	training's RAE: 0.774098
[4]	training's auc: 1	training's l2: 44.3804	training's RMSLE: 0.292515	training's RAE: 0.716202
[5]	training's auc: 1	training's l2: 38.3112	training's RMSLE: 0.273322	training's RAE: 0.663217
[6]	training's auc: 1	training's l2: 33.311	training's RMSLE: 0.255968	training's RAE: 0.615143
[7]	training's auc: 1	training's l2: 29.319	training's RMSLE: 0.240791	training's RAE: 0.573687
[8]	training's auc: 1	training's l2: 25.8508	training's RMSLE: 0.226686	training's RAE: 0.536238
[9]	training's auc: 1	training's l2: 22.9094	training's RMSLE: 0.213802	training's RAE: 0.502092
[10]	training's auc: 1	training's l2: 20.5088	training's RMSLE: 0.20243	training's RAE: 0.471546

[dreamflyer]

Why not just simply:
clf.fit(X, y, eval_set=[(X, y)], eval_metric=["metric1", "metric2", callable1, callable2])
?

and why y_true and y_pred in custom metrics are in different format? It is so non-intuitive.

Furthermore, why input label vectors should be converted into number of classes for multiclass classification?

Isn't such stuff should be fixed?