custom loss functions

[jeremyhermann]

We are assessing the viability of moving a number of our modeling projects to xgboost and wanted to understand the feasibility of supporting several new loss functions, using the existing plugin system and/or making modifications to xgboost source code.

Do you think we can get these to work?

• max entropy
• least squares
• least absolute deviation
• huber
• multi-bin (eg Multiple output regression #2087)

If these require changes to xgboost source, we'd be interested in contributing back.

[hcho3]

@jeremyhermann Currently, the Python binding supports custom loss function. The format will look like

def custom_loss(preds, dtrain): 
    ... do some gory computation ...
    return grad, hess

# at training time
bst = xgboost.train(params, dtrain, 10, eval_list, obj=custom_loss)

where grad is the list of first-order gradients and hess is the list of second-order gradients. If your loss function is twice-differentiable, great!

Things get tricky when your loss function is not twice differentiable. For now, you should look at #1825. Basically, what we need is to find g and h such that

[objective] <= [some constant] + g dy + 0.5 h dy^2       (1)

For twice-differentiable objectives, g and h are trivially found by taking first- and second-order gradients. (See slide 14 of this presentation.) But really any g and h that satisfy (1) will do. So if one can find suitable g and h for the huber loss, for instance, the huber loss can be used in XGBoost.

For next week or two, I plan to do some self-study on gradient boosting. I will get back to you if I have better idea about supporting non-smooth functions.

[adamwlev]

It would be really nice if huber loss was supporting natively in xgboost. Implementing it in python works but it significantly slows the learning process.

[pommedeterresautee]

@adamwlev you may use a pseudo huber loss https://en.wikipedia.org/wiki/Huber_loss which is differentiable.

[adamwlev]

@pommedeterresautee Thanks, I am actually using this and I like it. The only thing is that training is 10-15% slower since the objective is implemented in python-numpy not native to xgboost. Unfortunately, I am intimated by the code base of xgboost/don't have the to invest in learning enough to be able to make this contribution.