[RFC] Inconsistent behavior when a single-leaf tree is encountered

[shiyu1994]

Description

With a single-leaf tree, CLI version stops training at once. But Python API continues to train.

Reproducible example

Example by @arnocandel in #4708.

Additional Comments

We have two choices:

1. Stopped training at once when a single leaf tree is encountered in all APIs.
2. Continue to train, add the same prediction value to all training samples. The prediction value should be
   - sum_of_gradients/sum_of_hessians in the root node, which is now not calculated.

Gently ping @guolinke @jameslamb @StrikerRUS @hzy46 @btrotta for your opinion.

[guolinke]

I think option 2 is better.

[shiyu1994]

I think option 2 is better.

Strongly agree.

[jameslamb]

Thanks for writing this up and for the description in #4708 (comment)!

I think option 2 is preferable.

Encountering a single-leaf tree doesn't necessarily mean that training should stop. I'd expect future boosting rounds could still find informative splits in some situations, like:

• using bagging_fraction to re-sample rows
• using feature_fraction to randomly choose features
• using a custom objective that has some sort of randomness in it or some behavior dependent on the number of iterations

[shiyu1994]

@jameslamb Thanks. Then let's go to option 2.