Initial support for multi-output regression.

[trivialfis]

• Implemented as 1 model per target.
• Not available for classification yet.

I will follow up with R and dask implementations if this PR is approved.

Related: #2087 .

Todos:

• Verify data types other than numpy array.
• Have better specification for return shape.

[trivialfis]

Hmm .. datatable is updated?

[Craigacp]

That's exciting. We'd be interested in a XGBoost4J wrapper too, and I might be able to write such a thing once you're happy with the native API. We currently fake this up by training a separate Booster for each output dimension.

[trivialfis]

Let me try to extend it to multi-class.

Update: Will implement it in a different PR. We need to extend the objective function and have more tests around the behavior of prediction. This will push XGBoost's tree grid into a cubic and needs some refactoring. But the interface established in this PR should be the same (num_target parameter).

[trivialfis]

@Craigacp Thanks for the support. I will keep you posted on the progress. I think it's best we wait until having a good solution for multi-class and other possible applications (like prob-forecasting). I will work on it in this release cycle.

[trivialfis]

Should I create a multi-output regressor and distinguish it from the normal regressor?

[trivialfis]

Need to have some utilities from #7331 first.

[trivialfis]

Cross linking #7083 . Need to resolve the same issue with matrix meta info.

[trivialfis]

TO-DO for initial support of multi-target model training.

• Initial setup for Python data structure handling. (Base margin support for multi-class classifier #7083).
• A typed tensor view that can be used in GPU kernels: Implement a general array view. #7365
• Typed tensor storage. (Implement typed storage for tensor. #7429)
• Type erased tensor for accepting user inputs. This is already available as the array interface class, we just need to extend it to a higher dimension. (Extend array interface to handle ndarray. #7434)
• Convert labels and base margin into matrix/tensor with proper shape attributes. ([Breaking] Accept multi-dim meta info. #7405, Convert labels into tensor. #7456)
• Infer the number of targets from input data.

Tests:

• Boost from prediction.
• Weighted metrics.
• Accuracy.

[trivialfis]

Most of the basic infrastructures are in place now. Need more thought on classification problem due to different number of classes per target.

[trivialfis]

The initial support is merged in #7514 .

[trivialfis]

@Craigacp I think for multi-target classification models, xgboost needs a new interface and potentially lots of refactoring. I will focus on regression for now. Thank you for joining the discussion and feel free to test the new feature. ;-)

[aniruddhghatpande]

@trivialfis Is this contain support for all multi output regression ? @Craigacp were you able to write the wrapper for XGBoost4J ?

[Craigacp]

I haven't looked into it. I've had less time to work on XGBoost related things for the past couple of years.