Benchmarks to track performance changes in 'hist' method

[SmirnovEgorRu]

It is PR №2 from the issue #5104.
It is required to understand impact of the optimizations. I'm planning to use them for all further PRs.

[trivialfis]

@RAMitchell We staged many benchmarking scripts in external projects. I also have a collection of them with dask. I'm open to have some of those XGBoost specific scripts to be maintained in one place. WDYT?

[SmirnovEgorRu]

Time by kernels after optimizations reverting collected by the benchmarks:

Data set	ApplySplit	EvaluateSplit	BuildHist	SyncHistogram	Prediction	Total, sec
higgs1m	36	62	156	186	3	446
airline-ohe	30	46	77	126	2	303
msrank-10k	162	244	1366	836	50	2680

Time by kernels before optimizations reverting:

Data set	ApplySplit	EvaluateSplit	BuildHist	SyncHistogram	Prediction	Total, sec
higgs1m	3.7	3.5	6.2	0	1.6	17.7
airline-ohe	9.0	6.1	28.8	0	0.7	64
msrank-10k	15.5	52.6	66.6	0	47.6	197

HW: c5.metal AWS instance

[RAMitchell]

As this becomes more sophisticated it begs the question, should this code be inside the xgboost main repo? It has no dependency on xgboost source code, only on having some installed version of xgboost. We can just as easily run it via our CI as a separate repo.

Also how is this different from https://github.com/NVIDIA/gbm-bench? Would you get the information you need by running this? Maybe we need a more neutrally hosted version of gbm-bench.

[RAMitchell]

Also, one of the problems with previous optimisations was that they caused performance regression in the distributed algorithm due to increasing the number of rabit calls. To resolve this we could run experiments with dask.

[trivialfis]

@hcho3

[hcho3]

@RAMitchell We can probably combine NVIDIA/gbm-bench and this pull request. For now, let us just benchmark XGBoost and not worry about other libraries (LightGBM, CatBoost etc). And as you mentioned, we should definitely test distributed training.

@tqchen Can I have admin right over https://github.com/dmlc/xgboost-bench ? This seems perfect for hosting benchmark script.

[hcho3]

@dmlc/xgboost-committer https://github.com/dmlc/xgboost-bench is now public. All committers of XGBoost should have push access to it.

[hcho3]

Closing this PR now. I will move this PR code to https://github.com/dmlc/xgboost-bench

[hcho3]

@SmirnovEgorRu I moved your benchmark code to xgboost-bench repo: dmlc/xgboost-bench@c787a59