[RFC] Benchmark scripts to detect performance regression

[hcho3]

One of the pain points I've observed in developing XGBoost is judging performance implication of a given code change. I believe that we should have a set of scripts that developers can easily run in order to obtain performance measurements. This will help us avoid introducing performance regressions (*). For now, developers will be asked to manually run the scripts, but I hope to obtain funding to run the benchmark as part of the CI.

In addition, we should have a set of tabular datasets that we'd use as reference. They would be "blessed" in that any proposed pull request should not introduce regression when running these datasets.

The scripts should collect the following:

• Model evaluation: accuracy, AUC, logloss etc
• Speed (run-time): local and distributed training
• Memory consumption

@dmlc/xgboost-committer Now the question: should the benchmark be hosted in this repository or a separate repository (https://github.com/dmlc/xgboost-bench/)? My opinion is that it should be hosted separately as the benchmark has no dependency on the internals of the XGBoost codebase. Also see @RAMitchell's comment in #5126 (comment).

(*) "Regression" here was used in the sense of "a trend or shift toward a lower or less perfect state" (Merriam-Webster).

[hcho3]

cc @SmirnovEgorRu @thvasilo @trams @rongou @sriramch

[terrytangyuan]

I agree that it should be hosted separately since the dependencies might be really heavy as the test suite expands in the future and it requires a bit of setup to work with distributed environments.

Although we could probably also maintain a minimal set of benchmark script in this repo if they are small enough that can be executed for each new commit so we can catch potential issues earlier during code reviews.

[hcho3]

@terrytangyuan I like your suggestion of having a small benchmark to run for every commit.

[RAMitchell]

I vote separate repository.

[trivialfis]

@terrytangyuan 's comment makes sense to me. Let's host it in bench repo then.

[hcho3]

Closing now. The benchmark script will be maintained in https://github.com/dmlc/xgboost-bench/