Testing hist_util

[RAMitchell]

This PR is for adding more tests for quantile generation in hist/gpu_hist. Part of the motivation is to create some absolute tests of accuracy for GPU sketching, instead of comparing against existing CPU algorithms.

So far I have tested the following expected behaviours with respect to a single feature:

• The histogram cut min_value should be less than all inputs
• The last histogram cut value should be greater than all inputs
• The rank of quantile cuts from the sketch should not exceed error ~0.01 as compared to rank from sorted values
• And input of k < num_bins unique values should output k unique bins. i.e. it should be possible to split over each unique value

I also expected that number of cuts (excluding the minimum value) should equal the number of bins requested if the unique input size is greater than num_bins. This is true for categorical features with <= unique values but not for continuous features. This currently means that if you ask for 256 bins you get 254 actual bins, which seems like a bug. Two of the samples get lost here (

xgboost/src/common/hist_util.h

Line 208 in 472ded5

for (size_t i = 2; i < summary.size; ++i) {

), one value gets added afterwards to be larger than everything. Another sample seems to get lost here (

xgboost/src/common/hist_util.cc

Line 319 in 472ded5

a.SetPrune(summary_array[fid], max_num_bins);

), where 256 samples go into SetPrune and 255 come out, despite max_num_bins being set at 256.

After resolving this I also want to add tests comparing the rank of the output quantiles against the correct rank by sorting the input.

I would also like to clarify the need for specialised cut points for categorical variables, implemented here (

xgboost/src/common/hist_util.h

Line 200 in 472ded5

/* specialized code categorial / ordinal data -- use midpoints */

) and discussed in #5095.

@hcho3 any comments would be much appreciated.

[RAMitchell]

I added tests for eps accuracy of sketch for combinations of number of bins and input sizes. I also fixed an off by one error where for categorical data the number of histogram bins would always be one less than the unique inputs, so certain categorical values were never used.

It still seems possible for summary.SetPrune() to unnecessarily remove cuts in some cases, but this is not a major issue and I'm not confident to change that function, so will leave that for another day.

I also did some experiments with specialised logic for categorical splits, I don't think they have any effect on accuracy, maybe someone else can provide an example of why this is necessary?

[trivialfis]

@hcho3 I vaguely remembered that you have some use cases for this specialization.

[RAMitchell]

I'm still working through a bunch of corner cases for this.

[trivialfis]

@RAMitchell My preference is either we don't support it, or provide full blown implementation.

[hcho3]

#5095 and #5096 may be relevant.

[RAMitchell]

The specialisation is unnecessary, we can use the same code to get correct results for both categorical and continuous. After this PR, if the number of unique feature values is less than or equal to max_bin, then each value will get its own bin and can be used for splitting. Before this PR, if the number of unique values was > 16 but less than max_bin, categorical features would get lost.

[RAMitchell]

So WXQSummary.SetPrune() seems to be quite broken. Sometimes it unnecessarily removes elements from a set before it reaches capacity and my tests show incorrect distributions of elements in its quantile ranges.

I replaced this with the simpler function WQSummary.SetPrune() for the hist and gpu_hist algorithms and everything is working as expected. I believe this function only gets called once towards the end of quantile calculation, so I am not expecting this to affect speed but will check with some benchmarks shortly.

[RAMitchell]

@rongou I noticed your sampling tests were quite volatile. The reason is because quantiles calculated on external memory end up slightly different compared to in memory.

The fix is to set max_bins equal to the number of training rows so that the quantiles end up the same for both external memory and in memory (each row has its own bin). Your tests comparing predictions from external memory/in memory now pass to a much higher accuracy.

[rongou]

@RAMitchell that's great! Thanks!

[RAMitchell]

I checked the performance of the different pruning methods to compute DenseCuts and there does not seem to be a meaningful difference. If anything WQSketch is faster.