Rewrite approx #7214

trivialfis · 2021-09-08T07:22:31Z

Based on #7183

This PR rewrites the approx tree method to use codebase from hist for better performance and prepare for categorical data support.

The rewrite has many benefits:

Support for both max_leaves and max_depth.
Support for grow_policy.
Support for mono constraint.
Support for feature weights.
Support for easier bin configuration (max_bin).
Support for categorical data.
Faster performance for most of the datasets. (many times faster)
Support for prediction cache.
Significantly better performance for external memory.
Unites the code base between approx and hist.

TODOs:

Figure out how to handle MatchThreadsToNodes.
Handle different sizes of batches.
Handle empty partition gracefully.
Add tests.
Distributed training.
Tests for build policies and related parameters (max_leave, max_depth).
Fix task initialization.

Close #7244 .

codecov-commenter · 2021-09-10T05:17:16Z

Codecov Report

Merging #7214 (8f1882b) into master (d434942) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #7214   +/-   ##
=======================================
  Coverage   83.71%   83.71%           
=======================================
  Files          13       13           
  Lines        3892     3892           
=======================================
  Hits         3258     3258           
  Misses        634      634

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d434942...8f1882b. Read the comment docs.

trivialfis · 2021-09-10T11:53:25Z

The PR is pretty much ready other than some cleanups. I won't merge it until 1.5 is made.

trivialfis · 2021-09-12T11:35:45Z

@ShvetsKS I did some quick tests with external memory:

The usability of external memory:

Current approx on the master branch: Unusable. It iterates the data for every node. Running for 40 minutes now with 40 GB of data, still not reaching the finish line of the first iteration.
Rewritten approx: Better, the bottleneck is in regenerating the histogram index page.
Hist: Not implemented yet. But I think that's the best place to have external memory since it needs only reading the histogram index.
GPU Hist: I have tested a prototype implementation before but copying from disk to cpu then to gpu seems to be too much overhead.

I have 64 GB of memory, so the kernel has done lots of caching for the data. Are you interested in implementing it for Hist? This PR has prepared most of the foundational work.

trivialfis · 2021-09-14T05:38:09Z

@RAMitchell @hcho3 @ShvetsKS Could you please give a high-level review on this rewrite? I will split up the PR for more detailed reviews, just want to know your thoughts about the idea of rewriting.

trivialfis · 2021-09-17T17:49:58Z

Close #7244 .

src/common/hist_util.cc

src/common/partition_builder.h

src/common/threading_utils.h

src/gbm/gbtree.cc

src/common/hist_util.cc

trivialfis · 2021-10-27T12:35:14Z

Categorical data support is implemented here.

src/common/hist_util.cc

trivialfis · 2021-10-27T14:10:58Z

The perf change for hist:

	Master	PR
Airline	1367.8661037589918	1365.7433312300127
Bosch	96.10689159599133	100.5928801370028
Covtype	72.04649400700873	71.20271111401962
Higgs	173.99613595800474	174.35436162599945
Year	27.867479905995424	30.441671705979388

trivialfis · 2021-11-09T17:20:42Z

Since all parameters are supported, we can start to drop the experimental tag on parameter validation once this PR is fully merged.

tests/cpp/common/test_span.cc

trivialfis · 2021-12-10T10:44:26Z

max_bin: 256
sketch_eps: 0.0078125

	AUC	AUC (rewrite)	Accuracy	Accuracy (rewrite)	MAE	MAE (rewrite)	Time	Time (rewrite)
airline	0.8083021582698621	0.8083021582679778	0.6653586256318165	0.6653586256318165			7452.502085362998	1946.756329375
bosch	0.7032129451430552	0.7064355238462672	0.8514382259767688	0.8494741288278775			249.46239526499994	183.57548371099983
covtype			0.8589967556775643	0.8574993760918392			155.14399087498896	85.87701512000058
epsilon	0.9258295171219089	0.9259069769846513	0.82366	0.82496			667.7028755839856	2270.314828159986
higgs	0.8224143371680539	0.8223955953797959	0.7132495454545454	0.7133218181818182			1277.4466711200075	347.96909062200575
year					6.341546058654785	6.340557098388672	35.443501809000736	39.575479084975086

One special case is epsilon, which is significantly slower due to the number of features of this dataset. For the accuracy they are pretty much the same, I tuned the sketching algorithm to match the result of the original implementation but then removed the tuning to avoid specialization.

At some point we might want to rewrite the CPU sketching algorithm, but I don't want to go down that rabbit hole at this point.

Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Regenerate the index. ama. Clang tidy. Retain page. Fix. Lint. Tidy. Integer backed enum. Convert to uint32_t. Prototype for saving gidx. Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Initial port. Pass in hessian. Init column sampler. Unused code. Use ctx. Merge sampling. Use ctx in partition. Fix init root. Force regenerate the sketch. Create a ctx. Get it compile. Don't use const method. Use page id. Pass in base row id. Pass the cut instead. Small fixes. Debug. Fix bin size. Debug. Fixes. Debug. Fix empty partition. Remove comment. Lint. Fix tests compilation. Remove check. Merge some fixes. fix. Fix fetching. lint. Extract expand entry. Lint. Fix unittests. Fix windows build. Fix comparison. Make const. Note. const. Fix reduce hist. Fix sparse data. Avoid implicit conversion. private. mem leak. Remove skip initialization. Use maximum space. demo. lint. File link tags. ama. Fix redefinition. Fix ranking. use npy. Comment. Tune it down. Specify the tree method. Get rid of the duplicated partitioner. Allocate task. Tests. make batches. Log. Remove span. Revert "make batches." This reverts commit 33f7072. small cleanup. Lint. Revert demo. Better make batches. Demo. Test for grow policy. Test feature weights. small cleanup. Remove iterator in evaluation. Fix dask test. Pass n_threads. Start implementation for categorical data. Fix. Add apply split. Enumerate splits. Enable sklearn. Works. d_step. update. Pass feature types into index. Search cut. Add test. As cat. Fix cut. Extract some tests. Fix. Interesting case. Add Python tests. Cleanup. Revert "Interesting case." This reverts commit 6bbaac2. Bin. Fix. Dispatch. Remove subtraction trick. Lint Use multiple buffers. Revert "Use multiple buffers." This reverts commit 2849f57. Test for external memory. Format. Partition based categorical split. Remove debug code. Fix. Lint. Fix test. Fix demo. Fix. Add test. Remove use of omp func. name. Fix. test. Make LCG impl compliant to std. Fix test. Constexpr. Use unsigned type. osx More test. Rebase error. Rebase error. Rebase error. Reverse unused changes. Config. Remove weird set thread. External memory test. Revert changes. Cleanup. wording. Fix doc. Test monotone constraint. Extract test for gamma. typo. Safe guard. Cleanup && comments. Update Python documents. Add push col page. hack. Port the sketch. Opt search bin. Cleanup. Reduce the gap. Fix sum hessian. Start cleaning up. Duplicated. Cleanup. lint. Test. Port the changes. test. Port the changes. Fixes && cleanup. Decide whether should sorted sketch be used. tests. Use regen. Lint. Revert. init. empty dataset. Handle empty dataset directly in quantile. empty. Update tests. Fix approx test. Revert "Fix approx test." This reverts commit d690afb.

RAMitchell

Awesome work. This will make life a lot easier.

trivialfis · 2022-01-10T13:19:43Z

@RAMitchell @hcho3 Thanks for all the reviews, I'm so excited to have this merged.

trivialfis mentioned this pull request Sep 12, 2021

Initial support for external memory in gradient index. #7183

Merged

trivialfis force-pushed the rewrite-approx branch from 9bbbaa2 to 0012969 Compare September 13, 2021 04:48

trivialfis mentioned this pull request Sep 14, 2021

Refactor split evaluator using device-wide Scan primitive #7197

Open

This was referenced Sep 24, 2021

Rabit Poll Timeout Error Flakiness when using Approx Tree Method #7250

Closed

[jvm-packages] training multi xgboost models using ThreadPool occurs error #7234

Open

This was referenced Oct 10, 2021

Add tests for tree grow policy. #7302

Merged

Dask arrays without distributed? #6652

Closed

trivialfis force-pushed the rewrite-approx branch from 2705c57 to 29c32fa Compare October 14, 2021 05:43

This was referenced Oct 14, 2021

Improve external memory demo. #7320

Merged

Extract CPUExpandEntry and HistParam. #7321

Merged

trivialfis force-pushed the rewrite-approx branch from 29c32fa to a583d51 Compare October 18, 2021 08:51

trivialfis mentioned this pull request Oct 18, 2021

Accept histogram cut instead of gradient index in evaluation. #7336

Merged

trivialfis commented Oct 18, 2021

View reviewed changes

trivialfis force-pushed the rewrite-approx branch from a583d51 to 068dd79 Compare October 20, 2021 10:55

trivialfis mentioned this pull request Oct 21, 2021

External memory support. #7354

Closed

8 tasks

trivialfis force-pushed the rewrite-approx branch from 068dd79 to 7140ba6 Compare October 27, 2021 06:06

trivialfis mentioned this pull request Oct 27, 2021

Categorical data support. #6503

Closed

67 tasks

trivialfis commented Oct 27, 2021

View reviewed changes

src/common/hist_util.cc Outdated Show resolved Hide resolved

src/common/hist_util.cc Outdated Show resolved Hide resolved

trivialfis commented Oct 27, 2021

View reviewed changes

src/common/hist_util.cc Outdated Show resolved Hide resolved

This was referenced Oct 27, 2021

Support building gradient index with cat data. #7371

Merged

Support external memory in CPU histogram building. #7372

Merged

[WIP] Implement categorical data support in CPU evaluation. #7374

Closed

trivialfis force-pushed the rewrite-approx branch 2 times, most recently from c56cbab to 1d7a620 Compare November 3, 2021 18:14

trivialfis mentioned this pull request Nov 3, 2021

CPU evaluation for cat data. #7393

Merged

trivialfis force-pushed the rewrite-approx branch from 1d7a620 to aed9d27 Compare November 7, 2021 18:31

trivialfis force-pushed the rewrite-approx branch from 58f62e8 to 9b0dde0 Compare November 22, 2021 17:51

This was referenced Nov 22, 2021

Test CPU histogram with cat data. #7465

Merged

Add approx partitioner. #7467

Merged

trivialfis force-pushed the rewrite-approx branch from b03ccfb to dc6359e Compare November 27, 2021 15:27

trivialfis commented Dec 5, 2021

View reviewed changes

tests/cpp/common/test_span.cc Outdated Show resolved Hide resolved

trivialfis changed the title ~~[WIP] Rewrite approx~~ Rewrite approx Dec 5, 2021

trivialfis marked this pull request as ready for review December 6, 2021 00:34

trivialfis force-pushed the rewrite-approx branch from 384fdd6 to 555acf5 Compare December 8, 2021 18:47

trivialfis mentioned this pull request Dec 10, 2021

Extract Sketch Entry from hist maker. #7503

Merged

trivialfis force-pushed the rewrite-approx branch from 49c4ce0 to dca1572 Compare December 17, 2021 21:45

trivialfis mentioned this pull request Dec 23, 2021

External memory support for hist #7531

Merged

trivialfis force-pushed the rewrite-approx branch from dca1572 to f4b8f31 Compare December 23, 2021 07:35

trivialfis added 2 commits January 6, 2022 17:11

Revert changes.

865b04e

trivialfis force-pushed the rewrite-approx branch from bac7485 to 865b04e Compare January 6, 2022 09:34

RAMitchell approved these changes Jan 10, 2022

View reviewed changes

trivialfis merged commit 0015031 into dmlc:master Jan 10, 2022

trivialfis deleted the rewrite-approx branch January 10, 2022 13:15

trivialfis mentioned this pull request Jul 14, 2022

Performance Degradation When Upgrading to v1.60 #8063

Open

eugeneyarovoi mentioned this pull request Jan 30, 2023

Troubleshooting XGBoost model performance jpmml/jpmml-sparkml#128

Closed

trivialfis mentioned this pull request Mar 16, 2023

Accuracy degradation with approx and noisy data. #8901

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite approx #7214

Rewrite approx #7214

trivialfis commented Sep 8, 2021 •

edited

Loading

codecov-commenter commented Sep 10, 2021 •

edited

Loading

trivialfis commented Sep 10, 2021

trivialfis commented Sep 12, 2021

trivialfis commented Sep 14, 2021

trivialfis commented Sep 17, 2021

trivialfis commented Oct 27, 2021

trivialfis commented Oct 27, 2021

trivialfis commented Nov 9, 2021

trivialfis commented Dec 10, 2021 •

edited

Loading

RAMitchell left a comment

trivialfis commented Jan 10, 2022 •

edited

Loading

Rewrite approx #7214

Rewrite approx #7214

Conversation

trivialfis commented Sep 8, 2021 • edited Loading

codecov-commenter commented Sep 10, 2021 • edited Loading

Codecov Report

trivialfis commented Sep 10, 2021

trivialfis commented Sep 12, 2021

trivialfis commented Sep 14, 2021

trivialfis commented Sep 17, 2021

trivialfis commented Oct 27, 2021

trivialfis commented Oct 27, 2021

trivialfis commented Nov 9, 2021

trivialfis commented Dec 10, 2021 • edited Loading

RAMitchell left a comment

Choose a reason for hiding this comment

trivialfis commented Jan 10, 2022 • edited Loading

trivialfis commented Sep 8, 2021 •

edited

Loading

codecov-commenter commented Sep 10, 2021 •

edited

Loading

trivialfis commented Dec 10, 2021 •

edited

Loading

trivialfis commented Jan 10, 2022 •

edited

Loading