External memory support for hist #7531

trivialfis · 2021-12-23T01:35:05Z

Generate column matrix from gHistIndex.
Avoid synchronization with the sparse page once the cache is written.
Cleanups: Remove member variables/functions, change the update routine to look like approx and gpu_hist.
Remove pruner.

trivialfis · 2021-12-23T05:47:17Z

Note to myself: hist seems to be doing some unnecessary evaluation.

Rewrite approx. Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Regenerate the index. ama. Clang tidy. Retain page. Fix. Lint. Tidy. Integer backed enum. Convert to uint32_t. Prototype for saving gidx. Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Initial port. Pass in hessian. Init column sampler. Unused code. Use ctx. Merge sampling. Use ctx in partition. Fix init root. Force regenerate the sketch. Create a ctx. Get it compile. Don't use const method. Use page id. Pass in base row id. Pass the cut instead. Small fixes. Debug. Fix bin size. Debug. Fixes. Debug. Fix empty partition. Remove comment. Lint. Fix tests compilation. Remove check. Merge some fixes. fix. Fix fetching. lint. Extract expand entry. Lint. Fix unittests. Fix windows build. Fix comparison. Make const. Note. const. Fix reduce hist. Fix sparse data. Avoid implicit conversion. private. mem leak. Remove skip initialization. Use maximum space. demo. lint. File link tags. ama. Fix redefinition. Fix ranking. use npy. Comment. Tune it down. Specify the tree method. Get rid of the duplicated partitioner. Allocate task. Tests. make batches. Log. Remove span. Revert "make batches." This reverts commit 33f7072. small cleanup. Lint. Revert demo. Better make batches. Demo. Test for grow policy. Test feature weights. small cleanup. Remove iterator in evaluation. Fix dask test. Pass n_threads. Start implementation for categorical data. Fix. Add apply split. Enumerate splits. Enable sklearn. Works. d_step. update. Pass feature types into index. Search cut. Add test. As cat. Fix cut. Extract some tests. Fix. Interesting case. Add Python tests. Cleanup. Revert "Interesting case." This reverts commit 6bbaac2. Bin. Fix. Dispatch. Remove subtraction trick. Lint Use multiple buffers. Revert "Use multiple buffers." This reverts commit 2849f57. Test for external memory. Format. Partition based categorical split. Remove debug code. Fix. Lint. Fix test. Fix demo. Fix. Add test. Remove use of omp func. name. Fix. test. Make LCG impl compliant to std. Fix test. Constexpr. Use unsigned type. osx More test. Rebase error. Rebase error. Rebase error. Reverse unused changes. Config. Remove weird set thread. External memory test. Revert changes. Cleanup. wording. Fix doc. Test monotone constraint. Extract test for gamma. typo. Safe guard. Cleanup && comments. Update Python documents. Add push col page. hack. Port the sketch. Opt search bin. Cleanup. Reduce the gap. Fix sum hessian. Start cleaning up. Duplicated. Cleanup. lint. Test. Port the changes. test. Port the changes. Fixes && cleanup. Decide whether should sorted sketch be used. tests. Use regen. Lint. Revert. init. empty dataset. Handle empty dataset directly in quantile. empty. Update tests. Implement external memory support for hist with dense data. Rewrite approx. Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Regenerate the index. ama. Clang tidy. Retain page. Fix. Lint. Tidy. Integer backed enum. Convert to uint32_t. Prototype for saving gidx. Save cuts. Prototype on fetching. Copy the code. Simple test. Add gpair to batch parameter. Add hessian to batch parameter. Move. Pass hessian into sketching. Extract a push page function. Make private. Lint. Revert debug. Simple DMatrix. Initial port. Pass in hessian. Init column sampler. Unused code. Use ctx. Merge sampling. Use ctx in partition. Fix init root. Force regenerate the sketch. Create a ctx. Get it compile. Don't use const method. Use page id. Pass in base row id. Pass the cut instead. Small fixes. Debug. Fix bin size. Debug. Fixes. Debug. Fix empty partition. Remove comment. Lint. Fix tests compilation. Remove check. Merge some fixes. fix. Fix fetching. lint. Extract expand entry. Lint. Fix unittests. Fix windows build. Fix comparison. Make const. Note. const. Fix reduce hist. Fix sparse data. Avoid implicit conversion. private. mem leak. Remove skip initialization. Use maximum space. demo. lint. File link tags. ama. Fix redefinition. Fix ranking. use npy. Comment. Tune it down. Specify the tree method. Get rid of the duplicated partitioner. Allocate task. Tests. make batches. Log. Remove span. Revert "make batches." This reverts commit 33f7072. small cleanup. Lint. Revert demo. Better make batches. Demo. Test for grow policy. Test feature weights. small cleanup. Remove iterator in evaluation. Fix dask test. Pass n_threads. Start implementation for categorical data. Fix. Add apply split. Enumerate splits. Enable sklearn. Works. d_step. update. Pass feature types into index. Search cut. Add test. As cat. Fix cut. Extract some tests. Fix. Interesting case. Add Python tests. Cleanup. Revert "Interesting case." This reverts commit 6bbaac2. Bin. Fix. Dispatch. Remove subtraction trick. Lint Use multiple buffers. Revert "Use multiple buffers." This reverts commit 2849f57. Test for external memory. Format. Partition based categorical split. Remove debug code. Fix. Lint. Fix test. Fix demo. Fix. Add test. Remove use of omp func. name. Fix. test. Make LCG impl compliant to std. Fix test. Constexpr. Use unsigned type. osx More test. Rebase error. Rebase error. Rebase error. Reverse unused changes. Config. Remove weird set thread. External memory test. Revert changes. Cleanup. wording. Fix doc. Test monotone constraint. Extract test for gamma. typo. Safe guard. Cleanup && comments. Update Python documents. Add push col page. hack. Port the sketch. Opt search bin. Cleanup. Reduce the gap. Fix sum hessian. Start cleaning up. Duplicated. Cleanup. lint. Test. Port the changes. test. Port the changes. Fixes && cleanup. Decide whether should sorted sketch be used. tests. Extract row partitioner. Work on et. Remove test. base rowid. Fix. Fix reduce grad. Generate column matrix. Port the changes from updated driver. test sample. Cleanup. Fixes. fix clang. debug. Fix test. Revert changes. Lint. Initial commit for sparse page. fixes. fix tests. Remove column matrix. Make sure ref is used. Remove any_missing & gmat. Remove part builder. Fix approx test. Remove thread test. Fix sketch tests. Avoid a loop. fix evaluation tests. fix ghist index test. fix approx test. Fix histogram test. Note. start working on io. IO. Fix empty. Print time message. Remove the need to load sparse page. benchmark the external memory. [don't upload] Revert "benchmark the external memory. [don't upload]" This reverts commit 7fe631cd359cf6eb256b3aa08a39a2917203e045. log info. Fix rebase. fix rebase. fix. Cleanup & more tests. lint. fixes ellpack. ellpack. spec. Add tests. type. apple. s390x s390x. fix rebase. remove renamed file. Fix. Update documents. Remove check. Remove pruner. Cleanup test.

trivialfis · 2022-03-20T16:50:59Z

	Master	PR
airline	1075.46906893095	1073.9123199158348
bosch	87.04401617310941	79.65869551105425
higgs	142.96828106371686	143.29013332398608

trivialfis · 2022-03-20T20:38:08Z

src/tree/updater_quantile_hist.h

@@ -256,69 +252,73 @@ class QuantileHistMaker: public TreeUpdater {
  CPUHistMakerTrainParam hist_maker_param_;
  // training parameter
  TrainParam param_;
-  // column accessor


The PR is also a cleanup to hist with 8 member variables being removed.

trivialfis · 2022-03-20T20:39:48Z

tests/cpp/tree/test_quantile_hist.cc

-        : RealImpl(1, param, std::move(pruner), fmat, ObjInfo{ObjInfo::kRegression}, ctx) {}
-
-   public:
-    void TestInitData(const GHistIndexMatrix& gmat,


This test is outdated as we have split up most of the initialization from hist.

trivialfis · 2022-03-20T20:41:13Z

src/tree/updater_quantile_hist.cc

-}
-
-template<typename GradientSumT>
-template <bool any_missing>
 void QuantileHistMaker::Builder<GradientSumT>::ExpandTree(


The structure of this function is now close to approx and gpu_hist, but I haven't been able to unify them in this PR, which is left as future work.

trivialfis · 2022-03-20T20:44:55Z

src/common/column_matrix.h

@@ -141,28 +134,26 @@ class DenseColumn: public Column<BinIdxType> {
 class ColumnMatrix {


I think the column matrix can use some cleanups, but for now, I will leave it as it's to avoid regression in performance.

trivialfis · 2022-03-20T20:45:30Z

doc/tutorials/external_memory.rst

@@ -127,9 +127,12 @@ the tree method still concatenate all the chunks into 1 final histogram index du
 performance reason, but in compressed format.  So its scalability has an upper bound but
 still has lower memory cost in general.

-********


This is removed as the previous section is sufficient.

trivialfis · 2022-03-20T20:46:09Z

demo/guide-python/feature_weights.py

@@ -27,7 +27,7 @@ def main(args):
    dtrain.set_info(feature_weights=fw)

    bst = xgboost.train({'tree_method': 'hist',
-                         'colsample_bynode': 0.5},
+                         'colsample_bynode': 0.2},


Changed due to unified tree building routines.

RAMitchell

Looks very nice, are these benchmarks for external memory or without?

trivialfis · 2022-03-21T15:28:43Z

Looks very nice, are these benchmarks for external memory or without?

It's without external memory. I haven't started optimizing it yet, just want to get rid of the experimental status.

trivialfis mentioned this pull request Dec 23, 2021

External memory support. #7354

Closed

8 tasks

trivialfis force-pushed the hist-et branch from 8d2f0b5 to f925fe6 Compare January 10, 2022 17:32

trivialfis changed the title ~~[WIP] External memory support for hist with dense data.~~ [WIP] External memory support for hist Jan 16, 2022

trivialfis force-pushed the hist-et branch from f925fe6 to 71af21f Compare January 25, 2022 06:56

trivialfis force-pushed the hist-et branch from 71af21f to 8392d80 Compare February 7, 2022 10:33

trivialfis mentioned this pull request Feb 8, 2022

Prepare external memory support for hist. #7638

Merged

trivialfis force-pushed the hist-et branch 2 times, most recently from a863281 to bb9ee02 Compare February 15, 2022 07:36

trivialfis force-pushed the hist-et branch 2 times, most recently from 644c436 to f458200 Compare March 20, 2022 10:20

trivialfis added 5 commits March 20, 2022 18:20

Cleanups.

3a7777b

Cleanup.

2f67e1d

Cleanup.

502309f

cleanup.

b6d9f12

trivialfis changed the title ~~[WIP] External memory support for hist~~ External memory support for hist Mar 20, 2022

trivialfis marked this pull request as ready for review March 20, 2022 16:22

blank line.

4442d4a

Test.

ebf950d

trivialfis commented Mar 20, 2022

View reviewed changes

trivialfis added 2 commits March 21, 2022 13:14

Prepare to unify the update tree procedure.

3005864

Move.

a2e3f78

RAMitchell approved these changes Mar 21, 2022

View reviewed changes

trivialfis merged commit 4d81c74 into dmlc:master Mar 21, 2022

trivialfis deleted the hist-et branch March 21, 2022 16:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External memory support for hist #7531

External memory support for hist #7531

trivialfis commented Dec 23, 2021 •

edited

Loading

trivialfis commented Dec 23, 2021

trivialfis commented Mar 20, 2022

trivialfis Mar 20, 2022

trivialfis Mar 20, 2022

trivialfis Mar 20, 2022

trivialfis Mar 20, 2022

trivialfis Mar 20, 2022

trivialfis Mar 20, 2022

RAMitchell left a comment •

edited

Loading

trivialfis commented Mar 21, 2022 •

edited

Loading

		@@ -141,28 +134,26 @@ class DenseColumn: public Column<BinIdxType> {
		class ColumnMatrix {

External memory support for hist #7531

External memory support for hist #7531

Conversation

trivialfis commented Dec 23, 2021 • edited Loading

trivialfis commented Dec 23, 2021

trivialfis commented Mar 20, 2022

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

trivialfis Mar 20, 2022

Choose a reason for hiding this comment

RAMitchell left a comment • edited Loading

Choose a reason for hiding this comment

trivialfis commented Mar 21, 2022 • edited Loading

trivialfis commented Dec 23, 2021 •

edited

Loading

RAMitchell left a comment •

edited

Loading

trivialfis commented Mar 21, 2022 •

edited

Loading