[BUG] 10x slowdown with cuML RF regression with new experimental backend #3128
Assignees
Labels
bug
Something isn't working
Comments
hcho3
added
? - Needs Triage
|
Performance profile from |
|
The old and new backends produce drastically different trees. Distribution of data count per node: Repro: from cuml.ensemble import RandomForestRegressor as rfr
from cuml.datasets import make_blobs
from cuml.metrics import r2_score
import numpy as np
import json
import time
import argparse
def init_tree(tree):
if 'count' not in tree:
tree['count'] = 0
if 'children' in tree:
init_tree(tree['children'][0])
init_tree(tree['children'][1])
def predict_with_json_tree(tree, x):
tree['count'] += 1
if 'children' not in tree:
assert 'leaf_value' in tree
return tree['leaf_value']
assert 'split_feature' in tree
assert 'split_threshold' in tree
assert 'yes' in tree
assert 'no' in tree
if x[tree['split_feature']] <= tree['split_threshold']:
return predict_with_json_tree(tree['children'][0], x)
return predict_with_json_tree(tree['children'][1], x)
def main(args):
X, y = make_blobs(n_samples=args.num_rows, n_features=args.num_features, centers=None,
random_state=42)
X, y = X.astype(np.float32), y.astype(np.float32)
clf = rfr(max_features=1.0, n_estimators=1, use_experimental_backend=(args.use_experimental_backend == 'true'), max_depth=args.max_depth)
clf.fit(X, y)
print(r2_score(y, clf.predict(X)))
json_obj = json.loads(clf.dump_as_json())
init_tree(json_obj[0])
tstart = time.perf_counter()
for idx, row in enumerate(X):
if idx % 5000 == 1:
tnow = time.perf_counter()
eta = (tnow - tstart) / idx * (X.shape[0] - idx)
print(f'{idx} of {X.shape[0]} ({idx * 100.0 / X.shape[0]} %, ETA: {eta} s)')
predict_with_json_tree(json_obj[0], row)
print(json.dumps(json_obj, indent=4))
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--num-rows', type=int, required=True)
parser.add_argument('--num-features', type=int, required=True)
parser.add_argument('--use-experimental-backend', type=str, choices=['true', 'false'], required=True)
parser.add_argument('--max-depth', type=int, required=True)
args = parser.parse_args()
main(args) |
|
This is the method responsible for deciding whether the current node needs to be split: https://github.com/rapidsai/cuml/blob/branch-0.17/cpp/src/decisiontree/batched-levelalgo/kernels.cuh#L243 We are missing the checks for Tagging @vinaydes JFYI. |
rapids-bot bot
pushed a commit
that referenced
this issue
Dec 4, 2020
…tical(#3243) Closes #3231 Closes #3128 Partially addresses #3188 The degenerate case (labels all identical in a node) is now robustly handled, by computing the MSE metric separately for each of the three nodes (the parent node, the left child node, and the right child node). Doing so ensures that the gain is 0 for the degenerate case. The degenerate case may occur in some real-world regression problems, e.g. house price data where the price label is rounded up to nearest 100k. As a result, the MSE gain is computed very similarly as the MAE gain. Disadvantage: now we always make two passes over data to compute the gain. cc @teju85 @vinaydes @JohnZed Authors: - Hyunsu Cho <chohyu01@cs.washington.edu> - Philip Hyunsu Cho <chohyu01@cs.washington.edu> Approvers: - Thejaswi Rao - John Zedlewski URL: #3243
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
RandomForestRegressorwith the new experimental backend is up to 10x slower than the old backend.Steps/Code to reproduce bug
which produces
on my machine. Note that
RandomForestClassifierdoes not have this issue.Environment details (please complete the following information):
The text was updated successfully, but these errors were encountered: