[BUG] Negative R-Squared with Nightly (0.17) #3305
Comments
ibrisa
added
? - Needs Triage
ibrisa
changed the title
|
R2 score can be negative depending on how the classifier fits. For more detailed explanation: https://stats.stackexchange.com/questions/12900/when-is-r-squared-negative Below is a small code to show that sklearn and cuml both produce negative R2 scores. In your case, these values are still different because you're using sklearn RFR and cuml.dask RFR - these might produce different results. |
Nanthini10
added
4 - Waiting on Author
|
Thank you so much for the response and the link, it was helpful to read. It sounds like the R2 is poor because of the model fit then. Our main concern is that there’s such a big difference between the sklearn and cuml, mainly that we always get a negative on cuml, and always get a positive on sklearn. Can you help us identify what the reason might be for that? Here are the MSE and R2 results for 5 different runs from the data I'd attached previously.
|
|
@ibrisa Sorry for the late response. Can you try to run the workflow on single-GPU instead of multi-GPU? And set |
|
Hi @Nanthini10, thanks for the response. I have tried it on single-GPU before (on a Power9 system with v14) and the MSE and R2 results from cuML and sklearn matched, leading me to think this is an issue specific to the multi-gpu version. I reran the single GPU code on x86 system with v17 with the previously linked data and Here are the MSE/R2 results run on Single GPU, x86 system, v17 with
Are there any other tests that would be helpful? |
|
Thanks for the details @ibrisa! This is helpful, let me debug this and get back to you. |
|
Hi @Nanthini10, An update on this ticket - I tried rerunning this using the dask version of the code but specifying only 1 GPU and it gave similarly poor performance metrics.
The only change in the python code is changing the following line to specify n_workers as 1: |
|
@ibrisa Thanks for the update! I was able to reproduce similar behavior and it looks like the predictions are not lining up. There's going to be an update for RandomForest estimators soon - https://github.com/rapidsai/roadmap/issues/299 which might fix the issue. I'll try to get you an update soon. |
|
@Nanthini10 Great news that you were able to reproduce it! Unfortunately, the link to the roadmap is broken. Is there a timeline for when the RF update will come out? |
|
@ibrisa Oh sorry! It should come out in about 6 weeks with 0.19 |
|
This issue has been labeled |
|
This is still an active issue. Waiting to test with 0.19 fixes |
|
This is an issue because of how the partitions are made by dask. When you call r2_score on Use a partitioned Outputs: As you can see, the |
Nanthini10
added
the
4 - Waiting on Author
Describe the bug
I am using nightly v0.17 RAPIDS on x86 to run the random forest regression algorithm and observing negative r-squared cuML values. I am comparing to sklearn on the same data and those r-squared values are positive. The MSE values do look correct (low and close to the sklearn MSE). The smallest example dataset I consistently see this occurring with is available here. Also linked to it below.
This occurs on MNMG (dask) version, which I've run with both 8 workers and just 1 worker. It is worth noting that I started by using the single node version of random forest regression (without dask) and that version never ran into a negative r-squared value. Any help to fix the negative r2 would be much appreciated.
Steps/Code to reproduce bug
Python Code:
Input and Sample Output
Input Link: https://drive.google.com/file/d/1zD533JPbTAmXU6zu9fr4Tp4Ef4k3VEbA/view?usp=sharing
Sample output (Nightly):
Test 1:
Sklearn MSE: 0.005 Sklearn R2: 0.7988 Cuml MSE: 0.0284 Cuml R2: -0.13Test 2:
Sklearn MSE: 0.0045 Sklearn R2: 0.8086 Cuml MSE: 0.0276 Cuml R2: -0.152etc
Environment details (please complete the following information):
x86:
Eight NVIDIA A100 40GB “Ampere” GPU accelerators
NVSwitch provides full 600GB/s NVLink connectivity between all GPUs
320GB total GPU Memory
Two AMD EPYC 7002-Series 64-core CPUs
1TB DDR4 System Memory
(4) 3.84TB NVMe SSDs for 15TB Local Storage
200GbE QSFP28 Ethernet adapter
Eight Mellanox ConnectX-6 QSFP28 200Gbps HDR InfiniBand/200GigE adapters
One Gigabit Ethernet RJ45 management port and one BMC RJ45 port
With conda:
Click to see `Conda List` output:
The text was updated successfully, but these errors were encountered: