[BUG] Increasing memory usage leading to OOM when running UMAP in loop #4068
Comments
ietz
added
? - Needs Triage
|
Hi @ietz, thank you are filing an issue for this. I ran your script on my V100 (rapids 21.08 nightly packages) and was able to reproduce the trending sawtooth pattern that you've pointed out. I ran the loop for about 25 minutes while running a Adding a |
cjnolet
added
Waiting on author verification
|
Hey @cjnolet and thank you for your response The If you still want to reproduce the OOM without In terms of results, it sadly seems that even with my parameter sweep I could not get outputs from cuML UMAP that are comparable to those of the umap-learn library, as ~¼ of points are mapped to strange outlier positions far away from the main structure. I guess I'll just watch #3467 and try again once that is resolved |
|
As a result of your experience with this problem in RAPIDS, do you think it might be helpful if we added some documentation about the use of |
cjnolet
added
Waiting for more information
|
Sure, I think some info about that might very well help someone as long as you can find it. My problem was that I thought I had to look for some RAPIDS-specific solution as the problem was about GPU memory. As it's just standard Python, a short " |
|
This issue has been labeled |
|
This issue has been labeled |
Describe the bug
When I fit multiple UMAP models one after another, the GPU memory usage increases with most iterations, even though I do not keep any references to prior models or their results. At some point, I get an OOM error. As I do not keep any references, I would expect any data to be garbage collected to prevent the OOM from happening.
Steps/Code to reproduce bug
Here is a link to my Jupyter notebook on Google Colab: https://colab.research.google.com/drive/1mZew58DdWdI2cBuSRW5F3uUD7lXjHMUk
The issue occurs in the code segment
Looking at the GPU memory usage over time, I can see that the model is not always garbage collected between iterations. The data accumulates over a few iterations and is then deleted every so often, but not all of it. At some point this seems to always lead to an out of memory error. With the input data shape I chose for the Colab demo this took a lot longer than I expected (approx. 20 minutes, 641 iterations), but I think plotting the memory usage over time shows the issue quite nicely:
With larger datasets such as those that I used when I originally encountered this issue, the OOM happens after way fewer iterations, maybe 10. In the image you can see small and large "teeth". I think when I originally encountered this issue, I had the OOM on one of the small teeth, even before the first large drop in memory usage.
Expected behavior
I would expect that the memory usage does not increase to the point of an OOM error.
Environment details
nvidia-smireports CUDA 11.2, but the conda installation includes cudatoolkit 11.0.221 (see cell outputs in Colab). Not sure which of the two is relevant.conda listThe text was updated successfully, but these errors were encountered: