You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We tracked the number of non-garbage-collected objects during SVI inference. We found that objects grow over iterations but are occasionally collected. This evidence points to a cyclic reference somewhere. Among the amassing objects were Variables and Pyro Trace objects, suggesting that the cyclic reference is either in Pyro internals or in the networkx DiGraph object from which Trace inherits.
The text was updated successfully, but these errors were encountered:
And I am happy to report that the changes in #801 fix the memory leak that has been plaguing me for a month.
My garbage collection calls now safely report unique counts of Variables and memory consumption is constant across training.
Very happy this is resolved, this bug caused me grief for a while. Fantastic effort @fritzo and @eb8680
For reference: this bug made all the difference between being able to run some types of models and not being able to run them at all, it was not a little thing.
@karalets noticed memory growth when using Pyro.
We tracked the number of non-garbage-collected objects during SVI inference. We found that objects grow over iterations but are occasionally collected. This evidence points to a cyclic reference somewhere. Among the amassing objects were
Variable
s and PyroTrace
objects, suggesting that the cyclic reference is either in Pyro internals or in the networkxDiGraph
object from whichTrace
inherits.The text was updated successfully, but these errors were encountered: