Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak due to cyclic references in Trace #798

Closed
fritzo opened this issue Feb 22, 2018 · 1 comment
Closed

Memory leak due to cyclic references in Trace #798

fritzo opened this issue Feb 22, 2018 · 1 comment
Assignees
Labels
Milestone

Comments

@fritzo
Copy link
Member

fritzo commented Feb 22, 2018

@karalets noticed memory growth when using Pyro.

We tracked the number of non-garbage-collected objects during SVI inference. We found that objects grow over iterations but are occasionally collected. This evidence points to a cyclic reference somewhere. Among the amassing objects were Variables and Pyro Trace objects, suggesting that the cyclic reference is either in Pyro internals or in the networkx DiGraph object from which Trace inherits.

@fritzo fritzo added the bug label Feb 22, 2018
@fritzo fritzo added this to the 0.2.0 release milestone Feb 22, 2018
@fritzo fritzo self-assigned this Feb 22, 2018
@karalets
Copy link
Collaborator

karalets commented Feb 22, 2018

And I am happy to report that the changes in #801 fix the memory leak that has been plaguing me for a month.
My garbage collection calls now safely report unique counts of Variables and memory consumption is constant across training.

Very happy this is resolved, this bug caused me grief for a while. Fantastic effort @fritzo and @eb8680

For reference: this bug made all the difference between being able to run some types of models and not being able to run them at all, it was not a little thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants