Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leaks with copy.deepcopy #514

Closed
MicahGale opened this issue Aug 27, 2024 · 6 comments · Fixed by #539
Closed

Memory leaks with copy.deepcopy #514

MicahGale opened this issue Aug 27, 2024 · 6 comments · Fixed by #539
Assignees
@MicahGale
Labels
bugs A deviation from expected behavior that does not reach the level of being reportable as an "Error".

Comments

@MicahGale
Copy link
Collaborator

So I was making a lot of copies of a cell with copy.deepcopy and it went very badly

import montepy
problem = montepy.read_input("foo")
for i in range(1_000):
    new_cell = copy.deepcopy(problem.cells[1])
    ...

This script kept crashing. It got really slow around i==11 and was using over 3GB of RAM!

This is very telling for when I ran ctrl-C:

^CTraceback (most recent call last):
  File "/home/mgale/dev/montepy/mod_big_model.py", line 11, in <module>
    new_cell = copy.deepcopy(base_cell)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 264, in _reconstruct
    args = (deepcopy(arg, memo) for arg in args)
KeyboardInterrupt

We have a copy recursion problem.
@MicahGale MicahGale added the bugs A deviation from expected behavior that does not reach the level of being reportable as an "Error". label Aug 27, 2024
@MicahGale
Copy link
Collaborator Author

I think we need __weakref__ for self._problem

@MicahGale
Copy link
Collaborator Author

Also to implement #469

@MicahGale
Copy link
Collaborator Author

Temporary workaround:

  1. Unlink problem
  2. Copy object
  3. relink object
problem =...
base_cell = problem.cells[1]
base_cell._problem = None
new_cell = copy.deepcopy(base_cell)
base_cell.link_to_problem(problem)
... # make more copies if needed, etc
new_cell.link_to_problem(problem)

This still isn't super memory efficient, but it works well enough.

@MicahGale MicahGale mentioned this issue Aug 27, 2024
Merged
2 tasks
@MicahGale MicahGale reopened this Aug 28, 2024
@MicahGale
Copy link
Collaborator Author

I have a hunch that this is causing #463.

@MicahGale
Copy link
Collaborator Author

Weakref does in fact resolve this, but breaks pickle.

@MicahGale MicahGale mentioned this issue Sep 10, 2024
Merged
3 tasks
@MicahGale
Copy link
Collaborator Author

I have a hunch that this is causing #463.

I tried fixing #463 with #539, and it does not seem to address the problem at all. But this would help break the infinite recursion that makes debugging impossible.

@MicahGale MicahGale self-assigned this Sep 11, 2024
MicahGale added a commit that referenced this issue Sep 18, 2024
@MicahGale
Added #514 to changelog.
50255a7
@MicahGale MicahGale mentioned this issue Oct 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MicahGale MicahGale

Labels
bugs A deviation from expected behavior that does not reach the level of being reportable as an "Error".
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implementing weakref and cloning idaholab/MontePy
1 participant
@MicahGale