Fix leak of IOLoopKernelManager object #361
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After analyzing various leaked items when running either Notebook or Jupyter Kernel Gateway, one item that recurred across each kernel startup and shutdown sequence was an instance of IOLoopKernelManager. (Of course, when using JKG, this instance was KernelGatewayIOLoopKernelManager since it derives from the former.) The leak is caused by the circular references established in the `self._restarter` and `self.session.parent` members. This change breaks the circular reference when the restarter is stopped and during `cleanup()` of the kernel manager. Once the references are broken, the kernel manager instance can be garbage collected.
kevin-bates
added a commit
to kevin-bates/enterprise_gateway
that referenced
this pull request
Mar 14, 2018
PR jupyter-server#279 introduced some fixes to address file descriptor leaks - one of which was to shutdown the communication port. Since the kernel launcher listening on the other side may have already terminated, the shutdown method could throw an exception. This change catches, logs, then ignores such exceptions. While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
This was referenced Mar 14, 2018
lresende
pushed a commit
to jupyter-server/enterprise_gateway
that referenced
this pull request
Mar 19, 2018
While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
|
@minrk - thank you for merging these PRs. Can we assume that 5.2.4 will be released in a couple weeks given the other 5.2.x releases have been about 3-4 weeks apart? |
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After analyzing various leaked items when running either Notebook or
Jupyter Kernel Gateway, one item that recurred across each kernel
startup and shutdown sequence was an instance of IOLoopKernelManager.
(Of course, when using JKG, this instance was KernelGatewayIOLoopKernelManager
since it derives from the former.)
The leak is caused by the circular references established in the
self._restarterand
self.session.parentmembers. This change breaks the circular reference whenthe restarter is stopped and during
cleanup()of the kernel manager. Once thereferences are broken, the kernel manager instance can be garbage collected.