Close HTTP idle connections to prevent goroutines leak #855
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When either ES or process manager http clients go out of scope, make sure we close keep-alive HTTP connections. Otherwise, the underlying goroutines stay alive forever (with their heap), provoking memory leaks leading to the operator being killed by k8s (OOM).
sebgl
requested review from
artemnikitin,
barkbay,
nkvoll,
pebrc and
thbkrkr
as code owners
sebgl
commented
May 14, 2019
| @@ -208,6 +208,7 @@ func (d *defaultDriver) Reconcile( | |||
| *min, | |||
| caCert, | |||
| ) | |||
| defer esClient.Close() | |||
There was a problem hiding this comment.
just realized one caveat of this approach is the client may be used by the observer (first iteration - not used in next iterations if no auth/cert change). No super big deal (idle connections closed, doesn't stop active ones), but this action still has an impact that is not immediately obvious.
I'll try to fix it so the observer client is a different one.
There was a problem hiding this comment.
Should be fixed in next commit, by passing a different client to the observers.
If we use the same client that gets closed at the end of a reconciliation, it would otherwise impact observers' client. Note: most of the times, the client passed to the observer is not used. Observer keeps using the same client it was created with unless the new one is different (different certs, auth, etc.).
|
Unfortunately it's hard to create a unit/integration test for this. |
thbkrkr
reviewed
May 15, 2019
operators/pkg/controller/elasticsearch/processmanager/client.go
Outdated
Show resolved
Hide resolved
thbkrkr
reviewed
May 15, 2019
operators/pkg/controller/elasticsearch/processmanager/client.go
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When either ES or process manager http clients go out of scope, make
sure we close idle keep-alive HTTP connections.
Otherwise, the underlying goroutines stay alive forever (with their heap).
Since we create new clients at every reconciliation iteration (and observer
iteration for the process manager), we end up provoking memory leaks,
leading to the operator being killed by k8s (OOM).
I manually tested that goroutine and memory leak does not occur anymore. See #854.
Fixes #854.