Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ticker leak in weighted round robin picker creation #6642

Closed
atollena opened this issue Sep 18, 2023 · 4 comments
Closed

Ticker leak in weighted round robin picker creation #6642

atollena opened this issue Sep 18, 2023 · 4 comments

Comments

@atollena
Copy link
Collaborator

What version of gRPC are you using?

1.57 (I believe the bug is still present on master)

What version of Go are you using (go version)?

1.21.0

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

I am using the weighted_round_robin balancer via service config.

I have servers set with MaxConnectionAge to 5 minutes as to force DNS re-resolultion per #1663 (comment).

What did you expect to see?

CPU and memory usage stays constant.

What did you see instead?

The CPU usage and memory used by the application keeps growing.

Screenshot 2023-09-18 at 07 51 25
@atollena
Copy link
Collaborator Author

Thanks to @HippoBaro for discovering this problem and the initial investigation.

I think the bug is just that this ticker:

go func() {
ticker := time.NewTicker(time.Duration(p.cfg.WeightUpdatePeriod))
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
p.regenerateScheduler()
}
}
is not properly closed. Suggested fix: #6643.

@easwars
Copy link
Contributor

easwars commented Sep 18, 2023

I hate the Ticker. This is the not the first time I'm seeing it cause a memory leak and probably wont be the last time either.

@atollena
Copy link
Collaborator Author

Yes, ideally there should be a vet rule for this.

@easwars
Copy link
Contributor

easwars commented Sep 18, 2023

Fixed by #6643.

@easwars easwars closed this as completed Sep 18, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants