Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

memory leak when jmxfetch can't connect to an instance #30

Closed
clofresh opened this issue May 20, 2014 · 3 comments · Fixed by #42
Closed

memory leak when jmxfetch can't connect to an instance #30

clofresh opened this issue May 20, 2014 · 3 comments · Fixed by #42
Assignees
@yannmh
Labels
bug
Milestone
5.2.0

Comments

@clofresh
Copy link

After 13 days of failing to connect to a jmx instance:

2014-05-19 19:59:22,832 | ERROR| App | Cannot connect to instance 127.0.0.1:7199 java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]

jmxfetch grew to about 6gb:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dd-agent 27364  0.1 75.9 9722184 6218296 ?     Sl   May06  20:56      \_ java -jar /usr/share/datadog/agent/checks/libs/jmxfetch-0.3.0-jar-with-dependencies.jar --check jmx.yaml --check_period 15000 --conf_di

OS and Java version coming soon
@clofresh clofresh added the bug label May 20, 2014
@remh
Copy link
Contributor

remh commented May 20, 2014

Could you also link the full log (or at least the 100 first lines) and the yaml files used please ?
Thanks!

@clofresh
Copy link
Author

Most info here: https://datadog.desk.com/agent/case/10658

@isaacdd
Copy link
Contributor

isaacdd commented Jul 1, 2014

Is there a reason a new connection is forced to be created here? https://github.com/DataDog/jmxfetch/blob/master/src/main/java/org/datadog/jmxfetch/App.java#L131

This is what's causing the leak.

Update: Just tested and it helps to not force creating new connections, but something else is going on as well.

isaacdd added a commit that referenced this issue Jul 1, 2014
@isaacdd
Reduce memory leak #30
201d340 
Create a new connection only if needed. Creating too many connection
is causing too many threads to be created. This reduces the memory
leak significantly but is not the sole reason for the memory leak.
@remh remh added this to the Agent 5.1.0 milestone Nov 4, 2014
@yannmh yannmh modified the milestones: 5.3.0, Agent 5.1.0 Feb 3, 2015
@yannmh yannmh assigned yannmh and unassigned remh Feb 3, 2015
@yannmh yannmh mentioned this issue Feb 5, 2015
Closed
@yannmh yannmh modified the milestones: 5.2.0, 5.3.0 Feb 10, 2015
@yannmh yannmh mentioned this issue Feb 11, 2015
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yannmh yannmh

Labels
bug
Projects
None yet
Milestone
5.2.0
4 participants
@clofresh @remh @yannmh @isaacdd