Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollbar stops sending after few days #246

Closed
amiram opened this issue Feb 18, 2021 · 7 comments · Fixed by #254, nbadal/ktlint-intellij-plugin#58 or AdminOfThis/Frequent#105
Closed

Rollbar stops sending after few days #246

amiram opened this issue Feb 18, 2021 · 7 comments · Fixed by #254, nbadal/ktlint-intellij-plugin#58 or AdminOfThis/Frequent#105

Comments

@amiram
Copy link

amiram commented Feb 18, 2021

We upgraded our java server recently to the recent spring libraries, and rollbar stops sending errors after few days of the server working. Nothing is found in our logs. Is there any way to log locally when rollbar fails to send?
We have other rollbar instances, not in java, that are sending as usual. We also have other java servers that are sending as usual but they are not in production and they are restarted very frequently.

@diegov
Copy link
Contributor

diegov commented Feb 26, 2021

Hi @amiram, rollbar-java and related projects use slf4j, and a failure to send a payload will log an error through slf4j. Whether that ends up in the logs for your app will depend on what slf4j can find in your classpath at runtime, but for example for an out of the box spring-boot app, a failure to send the payload due to a connection refused would result in this, via Logback:

2021-02-26 13:06:07.504 ERROR 340268 --- [buffered_sender] com.rollbar.notifier.sender.SyncSender   : Error sending the payload.

com.rollbar.notifier.sender.exception.SenderException: java.net.ConnectException: Connection refused (Connection refused)
        at com.rollbar.notifier.sender.AbstractSender.send(AbstractSender.java:46) ~[rollbar-java-1.7.5.jar:1.7.5]
        at com.rollbar.notifier.sender.BufferedSender$SendTask.run(BufferedSender.java:239) ~[rollbar-java-1.7.5.jar:1.7.5]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

If you have a Logback configuration in your application, the loggers used by rollbar are under com.rollbar, so you should ensure those are configured at level ERROR at least. The specific logger that would report a failure to send an error to Rollbar would be under com.rollbar.notifier.sender.

Can you add here the versions of the rollbar and spring dependencies, as well as the java version? And if possible, the versions under which things were working (what you upgraded from)?

Thanks
Diego

@amiram
Copy link
Author

amiram commented Mar 2, 2021

We discovered that lombok was adding an override for toString(). If two classes had a reference to each other, the serializer failed to serialize the object, and also it caused all other rollbar payloads to not being sent right after the first serializer error, even if they had no serializer error.

Example:

@Data
public class User {
    @ToString.Exclude // This solves it. Needs to be in both places if it is lazy loaded
    @ManyToMany
    @JoinTable(...)
    private List<Account> accounts;
} 

@Data
public class Account {
    @ToString.Exclude // This solves it. Needs to be in both places if it is lazy loaded
    @ManyToMany
    @JoinTable(...)
    private List<User> users;
} 

@diegov
Copy link
Contributor

diegov commented Mar 2, 2021

The fact that a class cannot be serialized is out of our control, but if that causes subsequent payloads to fail as well, we need to look into it. Could you please add the versions of the rollbar dependencies you're using?

Thanks
Diego

@amiram
Copy link
Author

amiram commented Mar 2, 2021

We're using rollbar 1.7.5

@klethys
Copy link

klethys commented Mar 2, 2021

@diegov For further clarification:

We are using Spring Boot as our Java framework. We use Lombok in Java which generates an automated toString method in human readable form. When you call toString on the payload, Lombok recursively moves through all non-primitive class members and prints the values of all primitives.

We were sending a class to rollbar that had a circular dependency (this is not uncommon with JPA). The toString function in this case was causing an infinite loop which led to a stack overflow. This stack overflow occurred in the async thread dedicated to sending data to rollbar. Because this is not an Exception but rather an Error, not only was it uncaught, but it also seems to have killed the thread that was sending data to rollbar. It seems as though there was only one thread responsible for this, and when it died, data stopped being sent to rollbar.

For a fix, you should probably focus on handling an Error in the async thread in a way that the thread does not die. Please let us know if you need any further information from us.

@diegov
Copy link
Contributor

diegov commented Mar 2, 2021

Hi @klethys

Thanks very much for the detailed info.

Yes, there are a few places where we catch Exception instead of Throwable, one for sure is the root cause of your issue, and we'll have a fix for that released in the next few days.

Thanks
Diego

@diegov
Copy link
Contributor

diegov commented Mar 15, 2021

@amiram @klethys this is now fixed in the latest version, 1.7.6.

Thanks for reporting the issue and helping identify the root cause!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants