Primary use case for enhanced failure monitoring

[nshelms]

Hi -- What's the primary use case for enhanced failure monitoring? Does it make sense to enable it if we're already using HikariCP?

We're running a Spring Boot Java app with HikariCP and Liquibase configured; using the aws-mysql-jdbc driver with a 2 node aurora mysql cluster (1 writer and 1 reader). We kept running into an issue where Liquibase was executing a long running (55+ second) DML operation but would never finish due to SQLSTATE(08S02), Error Code (0) exception being return from the driver. Once we disabled Enhanced Failure Monitoring everything seemed to work as expected.

My theory is that the query was running longer than the failureDetectionTime + 3 failureDetectionIntervals, causing the plugin to kill the connection. Once the plugin kills the connection there's no other node to retry on since it's a DML operation which must be performed on a writer node. Given it's a DML operation, we wouldn't want to retry this operation if it failed in the middle of execution.

Does my theory seem like a reasonable explanation for the issue we were running into?

[karenc-bq]

Hi @nshelms, the primary use for EFM is to check the node’s health during long running queries.

By default, EFM checks the node’s health status every 5 seconds. In the event of a network failure, the EFM would detect the error as soon as possible and notify users earlier instead of waiting until the socket timeout. On the other hand, HikariCP only verifies the connection status before returning connections from the pool to the user. HikariCP does not provide any mechanism to detect network outages while executing long queries.

The 08S02 error you are seeing indicates a failover has been successfully performed, for more information on the error see here. Since it is a specific exception from the driver, you would need to ensure HikariCP correctly handles it instead of closing the connection immediately by setting the exception override class. You can find a sample implementation of HikariCPSQLException here.

[nshelms]

Thanks @karenc-bq. And thank you providing the HikariCPSQLException code. We implemented that class but it didn't seem to prevent the issue from occurring. Would you have any idea as to why the exception would be raised in the first place. I could not find any evidence of a failover actually being performed on the Aurora Writer Instance, manual or automatic.

[karenc-bq]

That is interesting, would you be able to provide us with the following to help with the investigation:

• driver logs
• sample code

[nshelms]

Version 1.1.9 seems to have fixed our issue. Marking this discussion as closed.

For reference, #425