not trigger the channelInactive() method in ConnectionWatchdog

[TomGitHome]

Environment

• Lettuce version(s): [6.1.5.RELEASE]
• Redis version: [6.2.7]

Basic Information:

There are two servers, one acting as the primary server and the other as the standby.
Both the primary and standby servers deploy Redis in Docker containers, and both are running.
The Spring Boot project is deployed in a Docker container.
The Spring Boot project uses Lettuce to connect to the Redis service.

Issue Description:

Currently, the Redis virtual IP is on the standby server. When using iptables -A INPUT -p tcp --dport 6379 -j REJECT on the standby server to bring Redis down, the Redis virtual IP automatically switches to the primary server. This triggers the channelInactive() method in ConnectionWatchdog, and the connection to the Redis service is restored successfully.
After Redis is restored on the standby server by using iptables -D INPUT -p tcp --dport 6379 -j REJECT, and then Redis is brought down on the primary server by executing iptables -A INPUT -p tcp --dport 6379 -j REJECT, the Redis virtual IP automatically switches to the standby server. However, this does not trigger the channelInactive() method in ConnectionWatchdog, and the connection to the Redis service remains abnormal for about 18 minutes before recovering.

[TomGitHome]

[tishun]

Hey @TomGitHome,

When changing the iptables rules of the container you might end up in a state where the driver does not receive any notification from the server that the connection was dropped. Depending on your KEEPALIVE and TCP_USER_TIMEOUT options you may end up waiting a large amount of time (specific to the OS of the container) until the connection drops.

You can read more details about this condition in the following article.

To make sure you do not run into the same problem, could you verify that you are following the best practices described in the Redis documentation?

Important

The suggested values in the article are sample values, you need to use the ones that make sense for your system, the basic recommendation is to follow the rule TCP_USER_TIMEOUT = TCP_KEEP_IDLE+TCP_KEEPINTVL * TCP_KEEPCNT

[TomGitHome]

Hey@tishun thanks!
The current configuration is as follows.

[TomGitHome]

@tishun
The following changes were made based on version 6.1.5.RELEASE. After testing, the issue still occurs. As you mentioned, it is possible that the client still does not receive the server's disconnection signal. I will conduct packet capture testing in the future.

The current solution on my side is:
when a RedisTimeoutException occurs, I re-establish the connection pool by checking the availability of the Redis service through the handshake returned by the RedisClient.

[tishun]

Hey @TomGitHome ,

I do not see in your configuration the setting for the TCP_USER_TIMEOUT?
In your case it would have to be part of the SocketTimeout settings:

   SocketOptions socketOptions = SocketOptions.builder()
            .keepAlive(keepAliveOptions)
            .connectTimeout(Duration.ofSeconds(10))
            .tcpUserTimeout(tcpUserTimeout)               // <--- THIS IS MISSING
            .build();

... and if you want to use the same setting as the one in the example you'd have to add this too:

   SocketOptions.TcpUserTimeoutOptions tcpUserTimeout = SocketOptions.TcpUserTimeoutOptions.builder()
            .tcpUserTimeout(Duration.ofSeconds(20))
            .enable().build();