redisEnableKeepAlive() fails,

[krha]

It seems that redisEnableKeepAlive options fails to maintain TCP connection.

This is gdb debug log and write call at redisBufferWrite(hiredis.c:1152) tries to write data to broken pipe.

Program received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7ffff5835700 (LWP 16210)]
0x00007ffff7bcbccd in write () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) where
#0 0x00007ffff7bcbccd in write () at ../sysdeps/unix/syscall-template.S:82
#1 0x000000000040c056 in redisBufferWrite (c=0x611f40, done=0x7ffff5834b50) at hiredis.c:1152
#2 0x000000000040c185 in redisGetReply (c=0x611f40, reply=0x7ffff5834b80) at hiredis.c:1195
#3 0x000000000040c3bb in __redisBlockForReply (c=0x611f40) at hiredis.c:1296
#4 redisvCommand (c=0x611f40, format=, ap=) at hiredis.c:1306
#5 0x000000000040c457 in redisCommand (c=, format=) at hiredis.c:1313
#6 0x0000000000404cfb in _redis_get_readdir (path=0x7fffec0051d0 "localhost", ret_list=0x7ffff5834d08) at redis.c:278

As you can see below c->fd, c->obuf, and c->err are all valid, but socket seems to lose connection to REDIS server even with redisEnableKeepAlive option.

1148 if (c->err)
1149 return REDIS_ERR;
1150
1151 if (sdslen(c->obuf) > 0) {
1152 nwritten = write(c->fd,c->obuf,sdslen(c->obuf));
1153 if (nwritten == -1) {
1154 if ((errno == EAGAIN && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
1155 /* Try again later _/
1156 } else {
1157 __redisSetError(c,REDIS_ERR_IO,NULL);
(gdb) p c->fd
$1 = 7
(gdb) p c->obuf
$2 = 0x7fffec004948 "_4\r\n$6\r\nLRANGE\r\n$11\r\nlocalhostβ\r\n$1\r\n0\r\n$2\r\n-1\r\n"
(gdb) p c->err
$3 = 0

[AllenDou]

@krha it looks like hiredis lost connection. Try to adjust REDIS_KEEPALIVE_INTERVAL, default is 15s, you can make it shorter. see what will happen.

[krha]

@AllenDou You're right. It lost connection. It's same with shorter REDIS_KEEPALIVE_INTERVA, e.g., 5s. For me, I made re-connection code to avoid this situation. But this is in general unexpected result of redisEnableKeepAlive() function.

At the server side, I can see the connected_clients is decreased. I also tested this with async connect, using redisAsyncConnect(). Async conncetion also loses a connection when I check at REDIS server side, but async conncetion seems to be automatically re-established without generating error.

[AllenDou]

can you show me your redis-server's redis.conf content? especially timeout ?

[krha]

@AllenDou connection timeout is set to 5 min in my configuration (I think it's default value).

$ cat /etc/redis/redis.conf
..

Close the connection after a client is idle for N seconds (0 to disable)

timeout 300
..

Since timeout is longer than REDIS_KEEPALIVE_INTERVAL. I changed the timeout to 1200 (20 min) with REDIS_KEEPALIVE_INTERVAL = 15 but same result.

FYI, here's full configuration file
$ cat /etc/redis/redis.conf
daemonize yes
pidfile /var/run/redis/redis-server.pid
port 6379
bind 127.0.0.1
timeout 1200
logfile /var/log/redis/redis-server.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /var/lib/redis
slave-serve-stale-data yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no

vm-enabled no
vm-swap-file /var/lib/redis/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
activerehashing yes

[pietern]

There is a difference between a TCP keepalive and an application level keepalive. The TCP keepalive is there to keep the TCP path between the client and server alive. This is necessary when for instance there is a NAT machine on the connection's route that aggressively terminates idle connections. TCP keepalive prevents this from happening. There also is Redis its own idle timeout. If a client hasn't made any request for this amount of time, Redis itself will terminate the connection. To prevent this from happening, you have to make sure the connection doesn't become idle and send a PING every now and then.

I suspect you're seeing Redis its timeout kick in, which cannot be solved with TCP keepalive.