Large number of exists() queries causes ArrayIndexOutOfBoundsException

[U-238]

If I have a redis connection open for a long time and often call exists(), the RedisOutputStream will eventually exceed the buffer size, because it is never flush()ed. Notice how the array index increases with each error message. A possible fix is to flush() the redis output buffer every time exists() is called.

DEBUG [main] - Starting thread YCBG|20140315002300
Exception in thread "YCBG|20140315002300" java.lang.ArrayIndexOutOfBoundsException: 11945
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:37)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:69)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:63)
    at redis.clients.jedis.Connection.sendCommand(Connection.java:84)
    at redis.clients.jedis.BinaryClient.exists(BinaryClient.java:110)
    at redis.clients.jedis.BinaryJedis.exists(BinaryJedis.java:143)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:213)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:219)
    at com.ourcast.radar.input.ftp.bom.BoMFTPDownloadThread.run(BoMFTPDownloadThread.java:84)
DEBUG [main] - Starting thread YWOL|20140315002300
Exception in thread "YWOL|20140315002300" java.lang.ArrayIndexOutOfBoundsException: 11946
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:37)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:69)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:63)
    at redis.clients.jedis.Connection.sendCommand(Connection.java:84)
    at redis.clients.jedis.BinaryClient.exists(BinaryClient.java:110)
    at redis.clients.jedis.BinaryJedis.exists(BinaryJedis.java:143)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:213)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:219)
    at com.ourcast.radar.input.ftp.bom.BoMFTPDownloadThread.run(BoMFTPDownloadThread.java:84)
DEBUG [main] - Starting thread YGBI|20140315002300
Exception in thread "YGBI|20140315002300" java.lang.ArrayIndexOutOfBoundsException: 11947
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:37)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:69)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:63)
    at redis.clients.jedis.Connection.sendCommand(Connection.java:84)
    at redis.clients.jedis.BinaryClient.exists(BinaryClient.java:110)
    at redis.clients.jedis.BinaryJedis.exists(BinaryJedis.java:143)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:213)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:219)
    at com.ourcast.radar.input.ftp.bom.BoMFTPDownloadThread.run(BoMFTPDownloadThread.java:84)
DEBUG [main] - Starting thread YPTN|20140315002300
Exception in thread "YPTN|20140315002300" java.lang.ArrayIndexOutOfBoundsException: 11948
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:37)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:69)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:63)
    at redis.clients.jedis.Connection.sendCommand(Connection.java:84)
    at redis.clients.jedis.BinaryClient.exists(BinaryClient.java:110)
    at redis.clients.jedis.BinaryJedis.exists(BinaryJedis.java:143)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:213)
    at com.ourcast.radar.persistence.GriddedDataManager.exists(GriddedDataManager.java:219)
    at com.ourcast.radar.input.ftp.bom.BoMFTPDownloadThread.run(BoMFTPDownloadThread.java:84)

[HeartSaVioR]

Hello!
Please describe what Redis / Jedis version you're using, part of source code that you used for reproduce this. (simplified but can reproduce this issue would be fine.)
Thanks for reporting!

[U-238]

I have been trying to reproduce this issue more recently, but I can't.

From looking at the Jedis code (latest git), and the stack trace, it appears that this would happen at RedisOutputStream.java:37 where "count" is bigger than the buffer size (buf.length). But I can't reproduce the conditions that led to count being bigger than buf.length :(

[HeartSaVioR]

@U-238 There're so many issues about this.
But I know we can check this before go further.

Please make sure that you call pool.returnBrokenResource or destroy instance when Jedis instance is broken.
(broken means JedisConnectionException, or JedisDataException with message included connection - it is due to twemproxy, I don't like it.)

Similar issue : #552

If we have IO exception with Jedis instance, internal inputstream and outputstream cannot work well. Maybe this issue is kind of that problems.

If your issue isn't resolved after applying this, please comment with full source codes with your Redis / Jedis version.

[sweetest]

having a same problem here with redis 2.6.4 and jedis 2.4.2

[HeartSaVioR]

@U-238 @sweetest It's been resolved by #651 and applied to 2.5.1.
Please upgrade Jedis to reflect. Sorry for inconvenience!