Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Problem
There are lots of "server is readable but waiting_queue is empty" warnings in our logs when connections are closed by redis. This log was mainly used to find whether this case exists: we didn't send any command to redis but still receive response data. However, now this warning arose just because we didn't check whether the readable event is to close connection.
Most of the time this will not do harm. But in a rare case this will result in "-ERR Proxy error\r\n" response. Here's how it happen:
(1) Client send command to corvus
(2) corvus save this command in
ready_queue
of the connection between corvus and redis and wait for the writable event to forward this command to redis.(3) Before corvus send the command, redis close the connection because of timeout.
(4) corvus receive readable event from redis but find its
waiting_queue
is empty. Then it send theProxy error
response to client.How To Solve It
Distinguishing whether the readable event is to close connection is the perfect solution, but this require changing many codes so I think it's not worth it.
An easy workaround is to let corvus always close the connection before redis.
Firstly, use a
server_timeout
in corvus config which is smaller than thetimeout
in redis config, for example 500 forserver_timeout
and 600 fortimeout
. The gap between these two timeouts should be big enough to prevent the case that when corvus is about to send request to redis, redis close the connection at the same time.Secondly, instead of the timeout of command request to redis, change
server_timeout
to be the connection timeout. This is exactly what this PR do.