Received a successful response to the tx request, but tx is not stored in mempool.

[egonspace]

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
latest

ABCI app (name for built-in, URL for self-written if it's publicly available):
cosmos-sdk

Environment:

• OS (e.g. from /etc/os-release):
• Install tools:
• Others:

What happened:
With multiple clients sending tx requests at the same time, some txs have been found to be lost without being stacked in mempools. Of course, this doesn't seem to cause a fatal flaw, but from the client's point of view, they has been successfully answered to the tx request, which is likely to cause inconvenience in usability if later found that it wasn't stored in blocks and even in mempools.

After investigating the cause, I think this scenario is possible. Suppose the mempool is almost full and processing two txs requests simultaneously. When the following code is met in the function that processes callback after successfully passing CheckTx and sending a successful response, the second thread returns an error because it is determined that the mempool is full when the first thread is filled with tx.

https://github.com/tendermint/tendermint/blob/master/mempool/clist_mempool.go#L416

What you expected to happen:

Have you tried the latest version: yes/no

How to reproduce it (as minimally and precisely as possible):

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):

Config (you can paste only the changes you've made):

node command runtime flags:

Please provide the output from the http://<ip>:<port>/dump_consensus_state RPC endpoint for consensus bugs

Anything else we need to know:

[tessr]

Thanks for this report, and especially for the theory on what's going on. It sounds like we should make sure that that second thread is returning the error all the way back to the client so that it (the client) knows that the transaction was not included in the mempool.

[melekes]

#4759 (review)

[melekes]

Closing for #3546