Mempool view is different across nodes

[hansieodendaal]

Describe the bug
With a recent stress test, the mempool view was different across nodes, resulting in many transactions not being mined.

To Reproduce
Steps to reproduce the behaviour:

1. Run a 36,000 transaction stress test from 6 sending wallets to 10 receiving wallets, with each wallet connected to its own base node.

Expected behaviour
The mempool view should be fairly consistent across base nodes for the stress test duration, i.e. until all transactions are mined.

Screenshots
n/a

Desktop (please complete the following information):

• OSX, Windows

Smartphone (please complete the following information):

• n/a

Additional context
Add any other context about the problem here.

[hansieodendaal]

This behaviour was reproduced in recent smaller stress tests - 20,000 transactions from 2x sender wallets to 2x receiver wallets.

• Restarting base node does not help
• Restarting wallet does not help
• Repointing the wallet to a different base node gets the wallet to resubmit its missing transactions to the new base node, thus improving the mempool view across different nodes

[hansieodendaal]

Some results about mempool sync behaviour during 2x consecutive stress tests. There were 2x sender and 2x receiver wallets with their own base nodes on one computer, and a base node + SHA3 miner + XMRig miner together with 3x monitor nodes on another computer. A total of 20,000 negotiated transactions were sent.

The primary setting under investigation was [base_node.mempool] service.initial_sync_num_peers
All nodes continued to run throughout the test, nothing was restarted.

[base_node.mempool] service.initial_sync_num_peers = 2

Both the sender and receiver nodes had virtually the same mempool stats throughout due to the connected wallets

[base_node.mempool] service.initial_sync_num_peers = 5

In the 2nd test the rest of the network was much more in tune, which is the behaviour we want, although it does not make sense that [base_node.mempool] service.initial_sync_num_peers can have that influence.

[hansieodendaal]

Using v0.38.5 I performed a stress test with interactive transactions to peer wallets, 3x senders (16,000 transactions each) on one computer to 4x receivers (12,000 transactions each) on another computer (48,000 transactions in total).

The latest changes seemed to solve all issues. All 8x base nodes I monitored were configured to sync mempools from 5x peers and re-sync when 5 blocks behind; none of the base nodes lagged with chain sync for any type of extended period, they always caught up again.