[BUG] Memory usage problem after changing to statesync

[gnongs]

Describe the bug
(A clear and concise description of what the bug is.)
Block sync was done through statesync. Since then, from 9 a.m. to 23 p.m. on the 14th(UTC), we have confirmed that the amount of change in memory use has changed significantly. I looked at the log, but there was no log that looked like a problem. A similar phenomenon was confirmed on the node that performed other statesync.

And I even checked that it suddenly changed to a stable state after 23:00.

To Reproduce
(Describe the steps to reproduce the behavior.)
I think it can't be reproduce. So I recorded the way I did it.

1. Prepare nodes for providing snapshots.
2. When a snapshot is ready, copy the wasm folder from the node providing the snapshot to the node to receive the snapshot.
3. Open the RPC port firewall to the node that provides the snapshot.
4. Execute the terrad-unsafe-reset-all command on the node to receive the snapshot.
5. Move the previously received Wasm folder to $HOME/.terra/data/.
6. After setting options related to statesync, execute the terrad start command.
7. Thereafter, the block height was waited for synchronization, and the memory usage was checked after synchronization was completed.

Context & versions
(Delineate the correct versions of all the components involved in reproducing the bug.)

• Nodes that provide snapshots: v0.5.16(terrad), Ubuntu 20.04.x
• Nodes that receive snapshots: v0.5.16(terrad), Ubuntu 20.04.x

(if applicable) suggested solution
(If you know the solution to the bug, let us know!)
No I don't have.

[hanjukim]

What happens after statesync and restarting it? I think it is not weird thing to consume memory when it is doing statesync. We need in-depth research whether it is an actual leak or to be garbage collected eventually.

[gnongs]

After statusync was completed (09:33), Restart was performed. (09:54) Then, the memory usage as above was shown.
I think this phenomenon is not memory leak. If the graph take a closer look, it increases too fast to be called a memory leak.

To add additional information, whenever memory usage increased or decreased, CPU usage was increased to maximum.

[yun-yeo]

I think we solved this problem in #690