Poor GC of leveled during handoff

[martinsumner]

During handoff (i.e. a node transfer), the PUT rate into a vnode may be higher than normal.

During the 2.9 cycle of release candidates, a change was introduced to do per-process GC on leveled_sst files if they still exist 10s after they were created. This stopped an issue with poor GC on these processes when running Riak on OTP16.

When put pressure is high, level 1 files may be moved to the delete_pending state (because they have been replaced), before the GC message is received. The GC message broke the timeout loop on delete, meaning that the file process would not correctly self-delete until the vnode is shutdown. This could lead to issues with the ulimit open files limit.

[martinsumner]

This could also have an impact where leveled is used as a Tictac AAE backend.

[martinsumner]

https://github.com/basho/riak_kv/tree/riak_kv-2.9.0p1