[leo_storage] objects belonging to a bucket deleted can remain when leo_storage stopped during enqueuing-phase of delete-bucket

[mocchira]

As described on #725 (comment), leo_storage_handler_del_directory:rollback_to_pending can fail the leo_backend_db already finished its termination processing.

[vstax]

The problem still persists. Repeating experiment described in #725 (comment) using 1.4.0rc3

leofs-adm delete-bucket body 8b8124cd9ba4e9e191bf
leofs-adm delete-bucket bodytest 8b8124cd9ba4e9e191bf

Few seconds later, when stage switches to "enqueuing" I stop storage_0 node. After it's stopped (takes some time), I stop storage_1 node. After it's stopped, I start storage_0 node, then start storage_1 node.

End result:

$ leofs-adm delete-bucket-stats
- Bucket: body
node                         | state            | timestamp                   
-----------------------------+------------------+-----------------------------
storage_0@192.168.3.53       | enqueuing        | 2018-02-27 20:17:16 +0300               
storage_1@192.168.3.54       | enqueuing        | 2018-02-27 20:17:14 +0300               
storage_2@192.168.3.55       | finished         | 2018-02-27 21:01:00 +0300               


- Bucket: bodytest
node                         | state            | timestamp                   
-----------------------------+------------------+-----------------------------
storage_0@192.168.3.53       | enqueuing        | 2018-02-27 20:17:13 +0300               
storage_1@192.168.3.54       | enqueuing        | 2018-02-27 20:17:11 +0300               
storage_2@192.168.3.55       | finished         | 2018-02-27 20:54:19 +0300               
[vm@leo-m0 ~]$ leofs-adm du storage_0@192.168.3.53
 active number of objects: 481690
  total number of objects: 1354898
   active size of objects: 52768953540
    total size of objects: 191667623380
     ratio of active size: 27.53%
    last compaction start: ____-__-__ __:__:__
      last compaction end: ____-__-__ __:__:__

[vm@leo-m0 ~]$ leofs-adm du storage_1@192.168.3.54
 active number of objects: 481690
  total number of objects: 1509955
   active size of objects: 52768953540
    total size of objects: 213421617475
     ratio of active size: 24.73%
    last compaction start: ____-__-__ __:__:__
      last compaction end: ____-__-__ __:__:__

[vm@leo-m0 ~]$ leofs-adm du storage_2@192.168.3.55
 active number of objects: 0
  total number of objects: 1493689
   active size of objects: 0
    total size of objects: 210052283999
     ratio of active size: 0.0%
    last compaction start: ____-__-__ __:__:__
      last compaction end: ____-__-__ __:__:__

mq-stats shows empty queues. Nothing going on at all. Logs for all nodes here: storage_logs.tar.gz
After that I restarted all storage nodes, nothing changed.

[yosukehara]

We have been reproducing this issue. However, we need more time to reproduce and fix it. So we decided that we fix it with v1.4.1.