Endless loop on DB.Close()

[pbudzik]

badger 2019/01/03 11:26:37 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:37 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000001.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:38 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:38 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000002.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:39 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:39 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000003.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:40 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:40 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000004.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:41 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:41 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000005.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:42 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:42 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000006.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:43 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:43 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000007.sst: no such file or directory. Retrying...
^Cbadger 2019/01/03 11:26:44 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:44 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000008.sst: no such file or directory. Retrying...
badger 2019/01/03 11:26:45 INFO: Storing value log head: {Fid:0 Len:42 Offset:53}
badger 2019/01/03 11:26:45 ERROR: Failure while flushing memtable to disk: : open /tmp/ii2/data/test_1546510662/000009.sst: no such file or directory. Retrying...

I have to then manually kill -9.... Has occured a few times recently.

[manishrjain]

It looks like your /tmp might have been cleaned, so the directory or the file is no longer there.

[pbudzik]

Seems like a good guess, but nope. I've just tried another directory with the same exactly behavior. It turned out to be a bug in my code, but maybe badger should recover and panic/exit. For sure no process deleted files, so likely some unsafe concurrent operation messed that up.

[manishrjain]

Badger is doing the right thing by retrying indefinitely until it is able to create those files because otherwise, you lose data. This also gives the user a chance to fix their directory issues if possible.

Killing it is always a possibility, but that's a decision a user should make. At Badger level, we can't decide when to give up trying.