In-mem database backend

[wilsonwang371]

Guys,

We have a case in our company to use in memory only database which do not need to persist kv onto disk but it need strong consistency.

Do you guys think it would be a good chance to make changes to boltdb interface to make it more general for other btree based in mem db alternatives?

@gyuho @ptabor

[gyuho]

Interesting idea.

@wilsonwang371 Can we research more if there is any existing system that supports pluggable storage backends? I think this requires multiple phases, starting with more generic peer-to-peer snapshot contracts. Or making existing boltdb optionally bypass file system calls is another option.

c.f. #13123

/cc @xiang90

[MakDon]

Is it a good idea that store the bbolt's data file in tmpfs?

[wilsonwang371]

Is it a good idea that store the bbolt's data file in tmpfs?

I am still doing related research. Basically, bbolt using tmpfs won't help much because bbolt is already using memmap to access the data. Using tmpfs may help to improve the performance a little bit but still, bbolt was not designed for a in-mem database.

I am still thinking about this issue. I will update it when I have some new idea.

[wilsonwang371]

Interesting idea.

@wilsonwang371 Can we research more if there is any existing system that supports pluggable storage backends? I think this requires multiple phases, starting with more generic peer-to-peer snapshot contracts. Or making existing boltdb optionally bypass file system calls is another option.

c.f. #13123

/cc @xiang90

Sorry for the late reply. Agreed. I think a generic peer-to-peer snapshot contracts would be more urgent for us on this topic. Maybe we can just firstly introduce a layer on top of bbolt at the very beginning?

[morigs]

Somewhat similar idea are currently being discussed in TiKV
tikv/tikv#6402

[warmchang]

Run with --unsafe-no-fsync flag? #11946

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.