Use Key-value DB for DataStore and PinStore #439

fetchadd · 2021-01-05T01:20:04Z

This PR is for using key-value db like sled as DataStore and PinSotre, so that it is more efficient for pinning and easy to realize MFS.

The FsDatastore is inefficiency now, when checking a indirect pinned cid, all file in the FsDatastore will be visited and read, what more important is that it is hard to distinguish the data for pinning and the data for DataStore. The key-value db like sled will bring convenience and efficiency in the next work with its high efficiency and easy key/value operation. Leveldb may be the first choice of key-value dbs, but now there are no suitable rust leveldb crates; Sled is fast and simple, and it is usable on servers and phones from any C-compatible language.

This PR is the reopen of PR434. For there is some verbose commits like "replaceing leveldb with sled", the master branch of fetchadd/rust-ipfs in the PR434 is deprecated. Now the same changes are applied to the staging branch which is synced with the master of rust-ipfs/rust-ipfs. So Thanks for the PR of fetchadd/rust-ipfs from @koivunej which uses mutex to work around init(&self) and unsafe, but as the master is deprecated, I can only copy the code to staging branch.

Accorrding to the review from @ koivunej in PR434, the pinstore_interface_tests is added to KvDataStore, and now all tests are passed.

Closes #434.

koivunej

This is looking great, thank you. I might be adding a few stylistic tweaks, changelog and comments and will be cc'ing you in a follow-up PR.

Closes #434. Nope this didn't work, adding it to the description.

Awaiting a sec @ljedrz wants to review as well..

src/repo/kv.rs

ljedrz · 2021-01-05T12:50:52Z

src/repo/kv.rs

+        let db = kv_db.get_db();
+
+        match db.get(key.as_str()) {
+            Ok(Some(_)) => return Ok(Some(mode.clone())),


note: PinMode isn't Copy, but it should be

very much agreed.

src/repo/kv.rs

src/lib.rs

ljedrz

LGTM 👍

koivunej · 2021-01-05T15:39:14Z

Thank you very much @fetchadd!

bors r+

bors · 2021-01-05T15:56:12Z

Build succeeded:

ci

koivunej · 2021-01-05T19:52:39Z

@fetchadd re: my review comments; I found a few things to fix, missing tests but I'll post a draft on thursday on my progress.

feat: Using sled as DataStore and PinStore

81f2f24

fetchadd mentioned this pull request Jan 5, 2021

Use Key-value DB for DataStore and PinStore #434

Closed

koivunej approved these changes Jan 5, 2021

View reviewed changes