Skip to content

Commit

Permalink
update level to 7.0.1
Browse files Browse the repository at this point in the history
  • Loading branch information
@tegefaulkes
tegefaulkes committed Nov 19, 2021
1 parent a6d005b commit 6107d9c
Show file tree
Hide file tree
Showing 2 changed files with 125 additions and 95 deletions.
218 changes: 124 additions & 94 deletions package-lock.json
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"@matrixai/workers": "^1.2.3",
"abstract-leveldown": "^7.0.0",
"async-mutex": "^0.3.1",
"level": "^6.0.1",
"level": "7.0.1",

This comment has been minimized.

Sign in to view

Copy link
@kriszyp

kriszyp May 10, 2022

Have you considered using lmdb-js? It should be significantly faster (and has more powerful transaction capabilities).

This comment has been minimized.

Sign in to view

Copy link
@CMCDragonkai

CMCDragonkai May 10, 2022 •
edited
Loading

Member

I wasn't aware of it until now. Some questions (I just read the readme):

  1. What kind of transaction concurrency does it support? We are looking for snapshot-isolation capable transactions. Where the get and any range queries like iterators operate within that transaction, and writes are overlaid on top, where a write conflict can trigger a conflict exception.
  2. It appears to support encryption natively, does this encrypt keys and values, or just values? If it encrypts keys, how can indexes be built? Or does the encryption occur at the block-level underneath the key-value interface?
  3. What is its cross-platform capability? Can it also run in Android and iOS and Electron?

This comment has been minimized.

Sign in to view

Copy link
@CMCDragonkai

CMCDragonkai May 10, 2022

Member

We also use vercel/pkg for packaging and noticed kriszyp/lmdb-js#167

This comment has been minimized.

Sign in to view

Copy link
@kriszyp

kriszyp May 10, 2022

What kind of transaction concurrency does it support?

LMDB supports write transactions that fully block other write transactions, and read transactions that don't block each other or other write transactions. All transactions have full snapshot isolation (so you can start a read transaction that reads and iterates a snapshot). I think I will refer you to http://www.lmdb.tech/doc/index.html for more info on LMDB transactions. lmdb-js works hard to ensure writes are batched into transactions to mitigate any bottlenecks from the exclusive writer locking that LMDB uses (and in practice this works very well)

where a write conflict can trigger a conflict exception.

Since write locks are exclusive, there is no inherent write conflicts, however lmdb-js support "optimistic-locking" where you can specify an expected value/version as a precondition for a queued write. You can also write your own transaction sequences that perform any checks you want to detect a conflict.

It appears to support encryption natively, does this encrypt keys and values, or just values?

Encryption is block level (so I think that answers your other questions; keys and values both encrypted).

What is its cross-platform capability? Can it also run in Android and iOS and Electron?

Well, that's why we use node-gyp-build and ran into each other :). lmdb-js should run on all node and electron platforms. However, I didn't think that included Android and iOS?

We also use vercel/pkg for packaging and noticed kriszyp/lmdb-js#167

The OP had deprioritized this, but I will take another look at it, knowing there is interest.

This comment has been minimized.

Sign in to view

Copy link
@CMCDragonkai

CMCDragonkai May 10, 2022 •
edited
Loading

Member

Since write locks are exclusive, there is no inherent write conflicts, however lmdb-js support "optimistic-locking" where you can specify an expected value/version as a precondition for a queued write. You can also write your own transaction sequences that perform any checks you want to detect a conflict.

Yea after a lot of research and trial and error we came to the conclusion that our app Polykey would benefit the most from having snapshot-isolation optimistic concurrency control where we expect to get a DB conflict exception thrown if the write-set conflicts between transactions. We even looked into Serializable Snapshot Isolation but that's for later. And since leveldb JS bindings don't support it we have to fork off classic-level and bring that feature from the C++ into this repo.

That version precondition seems similar to how SI normally detects write-set conflicts. I'm not sure how transaction write-set conflict detection could be implemented on top of version preconditions though.

Encryption is block level (so I think that answers your other questions; keys and values both encrypted).

This is actually really great, we would love to have this, and since we started with leveldb, we haven't been able to get this feature in yet: #5. Is this done in the lmdb C++ layer? If so, we would need to see how cross-platform it can be atm.

Well, that's why we use node-gyp-build and ran into each other :). lmdb-js should run on all node and electron platforms. However, I didn't think that included Android and iOS?

Yea that is the case, android and ios compilation would require more cross-compilation infrastructure like how it is done in classic-level https://github.com/Level/classic-level.

Overall lmdb-js looks like a great project, if only we had started working off that than leveldb, maybe some original problems could have been solved. I might add it to our subsequent roadmap to explore whether it can replace the leveldb we are using.

This comment has been minimized.

Sign in to view

Copy link
@kriszyp

kriszyp May 10, 2022

Is this done in the lmdb C++ layer?

Yes, this is done at the C/C++ level.

If so, we would need to see how cross-platform it can be atm.

If there are some other prebuild architectures of value, I could probably add them (currently doing x64, arm64, arm7).
"levelup": "^5.0.1",
"sublevel-prefixer": "^1.0.0",
"subleveldown": "^5.0.1",
Expand Down

0 comments on commit 6107d9c

Please sign in to comment.