feature-request - add appendFile capability to overcome nedb performance-bottleneck

[kaizhu256]

nedb emulates appendFile by re-saving its entire database as a blob to indexedb whenever a row insert/delete/update is performed [1]. this performance-hit makes it impractical for persisting largish datasets (> 20mb).

adding some kind of incremental appendFile feature would help.

[1] https://github.com/louischatriot/nedb/blob/v1.6.0/browser-version/browser-specific/lib/storage.js#L49

[kaizhu256]

this feature-request originated from tc39/proposal-built-in-modules#16 (comment)

[inexorabletash]

There are plans for a rebooted FileSystem API with both native and sandboxes support, which is probably a better fit, with the addition of locking primitives.

[pwnall]

Typical databases have some sort of block cache, and need to write modified blocks to disk. The sub-system is sometimes called the page cache or the pager.

If nedb has something similar, it should use a Blob for each block / page, instead of writing the entire database in a single Blob. Each block would be a Blob in an object store keyed by block ID. You can use this approach to build a database or a filesystem on top of IndexedDB. There's still overhead, but at least it's constant, not O(database size).

If you need something working soon, the approach above should give you some relief until we're able to build and ship the future APIs mentioned by @inexorabletash. I hope this helps!

[vweevers]

Off-topic: where can I follow progress on that rebooted FileSystem API? Thanks!

[inexorabletash]

https://github.com/WICG/native-file-system

[inexorabletash]

TPAC 2019 Web Apps Indexed DB triage notes:

We agree with advice in #260 (comment) - implementations could also optimize blob storage using e.g. ropes.

There are explorations by Google Chrome of WASM-friendly storage APIs happening, which would allow running sqlite in the browser directly. Will add links if we can find them. And the people working on native-file-system would probably appreciate feedback too.

Mutable files doesn't seem to align well with the rest of Indexed DB and given the other possible approaches above, closing this out.

[kaizhu256]

cool, interested in links on sqlite3 when available.