feat: next level of the datastore

.flowconfig

@@ -0,0 +1,11 @@
+[ignore]
+.*/radium/.*
+.*/standard-changelog/.*
+.*/conventional-changelog-core/.*
+
+[include]
+
+[libs]
+flow-typed/.*
+
+[options]

.gitignore

@@ -0,0 +1,35 @@
+**/node_modules/
+**/*.log
+test/repo-tests*
+
+# Logs
+logs
+*.log
+
+coverage
+
+# Runtime data
+pids
+*.pid
+*.seed
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# node-waf configuration
+.lock-wscript
+
+build
+
+# Dependency directory
+# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
+node_modules
+
+lib
+dist

.travis.yml

@@ -0,0 +1,30 @@
+sudo: false
+language: node_js
+
+matrix:
+  include:
+  - node_js: 4
+    env: CXX=g++-4.8
+  - node_js: 6
+    env:
+      - SAUCE=true
+      - CXX=g++-4.8
+  - node_js: stable
+    env: CXX=g++-4.8
+
+script:
+  - npm run lint
+  - npm run flow
+  - npm test
+
+before_script:
+  - export DISPLAY=:99.0
+  - sh -e /etc/init.d/xvfb start
+
+addons:
+  firefox: 'latest'
+  apt:
+    sources:
+      - ubuntu-toolchain-r-test
+    packages:
+      - g++-4.8

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 IPFS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,92 +1,228 @@
-# node-datastore interface
+# interface-datastore
 
-datastore is a generic layer of abstraction for data store and database access. It is a simple API with the aim to enable application development in a datastore-agnostic way, allowing datastores to be swapped seamlessly without changing application code. Thus, one can leverage different datastores with different strengths without committing the application to one datastore throughout its lifetime.
+[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)
+[![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/)
+[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)
+[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)
+[![Build Status](https://travis-ci.org/ipfs/interface-datastore.svg)](https://travis-ci.org/ipfs/interface-datastore) [![Circle CI](https://circleci.com/gh/ipfs/interface-datastore.svg?style=svg)](https://circleci.com/gh/ipfs/interface-datastore)
+[![Coverage Status](https://coveralls.io/repos/github/ipfs/interface-datastore/badge.svg?branch=master)](https://coveralls.io/github/ipfs/interface-datastore?branch=master) [![Dependency Status](https://david-dm.org/diasdavid/js-peer-id.svg?style=flat-square)](https://david-dm.org/ipfs/interface-datastore)
+[![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/feross/standard)
+![](https://img.shields.io/badge/npm-%3E%3D3.0.0-orange.svg?style=flat-square)
+![](https://img.shields.io/badge/Node.js-%3E%3D4.0.0-orange.svg?style=flat-square)
 
-In addition, grouped datastores significantly simplify interesting data access patterns (such as caching and sharding).
+> Implementation of the [datastore](https://github.com/ipfs/go-datastore) interface in JavaScript
 
-Based on [datastore.py](https://github.com/jbenet/datastore).
 
-Note: this is similar to [rvagg/abstract-leveldown](https://github.com/rvagg/abstract-leveldown/). Though I wrote [my original datastore](https://github.com/jbenet/datastore) many years ago. :)
+## Table of Contents
 
-## Example
+- [Implementations](#implementations)
+- [Install](#install)
+- [Usage](#usage)
+- [Api](#api)
+- [Contribute](#contribute)
+- [License](#license)
 
-### Usage
+## Implementations
 
-See [datastore.memory/try.js](https://github.com/jbenet/node-datastore.memory/blob/master/try.js):
+- Backed Implementations
+  - Memory: [`src/memory`](src/memory.js)
+  - level: [`datastore-level`](https://github.com/ipfs/js-datastore-level) (supports any levelup compatible backend)
+  - File System: [`datstore-fs`](https://github.com/ipfs/js-datastore-fs)
+- Wrapper Implementations
+  - Mount: [`datastore-core/src/mount`](https://github.com/ipfs/js-datastore-core/tree/master/src/mount.js)
+  - Keytransform: [`datstore-core/src/keytransform`](https://github.com/ipfs/js-datastore-core/tree/master/src/keytransform.js)
+  - Sharding: [`datastore-core/src/sharding`](https://github.com/ipfs/js-datastore-core/tree/master/src/sharding.js)
+  - Tiered: [`datstore-core/src/tiered`](https://github.com/ipfs/js-datastore-core/tree/master/src/tirered.js)
+  - Namespace: [`datastore-core/src/namespace`](https://github.com/ipfs/js-datastore-core/tree/master/src/namespace.js)
+
+If you want the same functionality as [go-ds-flatfs](https://github.com/ipfs/go-ds-flatfs), use sharding with fs.
 
 ```js
-var memDS = require('datastore.memory')
-ds.put('foo', 'bar', function(err, val, key) {
-  if (err) throw err
-  console.log('put ' + key + ': ' + val)
-  assert(val === 'bar')
+const FsStore = require('datastore-fs)
+const ShardingStore = require('datastore-core').ShardingDatatstore
+const NextToLast = require('datastore-core').shard.NextToLast
+
+const fs = new FsStore('path/to/store')
+ShardingStore.createOrOpen(fs, new NextToLast(2), (err, flatfs) => {
+  // flatfs now works like go-flatfs
 })
+```
+
+## Install
+
+```
+$ npm install interface-datastore
+```
+
+## Usage
 
-ds.has('foo', function(err, has, key) {
-  if (err) throw err
-  console.log(key + ' exists? ' + has)
-  assert(has === true)
+### Wrapping Stores
+
+```js
+const MemoryStore = require('interface-datastore').MemoryDatastore
+const MountStore = require('datastore-core').MountDatastore
+const Key = require('interface-datastore').Key
+
+const store = new MountStore({prefix: new Key('/a'), datastore: new MemoryStore()})
+```
+
+### Testsuite
+
+Available under [`src/tests.js`](src/tests.js)
+
+```js
+describe('mystore', () => {
+  require('interface-datastore/src/tests)({
+    setup (callback) {
+      callback(null, instanceOfMyStore)
+    },
+    teardown (callback) {
+      // cleanup resources
+      callback()
+    }
+  })
 })
+```
+
+## API
+
+### Keys
+
+To allow a better abstraction on how to address values, there is a `Key` class which is used as identifier. It's easy to create a key from a `Buffer` or a `string`.
+
+```js
+const a = new Key('a')
+const b = new Key(new Buffer('hello'))
+```
+
+The key scheme is inspired by file systems and Google App Engine key model. Keys are meant to be unique across a system. They are typical hierarchical, incorporating more and more specific namespaces. Thus keys can be deemed 'children' or 'ancestors' of other keys:
+
+- `new Key('/Comedy')`
+- `new Key('/Comedy/MontyPython')`
+
+Also, every namespace can be parametrized to embed relevant object information. For example, the Key `name` (most specific namespace) could include the object type:
+
+- `new Key('/Comedy/MontyPython/Actor:JohnCleese')`
+- `new Key('/Comedy/MontyPython/Sketch:CheeseShop')`
+- `new Key('/Comedy/MontyPython/Sketch:CheeseShop/Character:Mousebender')`
+
+
+### Methods
+
+> The exact types can be found in [`src/index.js`](src/index.js).
+
+These methods will be present on every datastore. `Key` always means an instance of the above mentioned Key type. Every datastore is generic over the `Value` type, though currently all backing implementations are implemented only for [`Buffer`](https://nodejs.org/docs/latest/api/buffer.html).
 
-ds.get('foo', function(err, val, key) {
-  if (err) throw err
-  console.log('get ' + key + ': ' + val)
-  assert(val === 'bar')
+### `put(Key, Value, (err: ?Error) => void): void`
+
+Store a value with the given key.
+
+```js
+store.put(new Key('awesome'), new Buffer('datastores'), (err) => {
+  if (err) {
+    throw err
+  }
+  console.log('put content')
 })
+```
 
-ds.delete('foo', function(err, key) {
-  if (err) throw err
-  console.log(key + ' deleted')
+### `get(Key, (err: ?Error, val: ?Value) => void): void`
+
+Retrieve the value stored under the given key.
+
+```js
+store.get(new Key('awesome'), (err, value) => {
+  if (err) {
+    throw err
+  }
+  console.log(got content: %s', value.toString())
+  // => got content: datastore
 })
+```
 
-ds.has('foo', function(err, has, key) {
-  if (err) throw err
-  console.log(key + ' exists? ' + has)
-  assert(has === false)
+### `delete(Key, (err: ?Error) => void): void`
+
+Delete the content stored under the given key.
+
+```js
+store.delete(new Key('awesome'), (err) => {
+  if (err) {
+    throw err
+  }
+  console.log(deleted awesome content :(')
 })
 ```
 
-### Implementation
+### `query(Query<Value>): QueryResult<Value>)`
 
-See [datastore.memory/index.js](https://github.com/jbenet/node-datastore.memory/blob/master/index.js):
+Search the store for some values. Returns a [pull-stream](https://pull-stream.github.io/) with each item being a `Value`.
 
 ```js
-var DS = require('datastore.abstract')
+// retrieve __all__ values from the store
+pull(
+  store.query({}),
+  pull.collect((err, list) => {
+    if (err) {
+      console.error(err)
+    }
+    console.log('ALL THE VALUES', list)
+  })
+)
+```
 
-module.exports = MemDS
+#### `Query`
 
-function MemDS() {
-  if (!(this instanceof MemDS))
-    return new MemDS
-  DS.call(this)
-  this.values = {}
-}
+Object in the form with the following optional properties
 
-DS.inherits(MemDS)
+- `prefix?: string`
+- `filters?: Array<Filter<Value>>`
+- `orders?: Array<Order<Value>>`
+- `limit?: number`
+- `offset?: number`
+- `keysOnly?: bool`
 
-MemDS.prototype._get = function(key, cb) {
-  var val = this.values[key.toString()]
-  if (val !== undefined) cb(null, val, key)
-  else cb(MemDS.errors.NotFound, null, key)
-}
+### batch(): Batch<Value>
 
-MemDS.prototype._put = function(key, val, cb) {
-  this.values[key.toString()] = val
-  cb(null, val, key)
-}
+This will return an object with which you can chain multiple operations together, with them only being executed on calling `commit`.
 
-MemDS.prototype._delete = function(key, cb) {
-  delete this.values[key.toString()]
-  cb(null, key)
-}
+```js
+const b = store.batch()
 
-MemDS.prototype._has = function(key, cb) {
-  var has = (this.values[key.toString()] !== undefined)
-  cb(null, has, key)
+for (let i = 0; i < 100; i++) {
+  b.put(new Key(`hello${i}`), new Buffer(`hello world ${i}`))
 }
+
+b.commit((err) => {
+  if (err) {
+    throw err
+  }
+  console.log(put 100 values')
+})
+
 ```
 
+#### `put(Key, Value): void`
+
+Queue a put operation to the store.
+
+#### `delete(Key): void`
+
+Queue a delete operation to the store.
+
+#### `commit((err: ?Error) => void): void`
+
+Write all queued operations to the underyling store. The batch object should not be used after calling this.
+
+### `close((err: ?Error) => void): void`
+
+Close the datastore, this should always be called to ensure resources are cleaned up.
+
+## Contribute
+
+PRs accepted.
+
+Small note: If editing the Readme, please conform to the [standard-readme](https://github.com/RichardLitt/standard-readme) specification.
 
 ## License
 
-MIT
+MIT 2017 © IPFS

circle.yml

@@ -0,0 +1,12 @@
+machine:
+  node:
+    version: stable
+
+dependencies:
+  pre:
+    - google-chrome --version
+    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
+    - sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
+    - sudo apt-get update
+    - sudo apt-get --only-upgrade install google-chrome-stable
+    - google-chrome --version