feat: ipns over pubsub #1559

vasco-santos · 2018-09-13T08:54:04Z

This PR addresses the IPNS over Pubsub feature.

It needs the following PRs:

js-ipfs-api#846
interface-ipfs-core#361
interface-ipfs-core#395
js-datastore-pubsub#6
enable interface-ipfs-core tests for ipns-pubsub(after interface-ipfs-core new release)
ipfs/js-datastore-pubsub#9

Unblocks:

interop tests ipfs/interop#39

vasco-santos · 2018-09-17T17:29:58Z

@diasdavid regarding the weekly sync question:

js-datastore-pubsub is used as middleware for pubsub to be used as a datastore with TieredDatastore. This way, we can combine multiple put and get in the IPNS land transparently with a "single" operation. Perhaps the module name is not the best.

We provide to the IPNS the routing, as a TieredDatastore composed by the OfflineDatastore and the PubsubDatastore. (NOTE: It now receives the OfflineDatastore, but this will be replaced with the DHT, once it gets ready). After that, when an IPNS operation happens, such as a publish, the IPNS just puts it to the routing.

All in all, pubsub-datastore receives a datastore (repo's datastore in this case), as well as a pubsub implementation. It provides the same API as a general store (put, get, ...) and translates those to what they really represent in the pubsub world (publish and subscribe). After that, it also takes care of the subscriptions handling and their consequent store in the node's repo datastore.

Moreover, go-team uses a similar approach with go-libp2p-pubsub-router, but their module is specific for IPNS, while pubsub-datastore is agnostic to where it is used. In addition, they also use the same logic for the TieredDatastore here.

I hope to have been clear with this explanation. Let me know if you agree, or if you have any other direction that this sould go.

src/core/ipns/publisher.js

dirkmc · 2018-10-16T18:38:48Z

This is a really nice approach 👍

alanshaw

Please note ipfs/js-datastore-pubsub#5

src/core/components/name-pubsub.js

src/core/components/name.js

src/core/ipns/path.js

README.md

src/core/ipns/routing/pubsub.js

test/cli/name-pubsub.js

src/core/components/name-pubsub.js

hugomrdias · 2018-11-30T11:04:36Z

src/core/components/start.js

@@ -2,11 +2,13 @@

 const series = require('async/series')
 const Bitswap = require('ipfs-bitswap')
+const get = require('lodash/get')


use smaller package
https://bundlephobia.com/result?p=lodash.get@4.4.2
vs
https://bundlephobia.com/result?p=dlv@1.1.2

Suggested change

const get = require('lodash/get')

const get = require('dlv')

Cool, thanks! Will add dlv to package.json as well.

I'm fine with using dlv over loadash.get but all other occurances of lodash.get need to be replaced as well otherwise we're just adding more to our bundle.

If you agree, I can create other PR with all those occurrences changed to dlv.

hugomrdias · 2018-11-30T11:06:40Z

src/core/config.js

@@ -28,6 +28,7 @@ const schema = Joi.object().keys({
  }).allow(null),
  EXPERIMENTAL: Joi.object().keys({
    pubsub: Joi.boolean(),
+    namesysPubsub: Joi.boolean(),


just pick one and use it everywhere

src/core/ipns/routing/offline-datastore.js

src/core/ipns/routing/pubsub-datastore.js

Co-Authored-By: vasco-santos <vasco.santos@ua.pt>

src/core/components/start.js

src/core/components/pubsub.js

alanshaw · 2018-12-03T13:39:20Z

src/core/ipns/routing/offline-datastore.js

@@ -48,7 +49,10 @@ class OfflineDatastore {
      return callback(errcode(new Error(errMsg), 'ERR_GENERATING_ROUTING_KEY'))
    }

-    this._repo.datastore.put(routingKey, value, callback)
+    // Marshal to libp2p record as the DHT does
+    let record = new Record(key, value)


Can be const

Should we open an issue in aegir to target this in the lint?

src/core/ipns/routing/pubsub-datastore.js

src/http/index.js

daviddias · 2018-12-03T14:50:31Z

src/core/index.js

+      if (!this._options.EXPERIMENTAL.pubsub) {
+        this.log('EXPERIMENTAL pubsub is enabled to use IPNS pubsub')
+        this._options.EXPERIMENTAL.pubsub = true
+      }


Does go-ipfs do IPNS over PubSub as an experimental flag? Why not just one flag that enables all of PubSub

Yes, go-ipfs uses an experimental flag for IPNS over Pubsub: --enable-namesys-pubsub is the CLI flag

"Experimental features on Alpha software", how much more bleeding edge can you go 😅

alanshaw · 2018-12-03T19:06:29Z

@vasco-santos released interface-ipfs-core@0.89 with ipfs-inactive/interface-js-ipfs-core#395

vasco-santos · 2018-12-03T19:19:32Z

Thanks! Updated the package.json with it

vasco-santos requested a review from alanshaw September 13, 2018 08:54

ghost assigned vasco-santos Sep 13, 2018

ghost added the status/in-progress In progress label Sep 13, 2018

vasco-santos changed the title ~~feat: ipns over pubsub~~ [WIP] feat: ipns over pubsub Sep 13, 2018

vasco-santos force-pushed the feat/ipns-over-pubsub branch 4 times, most recently from 0a2d8ca to 3c90cbf Compare September 14, 2018 09:46

vasco-santos force-pushed the feat/ipns-over-pubsub branch from 3c90cbf to 61a44fb Compare October 3, 2018 17:15

dirkmc reviewed Oct 16, 2018

View reviewed changes

src/core/ipns/publisher.js Outdated Show resolved Hide resolved

vasco-santos force-pushed the feat/ipns-over-pubsub branch from 61a44fb to 0627b06 Compare October 17, 2018 16:25

alanshaw suggested changes Oct 18, 2018

View reviewed changes

alanshaw mentioned this pull request Oct 19, 2018

Resolve ipns on browser side #1644

Closed

vasco-santos force-pushed the feat/ipns-over-pubsub branch 7 times, most recently from 84d851c to ba944df Compare October 27, 2018 22:26

vasco-santos changed the title ~~[WIP] feat: ipns over pubsub~~ feat: ipns over pubsub Oct 28, 2018

vasco-santos added the status/blocked Unable to be worked further until needs are met label Oct 28, 2018

vasco-santos force-pushed the feat/ipns-over-pubsub branch 3 times, most recently from aac7a52 to 83fee00 Compare November 7, 2018 10:35

vasco-santos mentioned this pull request Nov 7, 2018

refactor: ipns routing logic moved to instantiation #1701

Merged

vasco-santos mentioned this pull request Nov 14, 2018

Implement IPNS #209

Closed

vasco-santos force-pushed the feat/ipns-over-pubsub branch from 83fee00 to 35e6295 Compare November 15, 2018 18:35

vasco-santos force-pushed the feat/ipns-over-pubsub branch 3 times, most recently from 196dd4c to 03c24e1 Compare November 29, 2018 12:37

vasco-santos requested review from hugomrdias and achingbrain November 29, 2018 14:00

vasco-santos force-pushed the feat/ipns-over-pubsub branch from 03c24e1 to 7f07aa2 Compare November 29, 2018 21:48

hugomrdias suggested changes Nov 30, 2018

View reviewed changes

src/core/components/name-pubsub.js Show resolved Hide resolved

src/core/components/name-pubsub.js Outdated Show resolved Hide resolved

hugomrdias suggested changes Nov 30, 2018

View reviewed changes

vasco-santos force-pushed the feat/ipns-over-pubsub branch 3 times, most recently from fb48ba9 to 6f83aa1 Compare November 30, 2018 15:18

hugomrdias approved these changes Nov 30, 2018

View reviewed changes

vasco-santos and others added 7 commits December 1, 2018 14:33

feat: ipns over pubsub

310c379

fix: updated according to the modification on the api

4c40fc1

fix: first pass of code review

66907b5

fix: second pass of code review

3a4b139

fix: code review for tests

9bb8816

fix: remove libp2p-record for pubsub

62f25a6

fix: code review

b2755a0

Co-Authored-By: vasco-santos <vasco.santos@ua.pt>

vasco-santos force-pushed the feat/ipns-over-pubsub branch from 6f83aa1 to b2755a0 Compare December 1, 2018 14:34

alanshaw suggested changes Dec 3, 2018

View reviewed changes

fix: code review

7edade9

daviddias reviewed Dec 3, 2018

View reviewed changes

chore: upgrade interface-ipfs-core

fa5b652

alanshaw approved these changes Dec 4, 2018

View reviewed changes

alanshaw merged commit 8712542 into master Dec 4, 2018

ghost removed the status/in-progress In progress label Dec 4, 2018

alanshaw deleted the feat/ipns-over-pubsub branch December 4, 2018 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ipns over pubsub #1559

feat: ipns over pubsub #1559

vasco-santos commented Sep 13, 2018 •

edited by alanshaw

Loading

vasco-santos commented Sep 17, 2018 •

edited

Loading

dirkmc commented Oct 16, 2018

alanshaw left a comment

hugomrdias Nov 30, 2018

vasco-santos Nov 30, 2018

alanshaw Dec 3, 2018

vasco-santos Dec 3, 2018

hugomrdias Nov 30, 2018

alanshaw Dec 3, 2018

vasco-santos Dec 3, 2018

daviddias Dec 3, 2018

vasco-santos Dec 3, 2018 •

edited

Loading

daviddias Dec 3, 2018 •

edited

Loading

alanshaw commented Dec 3, 2018

vasco-santos commented Dec 3, 2018

feat: ipns over pubsub #1559

feat: ipns over pubsub #1559

Conversation

vasco-santos commented Sep 13, 2018 • edited by alanshaw Loading

vasco-santos commented Sep 17, 2018 • edited Loading

dirkmc commented Oct 16, 2018

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos Dec 3, 2018 • edited Loading

Choose a reason for hiding this comment

daviddias Dec 3, 2018 • edited Loading

Choose a reason for hiding this comment

alanshaw commented Dec 3, 2018

vasco-santos commented Dec 3, 2018

vasco-santos commented Sep 13, 2018 •

edited by alanshaw

Loading

vasco-santos commented Sep 17, 2018 •

edited

Loading

vasco-santos Dec 3, 2018 •

edited

Loading

daviddias Dec 3, 2018 •

edited

Loading