Awesome Async Crypto + Less magic to 'run in the browser' #485

dignifiedquire · 2016-09-13T15:20:13Z

These are the changes that needed to bubble up from deep down in libp2p-crypto and multihashing.

This reduces the size of the browser build down to 2.4M and 1.2M unminified and minified respectively.

Associated PRs that need to be merged and shipped before this can get merged are

PR	Next Aegir	CR	Sauce Credentials	RFM	M
libp2p/js-libp2p-crypto#12	✔️	✔️	✔️	✔️	✔️
multiformats/js-multihashing-async#1	✔️	✔️	✔️	✔️	✔️
libp2p/js-peer-id#33	✔️	✔️	✔️	✔️	✔️
libp2p/js-peer-info#22	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-identify#6	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-switch#108	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-secio#19	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-webrtc-star#27	✔️	✔️	✔️	✔️	✔️
ipfs-inactive/js-libp2p-ipfs-nodejs#24	✔️	✔️	N/A	✔️	✔️
ipfs-inactive/js-libp2p-ipfs-browser#99	✔️	✔️	✔️	✔️	✔️
ipld/js-ipld-block#4	✔️	✔️	✔️	✔️	✔️
ipfs/js-ipfs-repo#88	✔️	✔️	✔️	✔️	✔️
ipfs-inactive/interface-js-ipfs-core#82	✔️	✔️	N/A	✔️	✔️
ipfs-inactive/js-ipfs-http-client#390	✔️	✔️	✔️	✔️	✔️
ipld/js-ipld-dag-pb#3	✔️	✔️	✔️	✔️	✔️
ipld/js-ipld-dag-cbor#28	✔️	✔️	✔️	✔️	✔️
ipfs/js-ipfs-unixfs#12	✔️	✔️	✔️	✔️	✔️
ipfs/js-ipfs-bitswap#41	✔️	✔️	✔️	✔️	✔️
multiformats/js-cid#5	✔️	✔️	✔️	✔️	✔️
ipfs/js-ipfs-block-service#35	✔️	✔️	✔️	✔️	✔️
libp2p/interface-connection#19	✔️	✔️	N/A	✔️	✔️
libp2p/js-libp2p-spdy#36	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-tcp#40	✔️	✔️	N/A	✔️	✔️
ipfs-inactive/js-ipfs-unixfs-engine#82	✔️	✔️	✔️	✔️	✔️
libp2p/js-libp2p-websockets#33	✔️	✔️	✔️	✔️	✔️
ipld/js-ipld#65	✔️	✔️	✔️	✔️	✔️
multiformats/js-multistream-select#25	✔️	✔️	✔️	✔️	✔️
#485	✔️	✔️	✔️	✔️	✔️
ipfs/aegir#66	✔️	✔️	N/A	✔️	✔️

TODOs

Update all PRs to use the new multihashing-async instead of multihashing
Use strings instead of buffers for protos, ref Async Crypto Endeavour js-ipfs-unixfs#12 (comment)
Release new aegir with changes from runtime support (Browser, Node.js versions) #536
Update to use new aegir
Rebase onto IPLD changes

daviddias · 2016-09-13T17:09:01Z

awesome! :D

remember: if > half a day :)

dignifiedquire · 2016-09-15T10:15:34Z

@diasdavid ready for review

daviddias · 2016-09-23T09:44:28Z

@dignifiedquire will defer my review until there are good benchmarks. Thank you for pushing this! :)

dignifiedquire · 2016-09-23T11:50:04Z

Doing more work on this, so not ready anymore ;)

dignifiedquire · 2016-09-29T09:07:36Z

@diasdavid tests are passing locally for node, so you can start reviewing :)

dignifiedquire · 2016-10-03T16:00:05Z

Added new todo

nikuda · 2016-10-03T19:34:28Z

Should ipfs-bitswap be on the above list too?

https://github.com/ipfs/js-ipfs-bitswap/tree/webcrypto

dignifiedquire · 2016-10-03T19:35:07Z

@nikuda yes it should :D

nikuda · 2016-10-03T20:08:41Z

Proto files buffer to string progress:

libp2p/js-libp2p-crypto Using string instead of buffer for proto file libp2p/js-libp2p-crypto#15
libp2p/js-libp2p-secio Using string instead of buffer for proto file libp2p/js-libp2p-secio#22
ipfs/js-ipfs-bitswap Using string instead of buffer for proto file js-ipfs-bitswap#42
ipfs/js-ipfs-merkle-dag Using string instead of buffer for proto file js-ipfs-merkle-dag#53
ipfs/js-ipfs-unixfs Using string instead of buffer for proto file js-ipfs-unixfs#13

dignifiedquire · 2016-10-04T08:44:54Z

@nikuda awesome, thank you

daviddias · 2016-10-05T13:11:30Z

From IRC

 <dignifiedquire> daviddias: good that you didn't review today :D I realised earlier that I hadn't tested the full down interop and had to fix some more issues, but now all is good and 💯 interop is back

Glad that I mentioned that several times :) Have you added CI tests to guarantee that interop remains? What were the tests you performed to verify?

dignifiedquire · 2016-10-05T13:51:53Z

Glad that I mentioned that several times :) Have you added CI tests to guarantee that interop remains? What were the tests you performed to verify?

This interop are the tests you wanted to create scripts for. I just start a go and js node manually and connect them manually.

There are a lot of tests to verify that we interop with go on libp2p-crypto already if you look at the tests there. But for the full integration test we need a full go-ipfs node from master that works

daviddias · 2016-10-05T14:40:18Z

@dignifiedquire reviewed a bunch of this PR. ipfs-block needs some changes that will cascade to to the remaining PRs that I haven't finalized reviewing (i.e removed the 'needs review label')

daviddias · 2016-10-24T13:04:59Z

Important to handle before moving forward with this PR - libp2p/js-libp2p-crypto#18

dignifiedquire · 2016-11-07T14:11:10Z

Now that the changes have happened we need to update https://github.com/ipfs/community/blob/master/js-project-guidelines.md accordingly

daviddias · 2016-11-10T12:03:42Z

Currently blocked by:

dignifiedquire · 2016-11-10T16:54:12Z

Unblocked with libp2p/js-libp2p-crypto#29

dignifiedquire · 2016-11-11T10:58:56Z

All dependencies were merged and released, doing the last updates now 🎉

dignifiedquire · 2016-11-11T14:45:50Z

@diasdavid can I get some last review on this?

daviddias · 2016-11-11T17:39:38Z

@dignifiedquire all CI is failing, is that what you are expecting?

daviddias · 2016-11-11T17:42:47Z

package.json

+    "libp2p-ipfs": "libp2p-ipfs-browser",
+    "./src/core/default-repo.js": "./src/core/default-repo-browser.js",
+    "./src/core/components/init-assets.js": false,
+    "fs-pull-blob-store": false,


Needs to be reviewed

need it as it is used in the tests, and does no harm otherwise

daviddias · 2016-11-11T17:43:28Z

src/cli/commands/block/get.js

        ? argv.key
-        : new Buffer(bs58.decode(argv.key))
+        : mh.fromB58String(argv.key)


Wait, does this still apply? we have standardised the block API

yes, tests will fail otherwise

daviddias · 2016-11-11T17:45:36Z

src/cli/commands/block/rm.js

-      const mh = new Buffer(bs58.decode(argv.key))
-
-      ipfs.block.del(mh, (err) => {
+      ipfs.block.del(mh.fromB58String(argv.key), (err) => {


block.del should support multihash as strings

that's exactly what I'm doing here, decoding the multihash from a string

daviddias · 2016-11-11T17:45:58Z

src/cli/commands/object/get.js

+      (cb) => utils.getIPFS(cb),
+      (ipfs, cb) => ipfs.object.get(argv.key, {enc: 'base58'}, cb),
+      (node, cb) => node.toJSON(cb)
+    ], (err, res) => {


s/res/nodeJSON

daviddias · 2016-11-11T17:46:54Z

src/cli/commands/object/new.js

+      (cb) => utils.getIPFS(cb),
+      (ipfs, cb) => ipfs.object.new(cb),
+      (node, cb) => node.toJSON(cb)
+    ], (err, node) => {


s/node/nodeJSON

daviddias · 2016-11-11T17:53:20Z

src/http-api/resources/object.js

+    waterfall([
+      (cb) => ipfs.object.get(key, cb),
+      (node, cb) => node.toJSON(cb)
+    ], (err, res) => {


s/res/nodeJSON

daviddias · 2016-11-11T17:53:40Z

src/http-api/resources/object.js

+    waterfall([
+      (cb) => ipfs.object.patch.setData(key, data, cb),
+      (node, cb) => node.toJSON(cb)
+    ], (err, res) => {


s/res/nodeJSON/g

daviddias · 2016-11-11T17:53:53Z

src/http-api/resources/object.js

+        },
+        (link, cb) => ipfs.object.patch.addLink(root, link, cb),
+        (node, cb) => node.toJSON(cb)
+      ], (err, res) => {


s/res/nodeJSON/g

daviddias · 2016-11-11T17:54:04Z

src/http-api/resources/object.js

+    waterfall([
+      (cb) => ipfs.object.patch.rmLink(root, link, cb),
+      (node, cb) => node.toJSON(cb)
+    ], (err, res) => {


s/res/nodeJSON

Seems overly verbose, and I really dislike all caps naming. Not sure about a better name

well, toJSON is also JSON all caps

because apis in JavaScript are note very consistent when it comes to naming :/

daviddias · 2016-11-11T17:54:28Z

test/core/both/index.js

+  require('./test-files')
+  require('./test-generic')
+  require('./test-init')
+  require('./test-object')


dignifiedquire · 2016-11-11T18:47:24Z

@dignifiedquire all CI is failing, is that what you are expecting?

yes, waiting for the bitswap PR I pinged you about

daviddias · 2016-11-11T19:17:35Z

@dignifiedquire there you go ipfs/js-ipfs-bitswap#67

dignifiedquire · 2016-11-11T22:00:12Z

CI is finally green, browsers are not perfect atm, most are failing due to one or the other issue

Firefox fails with this

Firefox 49.0.0 (Linux 0.0.0) ERROR
  SyntaxError: missing = in const declaration
  at webpack:///~/cbor/lib/encoder.js:383:0 <- test/browser.js:41233

though this code runs fine in Chrome.

@diasdavid if you are happy with the code please merge and ship at your earliest convenience, and I will investigate and fix various browser related issues next week.

daviddias · 2016-11-12T03:18:24Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

* chore(package): update aegir to version 19.0.3 Closes ipfs#480 * fix: appease linter License: MIT Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

dignifiedquire added the status/in-progress In progress label Sep 13, 2016

dignifiedquire changed the title ~~[WIP] refactor: use async crypto~~ refactor: use async crypto Sep 15, 2016

dignifiedquire force-pushed the fast-keys branch from b8c27be to fd22c82 Compare September 15, 2016 10:15

dignifiedquire added the needs review label Sep 15, 2016

dignifiedquire assigned daviddias Sep 15, 2016

dignifiedquire removed the needs review label Sep 23, 2016

dignifiedquire assigned dignifiedquire and unassigned daviddias Sep 23, 2016

dignifiedquire force-pushed the fast-keys branch from fd22c82 to 2d0c802 Compare September 26, 2016 14:24

dignifiedquire force-pushed the fast-keys branch from 8a2688e to d6a7d98 Compare September 30, 2016 14:40

daviddias mentioned this pull request Oct 3, 2016

Getting js-ipfs to load in browser #509

Closed

dignifiedquire changed the title ~~refactor: use async crypto~~ feat: webcrypto and improved browser compatability Oct 3, 2016

dignifiedquire added the needs review label Oct 3, 2016

dignifiedquire assigned daviddias and unassigned dignifiedquire Oct 3, 2016

daviddias changed the title ~~feat: webcrypto and improved browser compatability~~ feat: Move to Async Crypto + Less magic to 'run in the browser' Oct 12, 2016

kumavis mentioned this pull request Nov 7, 2016

libp2p mesh testing MetaMask/metamask-extension#719

Closed

36 tasks

daviddias and others added 2 commits November 11, 2016 10:58

chore: update deps

a1cd2ac

test: bring back bitswap

8d5b10a

dignifiedquire mentioned this pull request Nov 11, 2016

fix: better accounting and less logs ipfs/js-ipfs-bitswap#67

Merged

daviddias suggested changes Nov 11, 2016

View reviewed changes

CR cleanup

19589ca

dignifiedquire added 4 commits November 11, 2016 20:27

last cr fixes

67fbc0f

feat(http): better error messages

cd7f77d

feat(http): set default headers for browsers

6a21cd0

test: proper handling of browser tests

4aad341

daviddias merged commit 219ef2c into master Nov 12, 2016

daviddias deleted the fast-keys branch November 12, 2016 03:18

daviddias removed the status/in-progress In progress label Nov 12, 2016

This was referenced Nov 12, 2016

browser compatibility #586

Closed

Regression in 0.10.0: readTimed is not a function libp2p/js-libp2p-spdy#44

Closed

dignifiedquire mentioned this pull request Nov 14, 2016

Bundling js-ipfs and js-ipfs-api for the Browser #429

Closed

daviddias changed the title ~~Async Crypto + Less magic to 'run in the browser'~~ Awesome Async Crypto + Less magic to 'run in the browser' Nov 22, 2016

daviddias added the awesome endeavour label Jan 8, 2017

daviddias mentioned this pull request Jul 3, 2018

libp2p Requirements Doc - Roadmap to 1.0.0 libp2p/libp2p#39

Merged

Awesome Async Crypto + Less magic to 'run in the browser' #485

Awesome Async Crypto + Less magic to 'run in the browser' #485

Conversation

dignifiedquire commented Sep 13, 2016 • edited by daviddias Loading

TODOs

daviddias commented Sep 13, 2016

dignifiedquire commented Sep 15, 2016

daviddias commented Sep 23, 2016

dignifiedquire commented Sep 23, 2016

dignifiedquire commented Sep 29, 2016

dignifiedquire commented Oct 3, 2016

nikuda commented Oct 3, 2016

dignifiedquire commented Oct 3, 2016

nikuda commented Oct 3, 2016 • edited Loading

dignifiedquire commented Oct 4, 2016

daviddias commented Oct 5, 2016

dignifiedquire commented Oct 5, 2016

daviddias commented Oct 5, 2016

daviddias commented Oct 24, 2016

dignifiedquire commented Nov 7, 2016

daviddias commented Nov 10, 2016

dignifiedquire commented Nov 10, 2016

dignifiedquire commented Nov 11, 2016

dignifiedquire commented Nov 11, 2016

daviddias commented Nov 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dignifiedquire commented Nov 11, 2016

daviddias commented Nov 11, 2016

dignifiedquire commented Nov 11, 2016

daviddias commented Nov 12, 2016

dignifiedquire commented Sep 13, 2016 •

edited by daviddias

Loading

nikuda commented Oct 3, 2016 •

edited

Loading