Awesome WebWorkers Support

[dignifiedquire]

Thanks to the efforts from @dryajov and @tswindell we are close to having full support for running js-ipfs in webworkers. This issue tracks the last outstanding issues for this.

Aegir

• Merge webworkers support into aegir
• Release aegir with webworker test support

PRs

• "Support for WebWorker." Fixing missing self definition which fails linting libp2p/js-libp2p-crypto#61
• fix support for WebWorkers multiformats/js-multihashing-async#11 and Added WebWorker environment support. multiformats/js-multihashing-async#12

Others

• Enable webworker tests for all js-ipfs* repos
• Enable webworker tests for all js-libp2p* repos
• Enable webworker tests for all js-multi* repos

[dryajov]

@dignifiedquire @diasdavid I can take this on:

Enable webworker tests for all js-ipfs* repos
Enable webworker tests for all js-libp2p* repos
Enable webworker tests for all js-multi* repos

This is required to move forward with #127

[daviddias]

This is awesome! Thank you @dryajov!

Can we then make a lot of noise sure everyones voice is represented on the request to have WebRTC in WebWorkers?

• https://lists.w3.org/Archives/Public/public-webrtc/2015Oct/0000.html
• Add support for WebRTC Data Channel in Workers w3c/webrtc-pc#230
• https://bugzilla.mozilla.org/show_bug.cgi?id=922363
• https://bugs.chromium.org/p/chromium/issues/detail?id=302019
• https://groups.google.com/forum/#!topic/discuss-webrtc/ulHSVgHnnrk
• Make RTCDataChannels Transferable w3c/webrtc-pc#317

[dryajov]

Wanted to check in with where things are with this endeavor.

So far, I've managed to switch the use of window to self, and this seem to fix 90% of the issues. I've got most of it committed in my forks on github, if anyone is interested to check it out, please do. I'm not entirely sure about using self being the right way of doing it, and I'm not as familiar with webpack to know if there is a better approach (i'd rather a bundler taking care of context/env), but so far this has worked. @diasdavid @dignifiedquire @kumavis If you guys know of a better way please let me know; otherwise, I think we have a best practice suggestion to make to the community at large - ditch window in favor of self. 👅

So far, the biggest issue I ran into is with brorand, which is not assuming the webworker env correctly. I've got a PR coming that addresses it, and that fixes most of the issues with multihashes. But I'm not sure when thats going to be merged in, I might need help creating noise around it.

I will be creating PRs for each project shortly (today/tomorrow), so that everypbody can keep an eye on the progress and contribute where they see fit.

The rest of the issues are less trivial, and I will be going over them one by one and fixing as I go.

I'll keep everyone posted on progress.

Cheers!

[dryajov]

PR to address brorand - indutny/brorand#7

[kumavis]

i bet you could polyfill it

if (typeof window === 'undefined') self.window = self

[dryajov]

wouldn't that mess things up a bit inside a WW for other packages as well? As in incorrectly assuming that its running in a dom enabled env? Otherwise, thats a better approach.

[kumavis]

yeah thats a good point, its a bit hacky so beware

[dryajov]

The linked PRs should address the full webworker support, with this changes I'm able to run js-ipfs tests in webworkers.

[dignifiedquire]

Looks like brorand shipped yesterday with the fixes :)

[daviddias]

woooot! So now, we just need to enable webworker tests in all the repos? Let's make all of this on time for #756 \o/

[dignifiedquire]

@diasdavid all you have to do for that is upgrade to aegir@10 which runs them by default

[daviddias]

-> ipfs-inactive/js-ipfs-unixfs-engine#137 (comment)

[dryajov]

@diasdavid for projects that we want to skip webworker tests you can pass the --dom parameter to aegir.

[dryajov]

@diasdavid I would also merge and release this PRs before bumping aegir on the rest of the projects, because they would fail without those changes, I've seen multihashing-async being the culprit in most cases...

[daviddias]

why --dom? That doesn't seem like to be something obvious for what it does

[dryajov]

there are two flags now, --webworker which will run webworker only tests and --dom which will run tests in a dom enabled env™ only, I've no attachment to that flag whatsoever and open to hearing suggestions - naming is hard. 😛

[daviddias]

indeed :) let's do --webworker=only and --webworker=false for only webworker and except webworker respectively, being --webworker=true the default

[dryajov]

@diasdavid what project is that? This might be a project that needs WW disabled. As for them not failing, are you on the latest Aegir with those fixes applied?

[daviddias]

This is js-ipfs itself

[dryajov]

Ah, thats weird. I'll take a look at it, wonder if any changes have been introduced which could bring window in again? In any case will take a look ASAP.

[daviddias]

@dryajov still seeing this one, any developments?

[daviddias]

Since @dignifiedquire asked me to update aegir in all the repos (thanks dig!), I'm going to list out here which repos still don't have webworker support.

I've checked so far:

• multiformats
  • needs to move the eslint-disable-line by dirty-chai
• ipld
  • needs to move the eslint-disable-line by dirty-chai
• libp2p (missing: below)
  • https://github.com/libp2p/js-peer-book
  • https://github.com/libp2p/js-libp2p-tcp
  • https://github.com/libp2p/js-libp2p-websockets
  • https://github.com/libp2p/js-libp2p-webrtc-star
  • https://github.com/libp2p/js-libp2p-crypto-secp256k1
  • https://github.com/libp2p/js-libp2p-crypto
  • https://github.com/libp2p/js-libp2p-ping
  • https://github.com/libp2p/js-libp2p-multiplex
  • https://github.com/libp2p/js-libp2p-spdy
  • https://github.com/libp2p/js-libp2p-identify
  • https://github.com/libp2p/js-libp2p-swarm
  • https://github.com/libp2p/js-libp2p-secio
  • https://github.com/libp2p/js-libp2p
• ipfs (missing: all, except js-ipfs)

Need webworker support

• js-multihashing test: run tests on webworkers multiformats/js-multihashing#19
• js-ipfs https://github.com/ipfs/js-ipfs
• js-ipfs-block-service ipfs/js-ipfs-block-service@a54e113
• js-ipfs-bitswap Missing webworker support js-ipfs-bitswap#120

[dryajov]

@diasdavid, we also need this ipfs/aegir#107 in aegir. For now, we can just disable window as a global without any specific message, if you have a chance to update aegirs eslint with the rule, it might be a good time.

[dryajov]

@diasdavid webrtcsupport lib is the one causing issues, it is the last lib we need fixed before full WW support is finally ready. Here is a PR addressing it HenrikJoreteg/webrtcsupport#30.

[fippo]

@dryajov: i'd advise against using the webrtcsupport package. Its legacy and contains 90% stuff that you don't need anymore in 2017.

@diasdavid i would strongly advise against "making noise", in particular making the kind of noise that has happened on the worker topic in the past.. Trying to get someone else to do work that benefits you like this typically doesn't work out. If you want to be constructive you work together with the people who are able to do that work or do the work (including specifying how things should work) yourself.

[dryajov]

@fippo thanks for pointing this out. Indeed all we use it for is detection of the presence of webrtc support in the environment, perhaps don't need that in the browser anymore, but what about nodejs?

[daviddias]

@fippo I clearly didn't word that in the best way possible. Entirely agree with you, it is orders of magnitude more valuable to provide constructive help and willingness to be part of the process than a simple 'I want this too'. Currently, I'm underwater and unavailable to provide significant help to that development, however, I do hope in the best way possible, that by bringing other people that could also benefit from it, we can signal that there is value on making it happen.

re: webrtcsupport - What is your favourite nowadays?

[fippo]

@diasdavid at least you didn't tweet about it like certain other people. If you just want to check for support RTCPeerConnection && ('createDataChannel' in RTCPeerConnection.prototype) should be sufficient.

[dryajov]

@diasdavid I'll do what @fippo is suggesting.

[kumavis]

@dignifiedquire found this recently https://github.com/Joris-van-der-Wel/karma-mocha-webworker
may be of interest

[dryajov]

@kumavis thats what we ended up using.

To get the last mile of this, we need a pretty small change in how we detect WebRTC. Looking up that issue...

ah... its right in this thread:

#725 (comment)

[IamCarbonMan]

As an outside observer, I have to ask: what does all this mean? Are you actually able to use WebRTC APIs in worker code as has been proposed as standard here? Or is this issue just to test that the code should theoretically work the same from a worker context, to be ready when worker WebRTC is standardized?

[dryajov]

Good question, glad you ask - this effort is to get IPFS to run in Web and Service workers. The specific outstanding issue right now is related to WebRTC detection functionality that breaks inside Workers because of explicit usage of window. WebRTC is tangential to this effort, since its not supported in web/service workers at all at this point.

[daviddias]

@IamCarbonMan you are not able to, but libp2p dark magic uses whatever the runtime has to offer it, it won't use WebRTC if there is no support but the issue is that we are checking the support by checking the window object which is also not defined.

[daviddias]

WebWorker tests are now passing! :D woot!

#857

[daviddias]

fully supported in v0.24.0