Skip to content
This repository has been archived by the owner on Mar 10, 2020. It is now read-only.

Bundling js-ipfs-api #353

Closed
haadcode opened this issue Aug 19, 2016 · 20 comments
Closed

Bundling js-ipfs-api #353

haadcode opened this issue Aug 19, 2016 · 20 comments

Comments

@haadcode
Copy link
Contributor

haadcode commented Aug 19, 2016 •
edited
Loading

We should explore the options and solutions to reducing the size of the distributed build of js-ipfs-api (dist/index.min.js).

Originally started here: ipfs/js-ipfs#398.

As it stands today, the dist file is at ~900kb. How far can we go? :)
@haadcode
Copy link
Contributor Author

ConsenSys is using https://github.com/pelle/browser-ipfs for their IPFS wrapper. Their reason is simply the size: 900kb (js-ipfs-api) vs. 2kb (browser-ipfs).

@diasdavid @dignifiedquire this begs the question: why is js-ipfs-api so large? What are the dependencies that make it so large? If js-ipfs-api can't use less dependencies, should we talk about a light api lib, much like browser-ipfs?

@haadcode
Copy link
Contributor Author

@diasdavid said:

@haadcode Thank you for bringing that up. It should go, however, into a js-ipfs-api issue, we will probably get a lot of savings from the dedup on js-ipfs-api.

This makes me thing that it would be pretty dope if js-ipfs-api was modular inside like async, so that devs can do require('ipfs-api/cat') and get just the bit they need.

@haadcode
Copy link
Contributor Author

This makes me thing that it would be pretty dope if js-ipfs-api was modular inside like async, so that devs can do require('ipfs-api/cat') and get just the bit they need.

That would be sweeeeeet!

@haadcode haadcode mentioned this issue Aug 19, 2016
Closed
@haadcode
Copy link
Contributor Author

Another comment that was related to this:

Some feedback from elsewhere (cc @pelle). @diasdavid @dignifiedquire make sure to note the comment re. dependencies.

Primary reason for using browser-ipfs:

Lots of unnecessary dependencies cause fragility and huge integration problems both in browser and ReactNative.

Secondary reason is size, which isn't an issue in ReactNative but a very big issue on the web in particular mobile web:

js-ipfs-api

99% 2016-08-18 11:38:01 ☆  |2.3.0| Big-Corn-Island in ~/code/js-ipfs-api
± |master ✓| → ls -l dist/
total 8600
-rw-r--r--  1 pelleb  staff  1550487 Aug 18 11:36 index.js
-rw-r--r--  1 pelleb  staff  1925637 Aug 18 11:36 index.js.map
-rw-r--r--  1 pelleb  staff   920487 Aug 18 11:36 index.min.js
browser-ipfs

± |master → origin {2} ✓| → ls -l dist/
total 8
-rw-r--r--  1 pelleb  staff  1592 Apr 25 21:53 ipfs.min.js

@dignifiedquire
Copy link
Contributor

It is important to note, that browser-ipfs does not nearly implement the full api, where as js-ipfs-api aims to implement the full spectrum. That said I believe there is a lot we can do to reduce the size, but probably not down to 2kb :)

@dignifiedquire
Copy link
Contributor

Dependency analysis with the latest master of aegir:

elliptic: 304.63 KB (14.2%)
bn.js: 170.29 KB (7.97%)
readable-stream: 146.03 KB (6.83%)
asn1.js: 126.35 KB (5.91%)
split2: 100.53 KB (4.70%)
  readable-stream: 94.3 KB (93.8%)
  through2: 4.18 KB (4.16%)
  isarray: 120 B (0.117%)
  <self>: 1.92 KB (1.91%)
hash.js: 80.62 KB (3.77%)
core-js: 70.75 KB (3.31%)
stream-http: 64.34 KB (3.01%)
  readable-stream: 49.61 KB (77.1%)
  <self>: 14.73 KB (22.9%)
browserify-aes: 61.74 KB (2.89%)
lodash._baseiteratee: 56.03 KB (2.62%)
sha.js: 52.65 KB (2.46%)
ndjson: 50.02 KB (2.34%)
  readable-stream: 47.15 KB (94.3%)
  through2: 2.09 KB (4.18%)
  isarray: 120 B (0.234%)
  <self>: 675 B (1.32%)
buffer: 47.47 KB (2.22%)
des.js: 40.69 KB (1.90%)
glob: 35.94 KB (1.68%)
diffie-hellman: 35.13 KB (1.64%)
browserify-sign: 29.14 KB (1.36%)
hoek: 28.21 KB (1.32%)
minimatch: 24.67 KB (1.15%)
parse-asn1: 23.97 KB (1.12%)
url: 23.08 KB (1.08%)
lodash: 22.44 KB (1.05%)
protocol-buffers: 20.93 KB (0.979%)
create-hash: 20.75 KB (0.971%)
tar-stream: 20.18 KB (0.944%)
  end-of-stream: 2.31 KB (11.5%)
  <self>: 17.87 KB (88.5%)
protocol-buffers-schema: 19.54 KB (0.914%)
lodash._stringtopath: 18.16 KB (0.849%)
wreck: 17.63 KB (0.825%)
public-encrypt: 17.62 KB (0.824%)
async: 16.37 KB (0.766%)
ipfs-merkle-dag: 16.24 KB (0.759%)
util: 15.4 KB (0.720%)
assert: 15.08 KB (0.705%)
qs: 14.96 KB (0.700%)
punycode: 14.33 KB (0.670%)
lodash._baseeach: 14.24 KB (0.666%)
multiaddr: 12.23 KB (0.572%)
ripemd160: 11.91 KB (0.557%)
is-property: 10.76 KB (0.503%)
boom: 10.1 KB (0.472%)
ip: 9.96 KB (0.466%)
fs.realpath: 9.58 KB (0.448%)
events: 8.13 KB (0.380%)
string_decoder: 7.61 KB (0.356%)
create-ecdh: 7.49 KB (0.350%)
lodash.map: 7.37 KB (0.345%)
miller-rabin: 6.45 KB (0.302%)
browserify-cipher: 6.25 KB (0.292%)
cipher-base: 6.25 KB (0.292%)
path-browserify: 6.04 KB (0.283%)
core-util-is: 5.9 KB (0.276%)
pbkdf2: 5.54 KB (0.259%)
bl: 5.23 KB (0.244%)
crypto-browserify: 5.12 KB (0.239%)
querystring-es3: 5.06 KB (0.237%)
process: 4.4 KB (0.206%)
constants-browserify: 4.33 KB (0.202%)
brace-expansion: 4.23 KB (0.198%)
lodash._basetostring: 4.16 KB (0.194%)
create-hmac: 4.03 KB (0.188%)
evp_bytestokey: 3.88 KB (0.181%)
vm-browserify: 3.71 KB (0.174%)
browserify-des: 3.63 KB (0.170%)
multihashes: 3.55 KB (0.166%)
stream-browserify: 3.54 KB (0.166%)
browserify-rsa: 3.53 KB (0.165%)
base64-js: 3.24 KB (0.152%)
buffer-shims: 3.16 KB (0.148%)
util-deprecate: 3.15 KB (0.147%)
stable: 2.94 KB (0.137%)
lodash.filter: 2.79 KB (0.131%)
brorand: 2.54 KB (0.119%)
sandwich-stream: 2.5 KB (0.117%)
babel-runtime: 2.47 KB (0.115%)
timers-browserify: 2.06 KB (0.0962%)
process-nextick-args: 2.03 KB (0.0950%)
base-x: 2.02 KB (0.0945%)
randombytes: 2.01 KB (0.0942%)
ieee754: 2.01 KB (0.0938%)
promisify-es6: 1.91 KB (0.0892%)
is-ipfs: 1.89 KB (0.0885%)
signed-varint: 1.84 KB (0.0863%)
  varint: 1.42 KB (77.0%)
  <self>: 435 B (23.0%)
builtin-status-codes: 1.71 KB (0.0800%)
varint: 1.67 KB (0.0779%)
multipart-stream: 1.42 KB (0.0664%)
inherits: 1.31 KB (0.0614%)
generate-function: 1.27 KB (0.0592%)
balanced-match: 1.15 KB (0.0539%)
multihashing: 1.14 KB (0.0532%)
webpack: 1.03 KB (0.0481%)
lodash._basefilter: 996 B (0.0455%)
ipfs-block: 934 B (0.0427%)
os-browserify: 927 B (0.0423%)
wrappy: 905 B (0.0413%)
inflight: 897 B (0.0410%)
to-arraybuffer: 881 B (0.0402%)
is-stream: 800 B (0.0365%)
xtend: 768 B (0.0351%)
streamifier: 685 B (0.0313%)
path-is-absolute: 599 B (0.0274%)
isstream: 588 B (0.0269%)
minimalistic-assert: 504 B (0.0230%)
once: 417 B (0.0190%)
buffer-xor: 412 B (0.0188%)
flatmap: 382 B (0.0174%)
concat-map: 345 B (0.0158%)
https-browserify: 325 B (0.0148%)
generate-object-property: 287 B (0.0131%)
isarray: 264 B (0.0121%)
bs58: 206 B (0.00941%)
detect-node: 202 B (0.00923%)
indexof: 199 B (0.00909%)
webcrypto: 154 B (0.00703%)
<self>: 50.22 KB (2.35%)

@pelle
Copy link

pelle commented Aug 19, 2016

My general beef with most of the JS code in the ethereum world and I guess also the IPFS world is the reliance on node infrastructure and many, many dependencies.

I think the reason being is that it's easy to develop libraries on this, but also the IMHO mistaken belief that having code that works on both node and in the browser is a good thing.

Size is the most obvious problem here, for example instead of using the browsers built in networking libraries much of what is implemented in these many dependencies are things that already exist. They may not be as good, but they do the trick for pretty much all browser cases, because in the end thats what is used under all the dependencies and fancy facades anyway.

So if the primary reason for creating js-ipfs-api is to run in node. Thats fine and it's awesome that it can in a pinch run in the browser. But if we want web apps interacting with ipfs we need to implement things in a way thats efficient for the browser.

The less obvious problem is fragility, due to so many dependencies, duplicated dependencies, multiple libraries that do the same thing. Each of these libraries causing problems with browserify, webpack what have you.

So I know and understand thats how node work and it works well in a simple uniform environment like the v8. But with all the different browsers and now also things like React Native it is a literal nightmare interacting with these libraries.

The most important thing I think the IPFS community can do is fully document the http api well. Interacting with http is the lowest common denominator and any good web developer understands how to do this. My browser-ipfs shows that for most cases it's not necessary to be particularly complex.

Secondly different classes of js api's need to be created. This project I only just learnt about yesterday, so I'm learning about it now and may not understand much about it yet.

But the needs of people interacting from the browser and from doing ipfs server integrations are very different.

Anyway I'm happy to help out and will attempt to make browser-ipfs as compliant as possible without adding to the dependency list and add more features as needed.

@jbenet
Copy link
Contributor

jbenet commented Aug 21, 2016

cc @RichardLitt for http api docs comments above

@jbenet
Copy link
Contributor

jbenet commented Aug 21, 2016 •
edited
Loading

@dignifiedquire im not sure why these are necessary. You could prob drop 30% on crypto libraries alone!

# okay why is there an impl of (what i can only guess is) ECC
elliptic: 304.63 KB (14.2%)

#  i imagine this is here because of the crypto libs
bn.js: 170.29 KB (7.97%)

readable-stream: 146.03 KB (6.83%)  # wow that's big.
asn1.js: 126.35 KB (5.91%)  # why do we need asn1.js encoding? TLS?

# more crypto stuff... why? TLS?
des.js: 40.69 KB (1.90%)
diffie-hellman: 35.13 KB (1.64%)
browserify-sign: 29.14 KB (1.36%)
ripemd160: 11.91 KB (0.557%)
create-ecdh: 7.49 KB (0.350%)

@dignifiedquire
Copy link
Contributor

@jbenet yes this is a known issue with our current set of crypto libraries, neither forge nor browserify-crypto are modular. This means using a single crypto function (in this case sha2 from multihashing) imports the complete! crypto library.

In this specific case we could replace https://github.com/multiformats/js-multihashing/blob/master/src/index.js#L4 with direct imports to sha.js but to properly fix these issues we need a modular crypto library, plus the things I'm currently drafting a proposal for, which will improve a lot of these issuse.

@dignifiedquire dignifiedquire mentioned this issue Aug 21, 2016
Closed
@pelle
Copy link

pelle commented Aug 21, 2016

I was looking through the code trying to find out where the digest code is called and it seems just from my skimming around to be wherever the DAGNode and DAGLink is called. I'm still learning about how these things work so forgive my ignorance. But do they ever actually need to create digests themselves in this particular use case?

If they do then perhaps a lightweight version of the library can be generated using webpack or browserify with only the higher level apis which would solve the majority of these issues.

I agree a good clean sha implementation with only a few dependencies is important. I'll see if I can help with that, since at least that part I do understand :-).

@pelle
Copy link

pelle commented Aug 21, 2016

Sorry I just realized that sha.js will do exactly that. @dignifiedquire replacing that line as you suggest will absolutely be a huge help. That may even let me use it on ReactNative.

@dignifiedquire
Copy link
Contributor

@pelle I think this is something we can do easily in the next days.

For long term improvements I have just written up something here ipfs/js-ipfs#398 (comment) and would value any feedback :)

@pelle pelle mentioned this issue Aug 21, 2016
Open
@pelle
Copy link

pelle commented Aug 21, 2016

Just created above PR to help on that side. I'll have a look at the other comment @dignifiedquire

@jbenet
Copy link
Contributor

jbenet commented Aug 21, 2016

Great! Yeah definitely rip out only the things you need.
On Sun, Aug 21, 2016 at 12:22 Pelle Braendgaard notifications@github.com
wrote:

Just created above PR to help on that side. I'll have a look at the other
comment @dignifiedquire https://github.com/dignifiedquire

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#353 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIcoZUox-HMsv1kZyqV7f4nkoYDs3noks5qiHtZgaJpZM4JoQ6W
.

@dignifiedquire
Copy link
Contributor

Replacing wreck for http requests with fetch and a polyfill for node & incompat browsers, reduces the size a decent amount. Combined with replacing webcrypto with sha.js in multihashing I got the raw bundle size down to 773K and the minified version to 422K.

@pelle
Copy link

pelle commented Aug 22, 2016

👍

@daviddias daviddias mentioned this issue Aug 23, 2016
Closed
@dignifiedquire dignifiedquire mentioned this issue Aug 23, 2016
Closed
6 tasks
@daviddias daviddias changed the title [technical exploration] Optimize the size of the dist build Bundling js-ipfs-api Oct 28, 2016
@daviddias
Copy link
Contributor

daviddias commented Oct 28, 2016 •
edited by dignifiedquire
Loading

This is coming! :D

  • async crypto merged
  • wreck -> fetch

Main tracking issue: ipfs/js-ipfs#429 (covers both js-ipfs and js-ipfs-api)

@dignifiedquire
Copy link
Contributor

We shipped the major parts of this. Which reduces the minified bundle size down to ~500k.

@dignifiedquire dignifiedquire removed their assignment Nov 8, 2016
@daviddias
Copy link
Contributor

All of the tasks for this issue are now complete and we have two awesome examples you can check out and try by yourself the new easy bundling, find them here:

https://github.com/ipfs/js-ipfs-api/tree/master/examples

Closing this issue but feel free to reopen if you encounter any other hurdles :)

@snyk-bot snyk-bot mentioned this issue Dec 18, 2021
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@pelle @jbenet @dignifiedquire @daviddias @haadcode