Compiled web3 bundle is HUGE

[josephg]

The browserify-compiled web3 bundle is crazy huge. I'm using the 1.0 beta and pre-minification web3 is 1.3 megs. Minification halves that, but its still way bigger than it needs to be.

$ cat foo.js 
require('web3')
$ browserify foo.js -o bundle.js
$ ls -l bundle.js 
-rw-r--r--  1 josephg  staff  1365873 15 Nov 16:00 bundle.js

Most of the bundle is transitive dependancies of dubious usefulness that the dapp I'm writing certainly won't need - like an ASN1 parser, an elliptic curve cyper, a 160k promise wrapper (wat) and the distributed filesystem libraries.

Have a browse of the contents of the compiled bundle.

Fixing this will not just lower download size, it will also make dapps load faster and improve the experience from nodejs. Node currently needs nearly 1 second to parse all that JS even on my fancy macbook:

$ node
> console.time('load'); require('web3'); console.timeEnd('load')
load: 750.081ms

[josephg]

... It also includes 2 copies of bn.js for some reason

[1blockologist]

Yeah, this is ridiculous. I'm preparing what is basically a static site and this is hugely problematic for many people worldwide that attempt access.

[williamchong]

as of beta34, I am having 6 bn.js of 2 different versions (4.11.6, 4.11.8) just by installing web3
please advise

[pubkey]

It looks like in beta.34, all submodules use the same bn.js@4.11.6. Maybe we can figure out why npm will install bn.js into each module instead of once. Webpack will then also only use bn.js once.

[pubkey]

The next step would be to move to es6-modules so we do not require the whole bn.js or underscore each time. Or is there already a plan for that?

[nivida]

Yes we have an 1.0ES6 branch @pubkey please feel free to contribute.
I will have a look on the dependencies.

[roshanrags]

@nivida Is there a status page or something summarizing the current status of the ES6 branch? I can contribute, but don't really know where to start or what has been done so far.

Maybe a Github Project tracking it would be helpful.

[sweetpalma]

+1 to this issue. Final size of my bundle instantly bloated from 240kb to more than 1mb just by importing web3, analyzer stats are similar to @pubkey - six imports of bn.js. This is absolutely unacceptable, is there any workaround for this issue? I am using beta 35.

[nivida]

Hay all, I'm tested it with severall webpack configs for optimizing the package dependencies and I also checked the BN.js versions but with no success. The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph. My current workarround would be to include the minified version of web3 directly from the node_modules dist folder this file is only ~700KB. I'll talk about this with fabian maybe we could change the project structure.

[pubkey]

I think it would be better to switch to a mono-package instead of publishing web3-eth web3-net etc as separate npm-modules. To me it does not make sense or will be possible to ever use different version of the sub-packages, so why not just publish web3 and the user can then require('web3/eth') and so on.

[ondratra]

@pubkey I agree with this.

Is there any progress on this issue? I am willing to help.

[josephg]

The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph.

If you reference a version range in package.json, npm can reuse the same bn install across all dependancies. Eg, "bn.js"="^4.11.8" (note the ^) will use any version of bn from version 4.11.8 up to version 5. If you do it that way, the dependancy graph won't necessarily need multiple copies of bn.

[johnhforrest]

What's the current status of this? The bundle size is continuing to grow. beta36 added almost 200KB minified to the bundle size. At this point we are punting upgrading because of the extra bloat.

beta36 - 791.88 KB
beta35 - 616.12 KB
beta34 - 582.78 KB

I think your comment on Aug 20 is the correct way to solve the BN.js duplication issue. Maybe that can be done for just the purposes of building the dist files if it's there's too much risk to make a sweeping change like this.

[leonprou]

@pubkey @nivida running npm ls | grep bn.js you can see that most of the dependencies are deduped (not duplicated). But there are some dependencies that define a strict version of bn.js, they are:

• https://github.com/SilentCicero/number-to-bn
• https://github.com/ethjs/ethjs-unit
• web3-utils ( I guess it's doing this because of the two above)

So we should ask these packages to loosen their dependency to ~4.11.6 for example

[leonprou]

Different but related thing I would like to ponder about is how to make the web3 library more slim by reducing web3 packages I don't use. For example probably you need just one provider (http, websocket, ipc), not all of them. I think the same goes for many libraries.

So package.json could look like this:

    "web3-slim": "^1.0.0-beta.37"
    "web3-providers-http": "^1.0.0-beta.37"

web3-slim is the module that will assure the same usage of web3 module for coding. But in fact this module will include only the specified dependencies.

[pubkey]

@josephg yes, asking them to loosen their bn.js version would be a great improvement.
About splitting web3 into more sub-packages, I'm not that sure. What you really want to have is that you can just import parts of Web3 and then your webpack or whatever bundling you to, should be able to tree-shake away the things you do not need. Also see my comment above. Splitting into more submodules would only be helpfull if you ever wanted to use different versions of parts of web3 which I do not think you need.

[leonprou]

@pubkey ok, for now the webpack is not working at all - #1969.

After this issue will be fixed we can get a look how the tree shaking is working. It's for sure the best option to get size improvements out of the box.

[nivida]

All of these problems will be solved after I got the PR #2000 merged :) @leonprou

[williamchong]

with beta.46, I am still getting 3 bn.js, and there are 4 lodash now
edit: also somehow the umd version was included instead of esm, but other lib I used (vue) has esm included no problem in the same build

[BigMurry]

can use webpack alias to solve this issue.

1. publish a forked version of bn.js
2. yarn add fork-bn.js
3. change the webpack config

module.exports = {
  // other configs
  resolve:{
    // other configs
    alias:{
      "bn.js":"fork-bn.js"
    }
  }
}

There is only one bn.js now, cheers :)

[nivida]

I've created a webpack example here and I think this is a good boilerplate to start. The long term goal will be to refactor the folder and module structure for preventing the multiple bundling of dependencies.

[BigMurry]

@nivida you list a better option

[josephg]

Not stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[hems]

not stale

[blaynem]

Still not stale, package is quite large.

[vphilot]

not stale

[yiyusong]

not stale

[ffmcgee725]

Is there really no way to reduce the load this takes on a bundled app ?
I'm trying to optimize a Next.JS app bundle size, this package takes up more than 40% space in final build, I even try to defer loading with dynamic imports whenever I need to use library but to no avail, when I'm analysing bundle through next bundle analyser, web3 packages are there from the get-go.

Is there no way to defer loading this package only when user interacts with a button that would require functionality from this ?

[hurtonypeter]

Is there really no way to reduce the load this takes on a bundled app ? I'm trying to optimize a Next.JS app bundle size, this package takes up more than 40% space in final build, I even try to defer loading with dynamic imports whenever I need to use library but to no avail, when I'm analysing bundle through next bundle analyser, web3 packages are there from the get-go.

Is there no way to defer loading this package only when user interacts with a button that would require functionality from this ?

I think analyzers are watching "the whole picture", so that could be the reason you can see it, but this is really an analyzer issue. Same with deferring package loading, that should be solved by next/your app.

The issue with the lib is its non-treeshakeable nature. It should be rewritten using ES6 modules.

[canversemike]

+1'ing here.
First Load JS shared by all 725 kB :<
Site was just shy of 60kb before adding web3 stuff.

[0xevm1]

Is there really no way to reduce the load this takes on a bundled app ? I'm trying to optimize a Next.JS app bundle size, this package takes up more than 40% space in final build, I even try to defer loading with dynamic imports whenever I need to use library but to no avail, when I'm analysing bundle through next bundle analyser, web3 packages are there from the get-go.
Is there no way to defer loading this package only when user interacts with a button that would require functionality from this ?

I think analyzers are watching "the whole picture", so that could be the reason you can see it, but this is really an analyzer issue. Same with deferring package loading, that should be solved by next/your app.

The issue with the lib is its non-treeshakeable nature. It should be rewritten using ES6 modules.

Seems like a perfect job for a gitcoin grant

[hems]

Seems like a perfect job for a gitcoin grant

I think most people rather use gitcoin to throw a 5 years party of this issue this year and hire the best djs !

Clearly nobody cares about 800kb download.

[jdevcs]

1.x is in min required maintenance phase and optimizations candidate is web3.js 4.x , we recently released 4.0.1-alpha.0 , doc: https://docs.web3js.org/ . There so many new features and optimizations in it. Additionally in 4.x+ we have plan of removing top 3 dependencies and supporting tree-shaking as initial steps as well. If you have cool ideas you are invited for 4.x contributions.

[shrpne]

https://bundlephobia.com/package/web3@4.0.1-alpha.0

it's dropped to 1.1MB minified and 310kb gzipped from 2MB minified, 640kb gzipped

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[josephg]

Not stale 🙃

[adrianmcli]

At this point, maybe we should make a POAP for those of us still tracking this.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[josephg]

Not stale

[Martin2037]

Not stale

[Loque18]

Not stale

[jdevcs]

closing this issue in favor of 4.x release
https://www.npmjs.com/package/web3/v/4.0.1
https://github.com/web3/web3.js/releases/tag/v4.0.1

All further optimizations will be done in 4.x only and 1.x will be in minimum maintenance phase.