feat: script to compare fresh install time

[olizilla]

A script to repeatably compare the first install time of a module via ipfs-npm against npm.

We know ipfs-npm is likely to be slower on first install time, but I want to do whatever we can to bring that time down, as it's the first experience folks will have with it.

We create a docker image with ipfs-npm in, then time how long it take to install takes there, then run the same install again in a stock node docker image via npm. Both will have empty caches for the run.

Example output

$ ./test/perf/docker-race.sh 
found ipfs-npm Docker image

---- ipfs-npm flavour ----
👿 Spawning an in-process IPFS node using repo at /root/.jsipfs
...
/usr/local/bin/iim -> /usr/local/lib/node_modules/iim/src/bin.js
+ iim@0.6.1
added 415 packages from 782 contributors in 115.216s
🎁 /usr/local/bin/npm exited with code 0
🔏 Updating package-lock.json

real	2m0.062s
user	0m0.033s
sys	0m0.024s

---- npm flavour ----
/usr/local/bin/iim -> /usr/local/lib/node_modules/iim/src/bin.js
+ iim@0.6.1
added 415 packages from 782 contributors in 8.459s

real	0m10.811s
user	0m0.032s
sys	0m0.014s

License: MIT
Signed-off-by: Oli Evans oli@tableflip.io

[olizilla]

Today it's 40s rather than 2 mins for ipfs-npm

$ ./docker-race.sh 
found ipfs-npm Docker image

---- ipfs-npm flavour ----
👿 Spawning an in-process IPFS node using repo at /root/.jsipfs
...
+ iim@0.6.1
added 415 packages from 782 contributors in 36.717s
🎁 /usr/local/bin/npm exited with code 0
🔏 Updating package-lock.json

real	0m40.804s
user	0m0.033s
sys	0m0.018s

---- npm flavour ----
/usr/local/bin/iim -> /usr/local/lib/node_modules/iim/src/bin.js
+ iim@0.6.1
added 415 packages from 782 contributors in 8.375s

real	0m10.934s
user	0m0.031s
sys	0m0.028s

[achingbrain]

Nice. I wonder if the time of day affects it?

[lidel]

Cool! I've run it multiple times and consistently get around 45s for npm-ipfs

Small ask: we should run tests in ephemeral containers (see notes below).

[victorb]

This might be relevant for you:

https://github.com/open-services/public-registry-benchmarks

Basically, the project above creates bunch of different Dockerfiles for every combination of a list of CLIs and a list of public registries: https://github.com/open-services/public-registry-benchmarks/tree/master/tests

Then it runs them all a couple of times and logs the time for each one. The average and mean gets recorded as well, then you'll get a report.md in the end with all the times.

npm-on-ipfs (the deployed version) is already there and gets run with all the rest of the registries.

You might be able to reuse it somehow to run it within your own infrastructure. Otherwise I could try to focus some time on running it nightly, and you can just use the results as-is for figuring out improvements.

[olizilla]

That is very cool @victorb. I'm assuming currently you run it add-hoc when you want to update the stats? Running it nightly or even weekly would be rad, then we could see the any improvements in published releases. I'll take a look and see what I can use to improve this test too.

[victorb]

@olizilla indeed. I just opened up a issue about running it each night: open-services/public-registry-benchmarks#1

[victorb]

Update: open-services/public-registry-benchmarks now runs the benchmarks once a day and updates the README with the latest results + stores each report in a directory in the git repository

If you can't figure out how to add a new test-case for the benchmarks, I'd be happy to help you