Provide parameter for retrying 404s automatically

[victorb]

The npm registry seems to suffer (at least in some geo-locations) of random 404s on package. This happens in Jenkins which affects everyone and also happens locally for me, from time to time.

Would be nice if this registry could provide a option to retry packages that gets 404s X times before actually trusting it.

Then we could start deploying npm-on-ipfs in the Jenkins infra.

[achingbrain]

ipfs/infra#432 will also need to be done

[victorb]

Hm, this feature is not blocked by having a S3 bucket right? We could still just have a large disk and things would work fine.

[achingbrain]

When npm was 500k packages it was estimated to need 3.5TB to mirror it. It's now about 800k packages and 8TB in size. Each worker can potentially be mirroring all of it, so it would have to be a very, very large disk.

Best just to set up the S3 bucket.

[victorb]

@achingbrain but we don't have to mirror the full npm registry, for the purpose of CI at least. It's only needed as a proxy for the packages we use in our projects. Don't know the full size of that but it won't be close to 8TB.

[achingbrain]

The idea is for it to watch skimdb.npmjs.com for updates to packages and eagerly fetch them, so it's going to be mirroring more stuff than just dependencies used by ipfs-related projects.

Even if that's turned off, we want it to be used by the public so predicting the size of the mirrored packages is hard.

I'd sleep better at night if it used a 'how-long-is-a-piece-of-string' sized storage engine than a disk on a server somewhere.

[achingbrain]

Released as 0.8.0 & deployed https://registry.js.ipfs.io