Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make server component separate and this module just install npm deps #70

Merged
merged 35 commits into from
Oct 25, 2018
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
2af14ba
feat: add registry cloner
achingbrain Oct 7, 2018
ca17768
chore: split into separate modules
achingbrain Oct 8, 2018
ddd5999
fix: dial all available addresses
achingbrain Oct 12, 2018
2bf789a
feat: copy and paste like an animal
achingbrain Oct 12, 2018
318e68e
feat: show root cid on registry mirrors
achingbrain Oct 12, 2018
05c578a
fix: change module path
achingbrain Oct 12, 2018
085e875
fix: update config
achingbrain Oct 12, 2018
c7eae4b
fix: correct deploy script
achingbrain Oct 12, 2018
b72aaf0
fix: make env prod
achingbrain Oct 12, 2018
05d0b06
fix: configure external ip
achingbrain Oct 12, 2018
cc64500
fix: correct url
achingbrain Oct 12, 2018
8a75368
test: adds lots of tests
achingbrain Oct 13, 2018
8e644b8
test: test all the things
achingbrain Oct 13, 2018
bb10fb9
chore: fix linting issues
achingbrain Oct 13, 2018
b37393a
test: run tests in parallel
achingbrain Oct 14, 2018
38efffe
chore: update readme
achingbrain Oct 14, 2018
1cd21ad
Merge remote-tracking branch 'origin/master' into split-into-modules
achingbrain Oct 14, 2018
da29363
chore: update image
achingbrain Oct 14, 2018
6c75058
chore: move services out of project
achingbrain Oct 14, 2018
8cb5bfa
fix: update paths and only delete local reg when we have dialled master
achingbrain Oct 14, 2018
3684624
fix: use cached index if master cannot be dialled
achingbrain Oct 14, 2018
45acd2d
chore: change description and remove old binary
achingbrain Oct 17, 2018
8472d70
chore: restore files
achingbrain Oct 17, 2018
606d3b9
chore: addres PR comments
achingbrain Oct 23, 2018
afe7a3d
chore: fix whitespace
achingbrain Oct 23, 2018
ebe84dd
chore: update readme
achingbrain Oct 23, 2018
3861272
feat: pass extra commands to package manager
achingbrain Oct 23, 2018
5bbe609
fix: use correct arg
achingbrain Oct 23, 2018
a0c41f6
chore: centre images
achingbrain Oct 23, 2018
8cd5d72
chore: centre images
achingbrain Oct 23, 2018
e27a54a
chore: fix up images
achingbrain Oct 23, 2018
f59f851
chore: move headings around
achingbrain Oct 23, 2018
877053d
fix: add flush and fix tests
achingbrain Oct 24, 2018
ce56654
fix: dial mirror instead of replication master
achingbrain Oct 25, 2018
1fe471e
feat: rewrite lockfiles with repo url
achingbrain Oct 25, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions Dockerfile

This file was deleted.

135 changes: 45 additions & 90 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
npm on IPFS
===========
# npm on IPFS

![](/img/ip-npm-small.png)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This image was great btw

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you prefer it to the new image? Since #55 was opened I thought you wanted it updating?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see the js-ipfs repo for a good size

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but the network of multiple npm registries is still super cool. Can we meet in the middle?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I'll put it back in :)

![npm on IPFS logo](./img/npm-on-ipfs.svg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nicpick, can you make it centered. Just like libp2p https://github.com/libp2p/js-libp2p?


[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](https://protocol.ai)
[![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/)
Expand All @@ -12,7 +11,9 @@ npm on IPFS

> Install your favourite modules from the Distributed Web using IPFS. Have a cache always ready and share them in all your local networks.

# Resources
![npm distributed on top of lots of connected IPFS nodes worldwide](/img/ip-npm-small.png)

## Resources

- [The original demo video](https://vimeo.com/147968322)
- [Lengthy introduction in a blog post](http://daviddias.me/blog/stellar-module-management/)
Expand All @@ -22,8 +23,6 @@ npm on IPFS

[Alex Potsides](https://github.com/achingbrain)

# Quick setup (probably all that you need)

## Install this module

```bash
Expand All @@ -32,106 +31,62 @@ npm on IPFS

# Usage

Wait for the `Server running` message:

```bash
$ docker run ipfs-npm
📦 Mirroring npm on localhost:50321
😈 Using in-process IPFS daemon
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/Qm...
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/Qm...
Swarm listening on /ip4/172.17.0.2/tcp/4002/ipfs/Qm...
🚀 Server running
🔧 Please either update your npm config with 'npm config set registry http://localhost:50321'
🔧 or use the '--registry' flag, eg: 'npm install --registry=http://localhost:50321'
```

Port `50321` is default and can be set with `--port`.
`ipfs-npm` wraps your chosen package manager (e.g. npm or yarn) with configuration to use IPFS to retrieve your dependences instead of over HTTP from the central npm registry.

## Configure npm

Set up your npm to use `ipfs-npm` with the default port through:
In the directory with your `package.json` file, run:

```bash
$ npm config set registry http://localhost:50321
$ ipfs-npm install
👿 Spawning an in-process IPFS node
🗑️ Replacing old registry index if it exists
☎️ Dialing replication master /ip4/127.0.0.1/tcp/40020/ipfs/QmeXyYCLSivUn5Ju31jjPBYNKdncbzzEf6zdN2DyrdLAbe
📠 Copying registry index /ipfs/QmQmVsNFw3stJky7agrETeB9kZqkcvLSLRnFFMrhiR8zG1 to /npm-registry
👩‍🚀 Starting local proxy
🚀 Server running on port 57314
🎁 Installing dependencies with /Users/alex/.nvm/versions/node/v10.8.0/bin/npm
...
```

If you picked another `--port` you need to adjust accordingly.

Good to npm install away! :)

# Usage

## CLI

```bash
$ ipfs-npm --help
ipfs-npm

Starts a registry server that uses IPFS to fetch js dependencies
Installs your js dependencies using IPFS

Options:
--help Show help [boolean]
--version Show version number [boolean]
--clone Whether to clone the registry in the background
[default: true]
--eager-download Whether to eagerly download tarballs
[default: true]
--mirror-host Which host to listen to requests on
[default: "localhost"]
--mirror-port Which port to listen to requests on
[default: 50321]
--mirror-protocol Which protocol to use with the server
[default: "http"]
--mirror-registry Where to download missing files from/proxy for
non-get requests
[default: "https://registry.npmjs.com"]
--mirror-upload-size-limit How large a file upload to allow when proxying for
the registry [default: "1024MB"]
--ipfs-port Which port the daemon is listening on
[default: null]
--external-host Which host to use when reaching this mirror
--external-port Which port to use when reaching this mirror
--external-protocol Which protocol to use when reaching this mirror
--ipfs-host Which host the daemon is listening on
[default: "localhost"]
--ipfs-base-dir Which mfs prefix to use
[default: "/commons-registry"]
--ipfs-flush Whether to flush the MFS cache [default: true]
--ipfs-max-requests How many concurrent requests to make to the IPFS
daemon [default: 5]
--ipfs-type "proc" to start an in process node, "go" or "js"
to connect to a remote daemon (in conjunction with
--ipfs-port and --ipfs-host). [default: "proc"]
--clone-skim Which registry to clone
[default: "https://replicate.npmjs.com/registry"]
--clone-user-agent What user agent to specify when contacting the
registry [default: "IPFS registry-mirror worker"]
--clone-delay How long in ms to wait between cloning each module
--help Show help [boolean]
--version Show version number [boolean]
--package-manager Which package manager to use - eg. npm or yarn
[default: "npm"]
--ipfs-registry-index Where to download the registry index from if we
do not have it
[default: "https://replication.registry.js.ipfs.io"]
--ipfs-registry Where to download any packages that haven't made
it into the registry index yet from
[default: "https://registry.js.ipfs.io"]
--registry-upload-size-limit How large a file upload to allow when proxying
for the registry [default: "1024MB"]
--registry-update-interval Only request the manifest for a given module
every so many ms [default: 60000]
--ipfs-mfs-prefix Which mfs prefix to use
[default: "/npm-registry"]
--ipfs-node "proc" to start an in-process IPFS node, "go" or
"js" to spawn an IPFS node as a separate process
or a multiaddr that resolves to a running node
[default: "proc"]
--request-max-sockets How many concurrent http requests to make while
cloning the repo [default: 10]
--request-retries How many times to retry when downloading
manifests and tarballs from the registry
[default: 5]
--request-retry-delay How long in ms to wait between retries
[default: 1000]
--clone-upgrade-to-https If a tarball is specifed with an http URL, whether
to upgrade it to https [default: true]
--request-max-sockets How many concurrent http requests to make while
cloning the repo [default: 10]
--request-timeout How long in ms we should wait when requesting
files [default: 30000]
```

## Docker

```
$ docker-compose build
$ docker-compose up -d --scale registry=4
```

### Upgrading

```
$ ./deploy.sh
```

## Important

If you are on Mac OS X, make sure to increase the limit of files open (with `ulimit -Sn 4096`), otherwise the ipfs daemon will be sad and throw 502 replies.

# Acknowledgements

This module takes a lot of inspiration from [reginabox](https://www.npmjs.com/package/reginabox). Big thank you to everyone that contributed with code or to the [discussion](https://github.com/ipfs/notes/issues/2) to make this happen.
21 changes: 0 additions & 21 deletions conf/proxy.conf

This file was deleted.

13 changes: 0 additions & 13 deletions deploy.sh

This file was deleted.

45 changes: 0 additions & 45 deletions docker-compose.yml

This file was deleted.

2 changes: 1 addition & 1 deletion img/npm-on-ipfs.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
25 changes: 12 additions & 13 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
{
"name": "ipfs-npm",
"version": "0.10.0",
"description": "Set up a NPM registry mirror, using your favourite storage, IPFS! :D",
"description": "Install your npm dependencies using IPFS and the distributed web! :D",
"leadMaintainer": "Alex Potsides <alex.potsides@protocol.ai>",
"main": "src/index.js",
"main": "src/cli/bin.js",
"bin": {
"registry-mirror": "src/cli/bin.js",
"ipfs-npm": "src/cli/bin.js"
},
"scripts": {
"test": "aegir test -t node",
"test:node": "aegir test -t node",
"coverage": "aegir coverage",
"lint": "aegir lint",
"start": "CLONE_EAGER_DOWNLOAD=false CLONE=false IPFS_FLUSH=true registry-mirror",
"start": "node .",
"release": "aegir release -t node",
"release-minor": "aegir release --type minor -t node",
"release-major": "aegir release --type major -t node"
Expand All @@ -34,28 +33,28 @@
},
"homepage": "https://github.com/ipfs-shipyard/npm-on-ipfs#readme",
"dependencies": {
"aws-sdk": "^2.325.0",
"datastore-s3": "~0.2.0",
"debug": "^4.0.1",
"dnscache": "^1.0.1",
"express": "^4.16.3",
"express": "^4.16.4",
"express-http-proxy": "^1.4.0",
"express-prom-bundle": "^4.2.1",
"follow-registry": "achingbrain/follow-registry",
"hat": "~0.0.3",
"ipfs": "~0.32.2",
"ipfs": "~0.32.3",
"ipfs-api": "^25.0.0",
"ipfsd-ctl": "~0.39.2",
"once": "^1.4.0",
"output-buffer": "^1.2.0",
"ipfs-registry-mirror-common": "^1.0.0",
"request": "^2.88.0",
"request-promise": "^4.2.2",
"which-promise": "^1.0.0",
"yargs": "^12.0.1"
},
"devDependencies": {
"aegir": "^15.0.0",
"chai": "^4.1.2",
"dirty-chai": "^2.0.1",
"mock-require": "^3.0.2",
"promise-delay": "^2.1.0",
"sinon": "^6.1.5"
"npm-run-all": "^4.1.3",
"sinon": "^7.0.0"
},
"contributors": [
"Danny Arnold <despair.blue@gmail.com>",
Expand Down
Loading