feat: add implementation #2

vasco-santos · 2020-11-30T10:58:21Z

This PR adds the initial implementation of an out of the box Libp2p hop relay to be deployed. It is highly customisable and comes with a Docker image.

Context: libp2p/js-libp2p#804

jacobheun

So the js-libp2p-daemon already supports a good chunk of this. Would it be worth just adding docker support there instead of maintaining the 2 of these? We might have to make some tweaks to support not needing a local control connection, but that might be a better consolidation of effort.

jacobheun · 2020-11-30T16:19:44Z

src/index.js

+      transport: [Websockets, TCP],
+      streamMuxer: [Muxer],
+      connEncryption: [Crypto],
+      contentRouting


I think we should just drop advertise support for now in here until we fix the DHT. The relay won't be auto discoverable but being able to quickly run a standalone, static relay is still valuable. The external dependency doesn't really seem worth it to me right now.

This should also use pubsub peer discovery https://github.com/libp2p/js-libp2p-pubsub-peer-discovery.

Ok, I removed the advertise stuff and added the pubsub discovery with options in the CLI for it

jacobheun · 2020-11-30T16:22:36Z

src/index.js

+        enabled: true, // Allows you to dial and accept relayed connections. Does not make you a relay.
+        hop: {
+          enabled: true, // Allows you to be a relay for other peers
+          active: true // You will attempt to dial destination peers if you are not connected to them


Disable active, support needs to be added, https://github.com/libp2p/js-libp2p/blob/master/src/circuit/circuit/hop.js#L53, this should also be configurable by users.

src/bin.js

package.json

README.md

vasco-santos · 2020-11-30T18:16:17Z

So the js-libp2p-daemon already supports a good chunk of this. Would it be worth just adding docker support there instead of maintaining the 2 of these? We might have to make some tweaks to support not needing a local control connection, but that might be a better consolidation of effort.

This is a good point.

I was looking here into an out of the box solution for spinning up a relay, in order to enable someone to easily setup it without any prior knowledge needed on js-libp2p configuration.

The big limitation here is that we cannot have multiple docker files in a single github repository totally integrated in Docker Hub for things like automated builds as the name of the image is the name of the repo. Otherwise, we could have a set of images in a single repo.

If we go with the daemon, we would need to have a more sane default setup for a typical libp2p node that you could run on docker. As a consequence, the configuration needed to run the container would be way more complex. Moreover, this module can also be used as a lib with the libp2p API instead of the daemon client.

With the above in mind, I think it makes sense to have it as a separate repo and I would not expect much maintenance overhead here with dependabot

jacobheun · 2020-12-01T13:02:29Z

If we go with the daemon, we would need to have a more sane default setup for a typical libp2p node that you could run on docker. As a consequence, the configuration needed to run the container would be way more complex.

The daemon should have sane defaults anyway. The main thing it's missing is pubsub discovery, it already enables hop by default. FWIW, the few relays libp2p currently runs on the public network use the go daemon.

Moreover, this module can also be used as a lib with the libp2p API instead of the daemon client.

The main advantage of this is being able to require a module that's good to go, but tweaking configuration isn't trivial. Thinking about preconfigured setups a bit more, how would you want to handle a variety of these? In their own modules? A mono repo?

I don't want to block this, as we can always archive it if it get's stale, but thinking about how and where preconfigured builds should exist, especially as we revamp configuration in libp2p is something we should spend some time thinking about.

jacobheun · 2020-12-01T13:06:49Z

Dockerfile

@@ -0,0 +1,28 @@
+FROM node:lts-buster


What's the size of this image?

I'd take a look at libp2p/js-libp2p-webrtc-star#223 to see the improvements made there. Buster likely has a lot more than is needed.

Very smaller this one! thanks

jacobheun · 2020-12-01T13:10:54Z

README.md

+DISCOVERY_TOPICS='_peer-discovery._app1._pubsub,_peer_discovery._app2._pubsub'
+```
+
+Please note that you should expose expose the used ports with the docker run command. The default ports used are `8003` for the metrics and `150003` for the websockets listener


Suggested change

Please note that you should expose expose the used ports with the docker run command. The default ports used are `8003` for the metrics and `150003` for the websockets listener

Please note that you should expose the used ports with the docker run command. The default ports used are `8003` for the metrics, `8000` for the TCP transport, and `15002` for the Websockets transport

Also, I don't see any defaults aside from metrics. The docker commands include them, but there aren't defaults in the code unless I am missing something.

Ok, perhaps the example below created confusion. I added now example before it.

So, the default is in the getListenAddresses and is '/ip4/127.0.0.1/tcp/15003/ws'. It is also mentioned above in the readme on the multiaddrs subsection. Should I also add a default tcp multiaddr?

jacobheun · 2020-12-01T13:21:50Z

README.md

+
+```sh
+libp2p-relay-server --discoveryTopics '_peer-discovery._app1._pubsub' '_peer_discovery._app2._pubsub'
+```


I'd just leave configuring this out, keep it simple. It binds to the global space out of the box, so just avoid people doing the wrong thing. IMO if people need to configure this in depth they can use the config as a template. The point of this repo is basically to get people up and running quickly, so we should just reduce the noise.

What's the bare minimum I need to do to get this running? The readme should focus on that. I think a consistent PeerId and Announce addresses are the only thing I should actually need to set. I'd prioritize and separate the need to have, from the optional stuff.

I divided them in two now. The required/recommended is in the usage section: peerId, multiaddrs and ssl

Then a new configuration section created for fine tuning of the image

jacobheun · 2020-12-01T13:24:18Z

README.md

+
+### SSL
+
+You should setup an SSL certificate with nginx and proxy to the API. You can use a service that already offers an SSL certificate with the server and configure nginx, or you can create valid certificates with for example [Letsencrypt](https://certbot.eff.org/lets-encrypt/osx-nginx). Letsencrypt won’t give you a cert for an IP address (yet) so you need to connect via SSL to a domain name.


Not needed right now, but it would be cool to see an example dockerfile extending this one, templated with nginx and letsencrypt.

Yes, this would be good! I added it in the libp2p production guides README libp2p/js-libp2p#804 (comment) as it should probably be not specific to the relay. We can also link it from here afterwards.

vasco-santos · 2020-12-01T13:43:09Z

The daemon should have sane defaults anyway. The main thing it's missing is pubsub discovery, it already enables hop by default. FWIW, the few relays libp2p currently runs on the public network use the go daemon.

I also agree that we should have a ready to go docker container for the daemon.

I did not remember that the daemon had hop enabled by default. What is the reasoning behind it? matching go? I think that setting up a js daemon out of the box should match exactly js libp2p defaults, which is not the case here.

The main advantage of this is being able to require a module that's good to go, but tweaking configuration isn't trivial. Thinking about preconfigured setups a bit more, how would you want to handle a variety of these? In their own modules? A mono repo?
I don't want to block this, as we can always archive it if it get's stale, but thinking about how and where preconfigured builds should exist, especially as we revamp configuration in libp2p is something we should spend some time thinking about.

This is really something we should think about. It would be great if we could provide at the same time preconfigured builds and libp2p configurations. Libp2p configurations could be a mono repo.

Once we do the configuration improvements, we need to have ready to go configurations. We need to check if we can have multiple Dockerfiles in a lerna mono repo with automated builds, but as far as I know this is not possible at the moment

If we could do that, a mono repo with configurations that can be imported and docker images that could be pulled for each configuration. This would enable to use the daemon for the Docker builds and it would be easy to import and use as a module too.

If we cannot use a mono repo, this will be difficult to achieve in terms of the builds. Specially thanks to the modularity nature of libp2p, there will be no way to have a docker image where we can simply add/remove a given module. In this case, we should probably have a mono repo with all the configurations and repos with the most common ready to go builds. Specially for things like relay, rendezvous servers, ... Considering for example browser based configurations, they would not need a ready to go build, so it might make sense to just have the configurations in a lerna repo.

We should discuss this better later on, I will mention this discussion in the configuration issue

… cli and docker params and optional

jacobheun · 2020-12-04T11:16:34Z

This is really something we should think about. It would be great if we could provide at the same time preconfigured builds and libp2p configurations. Libp2p configurations could be a mono repo.

We may want to look at just making js-libp2p itself a monorepo, akin to js-ipfs, as this would make shipping preconfigured builds much easier. This would also enable us to pull in things like the daemon and daemon client to make that workflow suck less. Can you open up an issue in js-libp2p to discuss this? It would be good to chat through the benefits, drawbacks and lessons learned from js-ipfs in an open context there.

I'll finish going through this PR and we can move the monorepo convo there.

README.md

jacobheun · 2020-12-04T11:54:37Z

src/index.js

+const Muxer = require('libp2p-mplex')
+const { NOISE: Crypto } = require('libp2p-noise')


I'd recommend just giving them their names, it makes the config easier to read. We use this format in testing to be generic, we should avoid it outside of testing.

jacobheun · 2020-12-04T11:57:32Z

src/utils.js

+}
+
+function getListenAddresses (argv) {
+  let listenAddresses = ['/ip4/127.0.0.1/tcp/15003/ws']


We're only doing WS by default? We should be doing TCP as well. Having only 1 really reduces the usefulness of the relay as we won't be able to relay TCP -> WS.

jacobheun · 2020-12-04T11:59:08Z

test/index.spec.js

+    await relay.start()
+
+    expect(relay.multiaddrs).to.have.lengthOf(listenAddresses.length)
+    relay.multiaddrs.forEach((m) => listenAddresses.includes(m.toString()))


This isn't checking anything, it's just iterating and returning a boolean to nowhere.

test/index.spec.js

jacobheun · 2020-12-04T12:16:32Z

README.md

+
+```sh
+docker build NAME -t libp2p-relay
+docker run -p 8003:8003 -p 15002:15002 -p 8000:8000 -e LISTEN_MULTIADDRS='/ip4/127.0.0.1/tcp/15002/ws,/ip4/127.0.0.1/tcp/8000' -d libp2p-relay


It would be helpful to show how to include the peer id file here. Setting PEER_ID='./id.json' will fail because no file has been given to docker. This example is just going to create new peer ids.

Also, if we added announce addresses to the example -e ANNOUNCE_MULTIADDRS='/dns4/localhost/tcp/8000,/dns4/localhost/tcp/15002/ws', people should be able to play around with this locally.

vasco-santos · 2020-12-04T12:39:26Z

We may want to look at just making js-libp2p itself a monorepo, akin to js-ipfs, as this would make shipping preconfigured builds much easier. This would also enable us to pull in things like the daemon and daemon client to make that workflow suck less. Can you open up an issue in js-libp2p to discuss this? It would be good to chat through the benefits, drawbacks and lessons learned from js-ipfs in an open context there.

Yes, that would be worth discussing! I will open the issue. I would add the interfaces to that list

Co-authored-by: Jacob Heun <jacobheun@gmail.com>

jacobheun · 2020-12-04T17:15:21Z

README.md

+ANNOUNCE_MULTIADDRS='/dns4/test.io/tcp/443/wss,/dns6/test.io/tcp/443/wss'
+```
+
+Docker does not provide a clean way of adding a file using the RUN command. As a file should be added before running the server, the easiest way is to create a `Dockerfile` that extends this Docker image and copy a file to a path that you can reference.


This isn't correct, you can mount the file. Here's an example you can just copy paste to get this working, assuming you have a peer id stored at id.json in your current directory (PWD). I've run this locally, but haven't tried connecting another node to it though. This could replace the example below.

peer-id --type=ed25519 > id.json docker run -p 8003:8003 -p 15002:15002 -p 8000:8000 \ -e LISTEN_MULTIADDRS='/ip4/127.0.0.1/tcp/8000,/ip4/127.0.0.1/tcp/15002/ws' \ -e ANNOUNCE_MULTIADDRS='/dns4/localhost/tcp/8000,/dns4/localhost/tcp/15002/ws' \ -e PEER_ID='/etc/opt/relay/id.json' \ -v $PWD/id.json:/etc/opt/relay/id.json \ -d libp2p-relay

Edit: added the peer-id command for good measure

Nice thanks! I tried it now and it works, this is a better solution.

vasco-santos marked this pull request as ready for review November 30, 2020 11:00

vasco-santos force-pushed the feat/add-implementation branch from c3b9607 to 7171f76 Compare November 30, 2020 11:04

feat: add implementation

2455d11

vasco-santos force-pushed the feat/add-implementation branch from 5cc24aa to 2455d11 Compare November 30, 2020 11:09

chore: use main brain in gh action

8229eda

vasco-santos force-pushed the feat/add-implementation branch 2 times, most recently from cb5b271 to d115119 Compare November 30, 2020 14:24

chore: add docker setup

a159564

vasco-santos force-pushed the feat/add-implementation branch from d115119 to a159564 Compare November 30, 2020 14:31

vasco-santos mentioned this pull request Nov 30, 2020

docs: production guide base setup libp2p/js-libp2p#804

Merged

1 task

vasco-santos requested a review from jacobheun November 30, 2020 14:39

jacobheun reviewed Nov 30, 2020

View reviewed changes

chore: address review

34df221

vasco-santos force-pushed the feat/add-implementation branch from 2a8f0e8 to 34df221 Compare December 1, 2020 10:26

vasco-santos requested a review from jacobheun December 1, 2020 10:38

jacobheun reviewed Dec 1, 2020

View reviewed changes

vasco-santos mentioned this pull request Dec 1, 2020

Improve libp2p config libp2p/js-libp2p#576

Closed

vasco-santos added 4 commits December 1, 2020 15:12

chore: update docker node image and remove custom topics

b679196

chore: change readme structure to divide between required/recommended…

cdbc8f3

… cli and docker params and optional

chore: fix ci removing custom topics remainings

f99d5a9

chore: remove unecessary docker deps install

a482dce

jacobheun reviewed Dec 4, 2020

View reviewed changes

chore: apply suggestions from code review

6ab04d4

Co-authored-by: Jacob Heun <jacobheun@gmail.com>

vasco-santos force-pushed the feat/add-implementation branch from ae63409 to 6ab04d4 Compare December 4, 2020 13:52

vasco-santos requested a review from jacobheun December 4, 2020 13:58

jacobheun reviewed Dec 4, 2020

View reviewed changes

chore: improve peer id usage with docker example

0b6238d

vasco-santos requested a review from jacobheun December 7, 2020 10:41

vasco-santos merged commit 6defd68 into main Dec 10, 2020

vasco-santos deleted the feat/add-implementation branch December 10, 2020 13:38

BigLep mentioned this pull request Apr 30, 2021

Manual "sync" isn't working in some cases ipfs-shipyard/ecosystem-dashboard#201

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add implementation #2

feat: add implementation #2

vasco-santos commented Nov 30, 2020 •

edited

Loading

jacobheun left a comment

jacobheun Nov 30, 2020

vasco-santos Dec 1, 2020

jacobheun Nov 30, 2020

vasco-santos commented Nov 30, 2020

jacobheun commented Dec 1, 2020

jacobheun Dec 1, 2020

vasco-santos Dec 1, 2020

jacobheun Dec 1, 2020

jacobheun Dec 1, 2020

vasco-santos Dec 1, 2020 •

edited

Loading

jacobheun Dec 1, 2020

vasco-santos Dec 1, 2020

jacobheun Dec 1, 2020

vasco-santos Dec 1, 2020

vasco-santos commented Dec 1, 2020

jacobheun commented Dec 4, 2020

jacobheun Dec 4, 2020

jacobheun Dec 4, 2020

jacobheun Dec 4, 2020

jacobheun Dec 4, 2020

vasco-santos commented Dec 4, 2020

jacobheun Dec 4, 2020 •

edited

Loading

vasco-santos Dec 7, 2020

	Please note that you should expose expose the used ports with the docker run command. The default ports used are `8003` for the metrics and `150003` for the websockets listener
	Please note that you should expose the used ports with the docker run command. The default ports used are `8003` for the metrics, `8000` for the TCP transport, and `15002` for the Websockets transport


		### SSL

		You should setup an SSL certificate with nginx and proxy to the API. You can use a service that already offers an SSL certificate with the server and configure nginx, or you can create valid certificates with for example [Letsencrypt](https://certbot.eff.org/lets-encrypt/osx-nginx). Letsencrypt won’t give you a cert for an IP address (yet) so you need to connect via SSL to a domain name.

		const Muxer = require('libp2p-mplex')
		const { NOISE: Crypto } = require('libp2p-noise')

feat: add implementation #2

feat: add implementation #2

Conversation

vasco-santos commented Nov 30, 2020 • edited Loading

jacobheun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Nov 30, 2020

jacobheun commented Dec 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Dec 1, 2020

jacobheun commented Dec 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Dec 4, 2020

jacobheun Dec 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Nov 30, 2020 •

edited

Loading

vasco-santos Dec 1, 2020 •

edited

Loading

jacobheun Dec 4, 2020 •

edited

Loading