Issues connecting from IPFS nodes over WSS #18

TheDiscordian · 2022-06-20T20:48:56Z

Commit: a321472

Issue:

Both js-ipfs and go-ipfs refuse to connect to this relay node over wss, works fine over ws.

What was tried:

Both go-ipfs and js-ipfs were tried, or simply: ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A:

Kubo 0.12 fails to connect:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
error: connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A failure: no good addresses

To debug @lidel tried websocat:

$ websocat wss://ipfs.thedisco.zone:4430/ 
/multistream/1.0.0

But it indicates it should be working correctly. We also tried regular websockets, and those work fine. The reverse proxy server in use is Nginx.

Just in case, I tried to also update the cert, but this didn't change anything. It's worth noting that this is the exact same setup I was using with go-ipfs for relaying, and it was working fine. Config files provided below.

Configs:

config.json:

{
  "RelayV2": {
    "Enabled": false
  },
  "RelayV1": {
    "Enabled": true
  },
  "Network": {
    "ListenAddrs": [
        "/ip4/0.0.0.0/tcp/4011/ws",
        "/ip6/::/tcp/4011/ws"
    ],
    "AnnounceAddrs": [
	"/dns6/ipfs.thedisco.zone/tcp/4430/wss",
	"/dns4/ipfs.thedisco.zone/tcp/4430/wss"
    ]
  },
  "Daemon": {
    "PprofPort": -1
  }
}

/etc/nginx/sites-enabled/ipfs:

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }
 
    upstream websocket {
        server 127.0.0.1:4011;
    }

map $remote_addr $proxy_forwarded_elem {
    # IPv4 addresses can be sent as-is
    ~^[0-9.]+$          "for=$remote_addr";

    # IPv6 addresses need to be bracketed and quoted
    ~^[0-9A-Fa-f:.]+$   "for=\"[$remote_addr]\"";

    # Unix domain socket names cannot be represented in RFC 7239 syntax
    default             "for=unknown";
}

map $http_forwarded $proxy_add_forwarded {
    # If the incoming Forwarded header is syntactically valid, append to it
    "~^(,[ \\t]*)*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*([ \\t]*,([ \\t]*([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?(;([!#$%&'*+.^_`|~0-9A-Za-z-]+=([!#$%&'*+.^_`|~0-9A-Za-z-]+|\"([\\t \\x21\\x23-\\x5B\\x5D-\\x7E\\x80-\\xFF]|\\\\[\\t \\x21-\\x7E\\x80-\\xFF])*\"))?)*)?)*$" "$http_forwarded, $proxy_forwarded_elem";

    # Otherwise, replace it
    default "$proxy_forwarded_elem";
}
 
    server {
        listen 4430 ssl;
        ssl_certificate /etc/letsencrypt/live/ipfs.thedisco.zone/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/ipfs.thedisco.zone/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
        location / {
            proxy_set_header Forwarded $proxy_add_forwarded;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_pass http://websocket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }

The text was updated successfully, but these errors were encountered:

TheDiscordian · 2022-06-20T20:50:53Z

Forgot to add, here's a minimal test for js-ipfs that returns the following error: Uncaught (in promise) AggregateError: No Promise in Promise.any was resolved

HTML:

<!DOCTYPE html>
<html lang="en">
<head><title>js-ipfs minimal relay test</title>

<script src="https://cdn.jsdelivr.net/npm/ipfs-core@0.15.2/dist/index.min.js"></script>

<script>
var ipfs;
async function main() {
	ipfs = await window.IpfsCore.create();
	await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");
}
main()
</script>
</head>
</html>

marten-seemann · 2022-06-20T20:53:00Z

Why are you using a reverse proxy? The WebSocket transport now supports WSS.

TheDiscordian · 2022-06-20T20:53:58Z

@marten-seemann can you link me to docs for how to point it to my cert please? ❤️ I wasn't aware that I could ditch the reverse proxy now.

marten-seemann · 2022-06-20T21:16:05Z

Here's the option: https://github.com/libp2p/go-libp2p/blob/707100a521dace4a7a87fd17c04042ea483380b5/p2p/transport/websocket/websocket.go#L56-L62

In code, this would look something like:

libp2p.New(
    libp2p.Transport(ws.New, ws.WithTLSConfig(tlsConf)),
    ....
)

aschmahmann · 2022-06-21T03:31:28Z

@TheDiscordian you will not be able to connect to a WSS node from go-ipfs without doing a bunch of work (meaning custom builds with custom TLS verification) because go-libp2p could not dial WSS addresses authenticated by DNS addresses until we merged libp2p/go-libp2p#1592 which was recent and therefore not yet released.

I ran vole.exe bitswap check bafkqaaa /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A (using
ipfs-shipyard/vole#18) and got a connection just fine. There's a protocol not supported error but that's just because it doesn't speak Bitswap which is... correct for the relay 😄.

Without the PR above vole instead fails with the message below because of the WSS issue that was recently resolved:

  * [/ip4/172.105.8.48/tcp/4430/wss] x509: cannot validate certificate for 172.105.8.48 because it doesn't contain any IP SANs

TheDiscordian · 2022-06-21T15:10:35Z

@marten-seemann Thanks! I'll work on ditching the reverse proxy today, and see if that narrows anything down (it'll be nice to axe Nginx)

@aschmahmann Ah thanks! This explains why go-ipfs failed, but not js-ipfs 🤔. Will try again with Marten's tip... (Edit: Or does the relay need bitswap for js-ipfs to be happy?)

TheDiscordian · 2022-06-21T18:36:23Z

I followed @marten-seemann's advice, you can see the code here: https://github.com/TheDiscordian/go-libp2p-relay-daemon/blob/master/cmd/libp2p-relay-daemon/main.go#L82

js-ipfs now returns a very unexpected error: Uncaught (in promise) Error: no protocol with name: "'dns'". Must have a valid family name: "{ip4, ip6, dns4, dns6}".

To confirm I'm trying the exact same thing: await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");

/dns4/ returns the same error.

I also tested with websocat, and it seems to be happy (exact same result as previous).

Is this a bug with js-ipfs?

Edit: For completeness, here's my config.json:

{
  "RelayV2": {
    "Enabled": false
  },
  "RelayV1": {
    "Enabled": true
  },
  "TLS": {
    "KeyPairPaths": [["/etc/letsencrypt/live/ipfs.thedisco.zone/fullchain.pem", "/etc/letsencrypt/live/ipfs.thedisco.zone/privkey.pem"]]
  },
  "Network": {
    "ListenAddrs": [
        "/ip4/0.0.0.0/tcp/4011/ws",
        "/ip6/::/tcp/4011/ws",
        "/ip4/0.0.0.0/tcp/4430/wss",
        "/ip6/::/tcp/4430/wss"
    ],
    "AnnounceAddrs": [
	"/dns6/ipfs.thedisco.zone/tcp/4430/wss",
	"/dns4/ipfs.thedisco.zone/tcp/4430/wss"
    ]
  },
  "Daemon": {
    "PprofPort": -1
  }
}

marten-seemann · 2022-06-22T10:54:56Z

js-ipfs now returns a very unexpected error: Uncaught (in promise) Error: no protocol with name: "'dns'". Must have a valid family name: "{ip4, ip6, dns4, dns6}".

Looks like JS expects an IP address, not a domain name. The domain name is needed though to allow the browser to verify the certificate. @achingbrain, this seems like bug in js-libp2p, doesn't it?

lidel · 2022-06-24T15:01:30Z

Related /dns/ issues:

JS: fix: support dns and dnsaddr multiformats/js-multiaddr#253 (done, fix bubbling up)
GO: libp2p/go-libp2p@4dd3b07 + Support and test /dns/ multiformats/go-multiaddr-dns#47 (queued, if someone wants to do this faster/prioritize, comment on the issue)

lidel · 2022-08-01T19:21:39Z

Relevant bug got fixed in https://github.com/libp2p/go-libp2p/pull/1612/files#diff-8a203f70505b8dae3ee69421aa2326392c2f301091f557e44649fa40205f246fR26
- This was released in go-libp2p 0.20.3, Kubo 0.14, I also bumped the dependency in this repo: chore: go-libp2p v0.21.0 #21

@TheDiscordian given that Kubo 0.14 has the updated go-libp2p already, one should be able to connect to relay via /dns just fine:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A success

Can you confirm things work on your end, and if so, close this issue?

This updates to https://github.com/libp2p/go-libp2p/releases/tag/v0.21.0 Tbd if this is enough to help with #18

lidel · 2022-08-05T19:50:46Z

@TheDiscordian after JS part is shipped in JS-IPFS (ipfs/js-ipfs#4178), please make sure you update to libp2p-relay-daemon v0.2.0 before you re-test things end-to-end. This way, we also confirm the latest go-libp2p works fine 🤞

lidel · 2022-08-08T15:42:25Z

After we confirm this is fixed, we need to add smoke/regression test that dialls /dns/.../wss from js-ipfs and Kubo (tbd if this should live in their respective repos, or in https://github.com/ipfs/interop)

This updates to https://github.com/libp2p/go-libp2p/releases/tag/v0.21.0 Tbd if this is enough to help with libp2p#18

TheDiscordian · 2022-10-04T17:02:22Z

@lidel @achingbrain this appears to be unresolved for js-ipfs, any theories?

Repo used: https://github.com/TheDiscordian/go-libp2p-relay-daemon

Code:

<!DOCTYPE html>
<html lang="en">
<head><title>js-ipfs minimal relay test</title>

<script src="https://cdn.jsdelivr.net/npm/ipfs-core@0.16.1/dist/index.min.js"></script>

<script>
var ipfs;
async function main() {
	ipfs = await window.IpfsCore.create();
	await ipfs.swarm.connect("/dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A");
}
main()
</script>
</head>
</html>

I can confirm this works fine /w Kubo. Error is the same as before /w js-ipfs, Chrome:

Firefox:

2color · 2022-10-04T17:12:44Z

I can confirm this works fine /w Kubo. Error is the same as before /w js-ipfs, Chrome:

So both Kubo and js-ipfs daemons are able to connect to /dns6/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A But js-ipfs in the browser fails to connect?

TheDiscordian · 2022-10-04T17:13:31Z

@2color I haven't actually tested the js-ipfs daemon, only the browser. However yes, Kubo definitely works fine, I can connect and see the node in my list of peers.

lidel · 2022-10-05T14:18:36Z

jsipfs daemon fails too:

$ ipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
connect 12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A success

$  jsipfs swarm connect /dns/ipfs.thedisco.zone/tcp/4430/wss/p2p/12D3KooWCyiHXACQpZxnvLTHXjFcFPPv69qPrX6svgdcmREZuS8A
(node:16151) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
fetch failed

lidel · 2022-10-05T14:20:05Z

@achingbrain I wonder if JS side of things needs to implement the /sni/ fix from libp2p/go-libp2p#1719 ?

marten-seemann · 2022-10-05T15:34:43Z

@achingbrain I wonder if JS side of things needs to implement the /sni/ fix from libp2p/go-libp2p#1719 ?

It's only used internally, although one could announce addresses containing /sni.

thibmeu · 2022-10-12T13:54:11Z

kubo fails to contact a websocket server behind a reverse proxy, should it be a ws or wss.
From my understanding, it comes from the following part in go-libp2p

// 1. initial addr is /dns/example.com/tcp/443/ws
resolvedAddrs, err := resolver.Resolve(ctx, a)
// 2. resolvedAddrs is /dns/example.com/tcp/443/ws
...
resolved, err := s.resolveAddrs(ctx, peer.AddrInfo{...})
// 3. resolved is /ip4/1.2.3.4/tcp/443/ws
// the information about the DNS host is lost, preventing a reverse proxy from directing the HTTP connection to the right host

The above is similar, but recoverable (not recovered) with wss. WSS adds /tls/sni/example.com/ws, which could be used by the websocket transport when dialing the resolved IP.

I think the multiaddr resolution is the component at fault here. /dns/example.com/tcp/443/ws is not equivalent to /ip4/1.2.3.4/tcp/443/ws. The IP is correct, but it's missing the domain name. websocket transport dial would be able to use an IP resolved by the Swarm resolver, but doesn't know the domain name to pass to the reverse proxy.

To address the issue (in go-libp2p), we could consider the following:

add an sni component in the multiaddr resolved by websocket transport, even for non wss connection. Then, use this multiaddr component as the host for the connection, while NetDial dials the IP resolved by swarm
add a dnsname component in the swarm resolved multiaddr. This would be a new multiaddr component. This dnsname should be ignored by other transports, but could be leveraged should they need to use the domain name.

I would be glad to contribute to an implementation. This caused issue on a server of mine, which cannot swarm connect with vanilla kubo. The fix I have is to use the sni component as Host (here), but this bypasses Swarm resolution.

BigLep · 2022-10-14T16:59:13Z

After we confirm this is fixed, we need to add smoke/regression test that dialls /dns/.../wss from js-ipfs and Kubo (tbd if this should live in their respective repos, or in https://github.com/ipfs/interop)

@lidel : I wish I knew the answer myself, but would you please be able to outline what tests are needed where so we don't regress on this in the future? I am then game to engage with the teams to make sure we harder ourselves here. Thanks.

p-shahi · 2022-11-02T16:45:05Z

go-libp2p v0.23.3 has been released https://github.com/libp2p/go-libp2p/releases/tag/v0.23.3 which incorporates @thibmeu 's fix for the missing HTTP Host header in /wss

lidel added a commit that referenced this issue Aug 1, 2022

chore: go-libp2p v0.21.0

134c27c

This updates to https://github.com/libp2p/go-libp2p/releases/tag/v0.21.0 Tbd if this is enough to help with #18

This was referenced Aug 1, 2022

chore: go-libp2p v0.21.0 #21

Merged

Support and test /dns/ multiformats/go-multiaddr-dns#47

Closed

lidel mentioned this issue Aug 5, 2022

deps: upgrade @multiformats/multiaddr to 10.4.0 ipfs/js-ipfs#4178

Merged

1 task

achingbrain closed this as completed in ipfs/js-ipfs#4178 Sep 7, 2022

TheDiscordian pushed a commit to TheDiscordian/go-libp2p-relay-daemon that referenced this issue Oct 4, 2022

chore: go-libp2p v0.21.0

db5d5a3

This updates to https://github.com/libp2p/go-libp2p/releases/tag/v0.21.0 Tbd if this is enough to help with libp2p#18

TheDiscordian pushed a commit to TheDiscordian/go-libp2p-relay-daemon that referenced this issue Oct 4, 2022

chore: go-libp2p v0.21.0

e7f2365

This updates to https://github.com/libp2p/go-libp2p/releases/tag/v0.21.0 Tbd if this is enough to help with libp2p#18

lidel reopened this Oct 5, 2022

thibmeu mentioned this issue Oct 12, 2022

fix: websocket transport should send host header libp2p/go-libp2p#1829

Closed

p-shahi mentioned this issue Nov 8, 2022

Release 0.17 ipfs/kubo#9319

Closed

lidel mentioned this issue Nov 15, 2022

docs: document /wss fix in 0.17 (go-libp2p 0.23.3) ipfs/kubo#9409

Merged

2color mentioned this issue Jan 27, 2023

Rewrite "Simple chat app" tutorial ipfs/ipfs-docs#1459

Closed

lidel closed this as completed Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues connecting from IPFS nodes over WSS #18

Issues connecting from IPFS nodes over WSS #18

TheDiscordian commented Jun 20, 2022 •

edited by lidel

Loading

TheDiscordian commented Jun 20, 2022

marten-seemann commented Jun 20, 2022

TheDiscordian commented Jun 20, 2022

marten-seemann commented Jun 20, 2022

aschmahmann commented Jun 21, 2022 •

edited

Loading

TheDiscordian commented Jun 21, 2022 •

edited

Loading

TheDiscordian commented Jun 21, 2022 •

edited

Loading

marten-seemann commented Jun 22, 2022

lidel commented Jun 24, 2022 •

edited

Loading

lidel commented Aug 1, 2022 •

edited

Loading

lidel commented Aug 5, 2022 •

edited

Loading

lidel commented Aug 8, 2022 •

edited

Loading

TheDiscordian commented Oct 4, 2022 •

edited

Loading

2color commented Oct 4, 2022

TheDiscordian commented Oct 4, 2022

lidel commented Oct 5, 2022

lidel commented Oct 5, 2022

marten-seemann commented Oct 5, 2022

thibmeu commented Oct 12, 2022

BigLep commented Oct 14, 2022

p-shahi commented Nov 2, 2022

Issues connecting from IPFS nodes over WSS #18

Issues connecting from IPFS nodes over WSS #18

Comments

TheDiscordian commented Jun 20, 2022 • edited by lidel Loading

TheDiscordian commented Jun 20, 2022

marten-seemann commented Jun 20, 2022

TheDiscordian commented Jun 20, 2022

marten-seemann commented Jun 20, 2022

aschmahmann commented Jun 21, 2022 • edited Loading

TheDiscordian commented Jun 21, 2022 • edited Loading

TheDiscordian commented Jun 21, 2022 • edited Loading

marten-seemann commented Jun 22, 2022

lidel commented Jun 24, 2022 • edited Loading

lidel commented Aug 1, 2022 • edited Loading

lidel commented Aug 5, 2022 • edited Loading

lidel commented Aug 8, 2022 • edited Loading

TheDiscordian commented Oct 4, 2022 • edited Loading

2color commented Oct 4, 2022

TheDiscordian commented Oct 4, 2022

lidel commented Oct 5, 2022

lidel commented Oct 5, 2022

marten-seemann commented Oct 5, 2022

thibmeu commented Oct 12, 2022

BigLep commented Oct 14, 2022

p-shahi commented Nov 2, 2022

TheDiscordian commented Jun 20, 2022 •

edited by lidel

Loading

aschmahmann commented Jun 21, 2022 •

edited

Loading

TheDiscordian commented Jun 21, 2022 •

edited

Loading

TheDiscordian commented Jun 21, 2022 •

edited

Loading

lidel commented Jun 24, 2022 •

edited

Loading

lidel commented Aug 1, 2022 •

edited

Loading

lidel commented Aug 5, 2022 •

edited

Loading

lidel commented Aug 8, 2022 •

edited

Loading

TheDiscordian commented Oct 4, 2022 •

edited

Loading