Websockets keep disconnecting

[geekgonecrazy]

1. What version of Caddy are you running (caddy -version)?

0.9.5

2. What are you trying to do?

Reverse proxy in front of an application with websockets

3. What is your entire Caddyfile?

http://chat.myserver.io {
        redir 301 {
                / https://{host}{uri}
        }
}

https://chat.myserver.io {
        proxy / 127.0.0.1:3000 {
                transparent
                websocket
        }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

Command:
sudo sh -c "ulimit -n 65535 && sudo /usr/local/bin/caddy -agree -log=stdout -conf=/etc/caddy/Caddyfile"

OS:

$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

5. Please paste any relevant HTTP request(s) here.

Was seeing this output frequently:

2017/04/04 13:55:44 http2: server: error reading preface from client 50.235.7.62:7630: 
read tcp <myserverip>:443->50.235.7.62:7630: i/o timeout
2017/04/04 13:55:44 http: TLS handshake error from 50.235.7.62:7631: read tcp <myserverip>:443->50.235.7.62:7631: i/o timeout

6. What did you expect to see?

7. What did you see instead (give full error messages and/or log)?

Websockets would disconnect about every 3-4 seconds indefinetely. Not matter how many times I restarted it.

8. How can someone who is starting from scratch reproduce the bug as minimally as possible?

My setup is: caddy with the above config which reverse proxy to a docker container running rocket.chat.

docker run -d --name db mongo
docker run -d --link db:db -e MONGO_URL=mongodb://db:27017/rocketchat -e ROOT_URL=https://chat.myserver.io --name rocketchat -p 3000:3000 rocketchat/rocket.chat:0.54.2

You'll see the connection banner flash every few seconds, if you open the dev console on browser you'll see the socket drop and come back every few seconds.

If I downgrade to 0.9.4 it works fine.

[elcore]

Hello @geekgonecrazy,

thank you for your report!

You need to extend the default timeouts or disable them

timeouts none

It is really important to read our release notes (https://github.com/mholt/caddy/releases/tag/v0.9.5)

P.S.: "default timeouts are going to be disabled in the next version (they already are on master)" Matt H.

[geekgonecrazy]

@elcore ah I remember seeing this now. Might want to consider adding to notes that this effects websockets too. I realize its just an upgraded http request, but still would help make this clearer.

Thanks for the quick response 👍

[elcore]

Thank you very much @geekgonecrazy!!!

Might want to consider adding to notes that this effects websockets too.

"[...] timeouts apply to an entire HTTP server which may serve multiple sites defined in your Caddyfile, the timeout values for each site will be reduced to their minimum values (with 0, or disabled, being the lowest) across all sites that share that server."

https://caddyserver.com/docs/timeouts

[mholt]

@geekgonecrazy The next release of Caddy will not have default timeouts.

[geekgonecrazy]

@elcore thanks for the link to docs.

I definitely understand the need for timeouts. Waiting around for too long is just wasting resources. No complaints for helping me save resources

I went ahead and disabled timeouts on mine.

But.. Per defaults I'm still unsure which would be effecting my websockets.

Read: 10s - this being a websocket is it expecting to handle all frames regardless in this amount of time before hanging up? Since the request is upgraded and happens in less then this.. I would assume does not apply.
Read Header: 10s - this one I can't see really effecting websockets much.
Write: 20s - again is this 20 seconds to write all frames?
Idle: 2m - this is the one after being reminded of this change(I saw a tweet when it was being made 👍) I assumed would effect websockets. But it never waited for 2 minutes. It has a ping/pong frequency of like 10-15seconds. So it was never idle this long.

@mholt I assume this is because of tickets like mine. Is there a ticket you have discussed remove vs improve defaults? Would be curious as to the decision.

I know most other web server / proxies have defaults. So this definitely isn't new to have them. Usually we only hit them when we are doing very edge case scenarios like servers that take too long to respond etc.

Thanks again for the responses guys! We love caddy over on the Rocket.Chat project. Our users love it too.

[mholt]

@geekgonecrazy Glad to hear, thank you. :)

Default timeouts were removed in part because, yes, issues of confusion (partly my fault for documentation fails, but it was very clear in the release notes) - despite the security benefits. Go doesn't let us customize timeouts per-request (yet, there are issues about it so maybe for 1.9 or 1.10) and since we lose that kind of flexibility, maybe best to turn them off by default. At least for now.

I didn't realize Rocket.Chat had docs for using it with Caddy. That's cool. Is there a reason this section is in your docs? https://rocket.chat/docs/installation/manual-installation/ubuntu/snaps/autossl#redirecting-http-to-https - Caddy does this for you automatically, no need to manually create a redirect (unless you are redirecting differently than Caddy does).

(To answer your question about which limits cause the disconnect, Idle doesn't apply until Go 1.8, which Caddy 0.9.5 is not built on, but the next version will be. I'm guessing you're hitting the Read or Write timeouts. I'm not actually sure which.)

[geekgonecrazy]

Go doesn't let us customize timeouts per-request

Ah! Ok that makes sense. Only done basic rest API's in go, generally leveraging gin never going that low.. So was unaware of that limitation.

I didn't realize Rocket.Chat had docs for using it with Caddy. That's cool.

We use it in our snaps trying to make it simple to get full setup going. So simplicity of caddy fit right in
I believe we also have directions for manual installs as well.

Caddy does this for you automatically, no need to manually create a redirect (unless you are redirecting differently than Caddy does).

I think that was oversight. Came from nginx where such verbose syntax is required. 😁 I'll update and simplify! Thanks!