-
Notifications
You must be signed in to change notification settings - Fork 477
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High latency and jitter when using rathole for a Minecraft Java edition server #171
Comments
Hey, @draand28 Thanks for writing the report. I have some questions about your description.
I wonder how this case is set up. If you have a server behind the NAT, how can you use haproxy only to access it from the Internet and have ping metrics? NAT traversal via relaying has essentially more delay than a direct connection. Also, you have multiple services in your config. Which is used by the server (TCP, or UDP, or both)? |
The minecraft server uses TCP only, port 25565. The "ping" is not really a ping, no ICMP. It is a built-in plugin for the Minecraft server, that measures every few seconds the latency between the client and the server, probably by sending a TCP minecraft packet and waiting for a response. The current setup The also tried setup, but same latency/jitter The setup with minimum latency and jitter If you have any more questions, I'll answer you asap. |
were you able to make normal addresses for the players on the server? |
how did you make UDP support on haproxy? |
Have you tried without nodelay? I understand that does reduce latency but heavily throttles bandwidth. That balance may not favor MC Java edition. I'm running an MC server perfectly fine over Rathole without nodelay. Moved from FRP this morning. |
the thread owner already had TCP_NODELAY enabled. Look at his configuration. @rapiz1 is it possible that the thread pool is to small? Line 34 in bf842b4
|
Yes. However, the client also needs to set up that. It's tedious so the latest version turns that on by default.
No. It only effects the delay of opening a connection. |
ok i haven't seen this... my fault.
How much delay are we talking about? in the screenshot you see 70ms worst and 23ms best case. Difference is 47 ms. This can't depend on routing latencies from rathole server<-> client. The time this "opening a connection" consumes should be part of the whole time of the ping, right? |
Hmm, I didn't look closely but I assume it's a ping-pong in an established TCP long connection. so opening up isn't part of the delay. That's what I would implement ping in the game protocol but I don't know if it's the case. |
ok. so your assumption is:
Any idea to tweak my configuration? TCP_NODELAY is already enabled. |
Pool size is exactly aimed to reduce latency in your case. It caches connections for later use. However, I didn't exposed it to config because it's too deep into the implementation and I don't think it's an elegant solution. But you can make it bigger and recompile if needed.
I'm aware of the spike and have poured some effort to investigate. The best guess I have now is the tokio scheduler isn't particularly good at minimizing the delay. I have the intention to further optimize around that. I prefer solving the issue other than mitigating it with pool size.
…---Original---
From: ***@***.***>
Date: Thu, Dec 8, 2022 21:45 PM
To: ***@***.***>;
Cc: "Yujia ***@***.******@***.***>;
Subject: Re: [rapiz1/rathole] High latency and jitter when using rathole for aMinecraft Java edition server (Issue #171)
The time this "opening a connection" consumes should be part of the whole time of the ping, right?
Hmm, I didn't look closely but I assume it's a ping-pong in an established TCP long connection. so opening up isn't part of the delay. That's what I would implement ping in the game protocol but I don't know if it's the case.
ok. so your assumption is:
same connection -> fast
new connection -> slow/with hickups?
I have a similar problem (not with minecraft server) with many short connections (~ 3-4/sec) from different sources. I have several spikes in TCP connections (21ms - 110ms). A PING to server/client is constantly low (~ 20 ms).
Any idea to tweak my configuration? TCP_NODELAY is already enabled.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I tried rising the pool size to 128. It didn't help. There are still spikes :(
I would be happy if you make an optimization here. My rust skills are not very good, so I don't know how to support here other than testing. |
Have you fixed this issue? Is the decision in progress? |
how did you turn on the "TCP/Noise transport" feature? |
Describe the bug
I am using this absolutely incredible peice of software (rathole) - really I am amazed by how simple and effective it is.
I have a single issue with it. Latency and jitter.
When I am using haproxy, I get 10-15 max ping, very stable.
When using rathole, the ping is allover the place, from 4 to 88, even with nodelay enabled.
I now use haproxy mid-way in order to keep player's IP with Proxy protocol, but ignore that, it works just as jittery without haproxy at all, tested it many times already.
I should mention that the ping nor the jitter is immediately noticeable (it is not bad at all, just nowhere near perfection, like haproxy).
To Reproduce
Steps to reproduce the behavior:
Configuration
Configuration used to reproduce the behavior:
server.toml
[server]
bind_addr = "0.0.0.0:2333" #
2333
specifies the port that rathole listens for clientsdefault_token = "SOMEPASS"
[server.transport]
type = "tcp"
[server.transport.tcp]
nodelay = true
[server.services.http]
bind_addr = "0.0.0.0:80"
type = "tcp"
[server.services.https]
bind_addr = "0.0.0.0:443"
type = "tcp"
[server.services.javatcn]
bind_addr = "0.0.0.0:25566"
type = "tcp"
nodelay = true
[server.services.ademc]
bind_addr = "0.0.0.0:1400"
type = "tcp"
[server.services.bedrocktcn]
bind_addr = "0.0.0.0:19132"
type = "udp"
Logs
CLIENT LOGS
root@serverhp:~/rathole# ./rathole client.toml Jul 28 03:48:35.578 INFO handle{service=bedrocktcn}: rathole::client: Starting 643aad576114f32b98d1673947d21e6a142e9685c1ff456b8e93831ca9c1fc0e Jul 28 03:48:35.578 INFO handle{service=javatcn}: rathole::client: Starting 6c138220128ab0992cab4654559c576b918012089529c6cab4af7bdb1fdbf5fd Jul 28 03:48:35.578 INFO handle{service=https}: rathole::client: Starting 3e1943522517e81328e3f3f7ea1f0d7fc2ca886dfe1639a7fa6508760d1cee3a Jul 28 03:48:35.578 INFO handle{service=http}: rathole::client: Starting e0603c499aae47eb89343ad0ef3178e044c62e70ae2309b35591d1d49a3211ec Jul 28 03:48:35.578 INFO handle{service=ademc}: rathole::client: Starting 0ad444b5d6a849be50583984d773445ea97bd0537137e2575c22388b08e05b01 Jul 28 03:48:35.578 INFO config_watcher{path="client.toml"}: rathole::config_watcher: Start watching the config Jul 28 03:48:35.584 INFO handle{service=https}:run: rathole::client: Control channel established Jul 28 03:48:35.610 INFO handle{service=ademc}:run: rathole::client: Control channel established Jul 28 03:48:35.615 INFO handle{service=javatcn}:run: rathole::client: Control channel established Jul 28 03:48:35.615 INFO handle{service=bedrocktcn}:run: rathole::client: Control channel established Jul 28 03:48:35.618 INFO handle{service=http}:run: rathole::client: Control channel established
SERVER LOGS
root@vps95107:~/rathole# ./rathole server.toml Jul 28 00:48:31.869 INFO config_watcher{path="server.toml"}: rathole::config_watcher: Start watching the config Jul 28 00:48:31.869 INFO rathole::server: Listening at 0.0.0.0:2333 Jul 28 00:48:35.692 INFO connection{addr=PERSONAL_IP:55612}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.694 INFO connection{addr=PERSONAL_IP:55612}: rathole::server: Control channel established service=https Jul 28 00:48:35.694 INFO connection{addr=PERSONAL_IP:55612}:handle{service=https}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:443 Jul 28 00:48:35.710 INFO connection{addr=PERSONAL_IP:53645}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.713 INFO connection{addr=PERSONAL_IP:57378}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.713 INFO connection{addr=PERSONAL_IP:51862}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.715 INFO connection{addr=PERSONAL_IP:23872}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.721 INFO connection{addr=PERSONAL_IP:53645}: rathole::server: Control channel established service=ademc Jul 28 00:48:35.721 INFO connection{addr=PERSONAL_IP:53645}:handle{service=ademc}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:1400 Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:57378}: rathole::server: Control channel established service=javatcn Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:57378}:handle{service=javatcn}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:25566 Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:51862}: rathole::server: Control channel established service=bedrocktcn Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:51862}:handle{service=bedrocktcn}:run_udp_connection_pool: rathole::server: Listening at 0.0.0.0:19132 Jul 28 00:48:35.728 INFO connection{addr=PERSONAL_IP:23872}: rathole::server: Control channel established service=http Jul 28 00:48:35.728 INFO connection{addr=PERSONAL_IP:23872}:handle{service=http}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:80
Environment:
Ubuntu 18.04.6 for the Public IP server
Ubuntu 20.04 for the behind CGNAT, dynamic IP server
rathole --version
output:root@serverhp:~/rathole# ./rathole --version rathole Build Timestamp: 2022-05-23T12:35:49.532232791+00:00 Build Version: 0.4.2 Commit SHA: Some("ee39a8e31e3fe26e58c01ecdfaa646b187f25b79") Commit Date: Some("2022-05-23T12:31:03+00:00") Commit Branch: Some("detached HEAD") cargo Target Triple: x86_64-unknown-linux-gnu cargo Profile: release cargo Features: base64,client,default,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls
The text was updated successfully, but these errors were encountered: