Very slow response from upstream servers with a lot of context deadline exceeded
#7357
Comments
SukkaW
changed the title
context deadline exceeded
|
Same issue, Environment: I'm stunned by the huge amount of changes between these two versions, not sure how to investigate what's going on from these |
|
@xWTF Even w/ verbose logging I can't see any useful information. So I assume the most likely cause would be the bumping of go from |
Yep, on my device the log simply says "timeout" without any additional information, appears to be a regular connection problem but curl and old version works fine. |
Updates AdguardTeam/AdGuardHome#7357. Squashed commit of the following: commit 4170865 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Nov 28 16:53:39 2024 +0300 upstream: imp docs commit 0e89c40 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Nov 28 15:57:38 2024 +0300 upstream: doh close iddle conns
|
@schzhn Hello, I see there is a commit mentioning the issue. Is there a nightly release I can download and help you test? |
Updates #7357. Squashed commit of the following: commit 599dee0 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Fri Nov 29 14:52:16 2024 +0300 all: upd golibs commit eb90c49 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Fri Nov 29 14:32:57 2024 +0300 all: upd proxy commit 12b8e51 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Nov 28 20:04:58 2024 +0300 all: upd proxy commit 77fcabc Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Nov 28 17:13:30 2024 +0300 all: upd chlog commit 714f93d Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Nov 28 17:02:34 2024 +0300 all: upd proxy
|
The edge release contains several bug fixes. You can also download the binary for your platform from the following link |
|
@schzhn Thanks for your reply! However, I can still re-produce the issue by using Here is how I started Here is what happened I tried to query And here is the log output of Also to make sure my network is working, I have tried sukka@sukka-pi:~ $ time dig example.com @120.53.53.53 +tls +short
93.184.215.14
real 0m0.121s
user 0m0.060s
sys 0m0.013s
sukka@sukka-pi:~ $ time dig example.com @120.53.53.53 +https +short
93.184.215.14
real 0m0.083s
user 0m0.023s
sys 0m0.012s
sukka@sukka-pi:~ $ time dig example.com @1.12.12.12 +tls +short
93.184.215.14
real 0m0.134s
user 0m0.051s
sys 0m0.020s
sukka@sukka-pi:~ $ time dig example.com @1.12.12.12 +https +short
93.184.215.14
real 0m0.130s
user 0m0.051s
sys 0m0.030s |
|
Same problem, when using doh.pub, there is a long waiting time, and a large number of timeouts appear in the log, which may be related to compatibility adgh.log It might be related to TLS version support. DNSPod does not support TLS 1.3. Could it be that an issue in the fallback handling is causing significant delays? Using WireGuard to capture packets with dnsproxy reveals a large number of handshake failures. 
|
|
@Skyxim The handshake failure within your pcap appears to be a MTU issue,
|
|
Well, I've confirmed this is indeed an MSS issue (see the additional comments at bottom). I never thought about this possibility before. In AGH 0.73.2, the TLS client hello is a classic 252 bytes, which is short enough to be sent without any issue.
However, in 0.73.3+, a
The change is related to the Golang version bump, as we guessed previously. See here for the relevant changelog. I can confirm this by setting
This strange MSS issue only occurs to Tencent DoH servers ( The maximum possible MSS to Tencent DoH servers is 8 bytes smaller (1444) than MSS advertised from their side (1452). I suspect that Tencent engineers may have misconfigured their network while setting up TOA (which are commonly used to pass the real IP from a load balancer to VMs/pods), since the TOA header is 8 bytes. My current solution is to clamp MSS from these two IPs using mangle rules:
This is, unfortunately, not an easy to implement solution for most home grade routers. If you're running AGH on a Linux system, you can do this via iptables and clamp on incoming packets. You may also set the |
OK, it seems that my problem is not related to this issue. Turning off tlskyber is indeed effective. Thanks for your help |
Sorry for the confusion caused by the previous reply. It is indeed the same problem. The MSS configuration error occurs on Tencent's side, not on our local side. You may want to check the edited version of my second comment for more details. |
Yes, I saw that reply as well. I will also try to report it to Tencent. Thanks again! |
|
Similar issue: hashicorp/terraform-provider-aws#39311 cc @xWTF That issue is caused by the AWS Firewall dropping over stretched TLS client hello (also caused by Go 1.23), resulting in Go 1.23 no longer being able to access AWS API. The true cause is not the same (badly made firewall vs wrong MSS), but the root cause (Go 1.23 w/ overstretched TLS client hello) is the same. I have submitted a report on V2EX: https://www.v2ex.com/t/1094021 |
|
Created a few social posts to raise the concern:
|
|
It appears that Tencent has updated the TLS configuration of their DNS servers, as I can no longer reproduce the issue on my setup. Thank you all for reporting, testing, and investigating this matter. |
Prerequisites
I have checked the Wiki and Discussions and found no answer
I have searched other issues and found no duplicates
I want to report a bug and not ask a question or ask for help
I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)
Platform (OS and CPU architecture)
Linux, ARM64
Raspberry Pi 4
sukka@sukka-pi:/tmp $ uname -a Linux sukka-pi 6.6.51+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/LinuxInstallation
GitHub releases or script from README
Setup
On one machine
AdGuard Home version
v0.107.53
Action
Replace the following command with the one you're calling or a
description of the failing action:
Expected result
The result returned immdiately.
Actual result
The result returned in about 20 seconds.
Additional information and/or screenshots
I have set following upstream servers in the
DNS settings:But when I use the
Test upstreamsbutton, AdGuardHome says all those servers could not be used:I have tested those servers locally, they return the result very fast, so definitely no internet connectivity issue, and also there is no problem with the upstream servers:
I have also tried with
curl:I then enabled
log.verbosein the config file (AdGuardHome.yaml), and I got those logs:Also average upstream servers response time:
So somehow AdGuardHome failed to connect to upstream servers while
digworks just fine.The text was updated successfully, but these errors were encountered: