运行一段时间后会出现无法连接的情况

[zreo0]

docker-compose 部署，host 模式，仅使用递归DNS功能，同一台机器上另一个pod中会指定 nameserver 为宿主机 ip。
启动之后能正常运行，但是运行一段时间后就会直接无法连接，然后使用 test.sh 结果如下：

*********************************************************************************

images build time : 2024-08-28 16:12:23 UTC
check for the latest version ,
go to https://github.com/kkkgo/PaoPaoDNS/discussions 
-> test start `1728637613`
```rust
UNBOUND MODE TEST.
yyyyCN-53 failed:;; communications error to 127.0.0.1#53: connection refused ;; communications error to 127.0.0.1#53: connection refused ;; communications error to 127.0.0.1#53: connection refused ; <<>> DiG 9.18.28 <<>> www.taobao.com @127.0.0.1 -p53 A +short ;; global options: +cmd ;; no servers could be reachedy
[INFO] TEST FAIL.❌
```
-> test end `1728637614`

*********************************************************************************

奇怪的是，使用 debug.sh 运行后，忽然就再次恢复正常了，此时再运行 test.sh 就是通过的。然后隔了几小时再次重复同样的情况。
不确定这时候 debug.sh 的内容是否还有效，求排查思路。

debug.sh 内容如下，ip 信息用 xx.xx.xx.xx 代替：

### == debug.sh : docker exec -it paopaodns sh ==
-> debug start `1728637621`
```rust
[INFO] images build time : 2024-08-28 16:12:23 UTC
[OK]DATA_writeable
[OK]DATA_readable
[INFO] NETWORK
*********************************************************************************

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP qlen 1000
    inet 172.16.187.185/16 brd 172.16.255.255 scope global ens18
    inet6 fe80::be24:11ff:fe0d:9bc1/64 scope link 
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
    inet6 fe80::42:38ff:feda:46de/64 scope link 
108: br-f006b38e2df6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-f006b38e2df6
    inet6 fe80::42:8cff:fef5:286/64 scope link 
116: veth801208a@if115: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue master docker0 state UP 
    inet6 fe80::a448:e7ff:fe6a:9c2f/64 scope link 
default via 172.16.0.1 dev ens18 onlink 
172.16.0.0/16 dev ens18 scope link  src 172.16.187.185 
172.17.0.0/16 dev docker0 scope link  src 172.17.0.1 
172.18.0.0/16 dev br-f006b38e2df6 scope link  src 172.18.0.1 
traceroute to 120.53.53.53 (120.53.53.53), 4 hops max, 46 byte packets
 1  172.16.0.1 (172.16.0.1)  0.241 ms  *  0.910 ms
 2  110.42.111.1 (110.42.111.1)  26.313 ms  6.400 ms  3.400 ms
 3  *  *  *
 4  *  *  *
PING 223.5.5.5 (223.5.5.5): 56 data bytes
64 bytes from 223.5.5.5: seq=0 ttl=111 time=11.574 ms

--- 223.5.5.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 11.574/11.574/11.574 ms
PING 119.29.29.29 (119.29.29.29): 56 data bytes
64 bytes from 119.29.29.29: seq=0 ttl=44 time=37.541 ms

--- 119.29.29.29 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 37.541/37.541/37.541 ms
Server:         223.5.5.5
Address:        223.5.5.5#53

Non-authoritative answer:
www.taobao.com  canonical name = www.taobao.com.danuoyi.tbcache.com.
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 61.174.8.55
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 61.174.8.56
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 122.228.15.216
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 122.228.15.215
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 240e:f7:e000:1200:3::20
Name:   www.taobao.com.danuoyi.tbcache.com
Address: 240e:f7:e000:1200:3::21

Server:         119.29.29.29
Address:        119.29.29.29#53

Non-authoritative answer:
www.qq.com      canonical name = ins-r23tsuuf.ias.tencent-cloud.net.
Name:   ins-r23tsuuf.ias.tencent-cloud.net
Address: 101.91.22.57
Name:   ins-r23tsuuf.ias.tencent-cloud.net
Address: 101.91.42.232
Name:   ins-r23tsuuf.ias.tencent-cloud.net
Address: 240e:e1:a800:120::76
Name:   ins-r23tsuuf.ias.tencent-cloud.net
Address: 240e:e1:a800:120::36

*********************************************************************************

[INFO] ENV
*********************************************************************************

====ENV TEST====
[OK]DATA_writeable-
[OK]DATA_readable-
MEM:1500m 3000m 10000000 4500mb
prefPC:100
CORES:-12-
POWCORES:-8-
ulimit :-1048576-
FDLIM :-4096-
TZ:-Asia/Shanghai-
UPDATE:-weekly-
DNS_SERVERNAME:-PaoPaoDNS-
SERVER_IP:-xx.xx.xx.xx-
ETHIP:-172.16.187.185-
DNSPORT:-53-
SOCKS5:-IP:PORT-
CNAUTO:-no-
IPV6:-no-
CNFALL:-yes-
CUSTOM_FORWARD:-IP:PORT-
AUTO_FORWARD:-no-
AUTO_FORWARD_CHECK:-yes-
USE_MARK_DATA:-yes-
RULES_TTL:-0-
CUSTOM_FORWARD_TTL:-0-
SHUFFLE:-no-
EXPIRED_FLUSH:-yes-
CN_TRACKER:-yes-
USE_HOSTS:-no-
HTTP_FILE:-no-
SAFEMODE:-no-
QUERY_TIME:-2000ms-
ADDINFO:-no-
PLATFORM:-Linux 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64 Linux-
====ENV TEST====
mosdns kkkgo/mosdns:240822.1
total 84K    
   4.0K drwxrwxrwx    2 0        0           4.0K Oct 11 15:51 .
   4.0K drwxr-xr-x    1 0        0           4.0K Oct 11 15:51 ..
   4.0K -rwxrwxrwx    1 0        0            233 Oct 11 12:29 custom_env.ini
   4.0K -rwxrwxrwx    1 0        0            416 Oct 11 12:29 custom_mod.yaml
   8.0K -rw-r--r--    1 0        0           5.7K Oct 11 15:51 redis.conf
   4.0K -rwxrwxrwx    1 0        0           3.6K Oct 11 14:29 redis_dns_v2.rdb
  52.0K -rw-r--r--    1 0        0          50.9K Oct 11 15:51 unbound.conf
   4.0K -rwxrwxrwx    1 0        0            289 Oct 11 12:29 unbound_custom.conf
*********************************************************************************

[INFO] PS
*********************************************************************************

PID   USER     TIME  COMMAND
    1 root      0:00 {init.sh} /bin/sh /usr/sbin/init.sh
   23 root      0:00 crond
   56 root      0:08 redis-server unixsocket:/tmp/redis.sock
  124 root      0:00 {watch_list.sh} /bin/sh /usr/sbin/watch_list.sh
  125 root      0:00 {data_update.sh} /bin/sh /usr/sbin/data_update.sh
  128 root      0:00 tail -f /dev/null
  132 root      0:00 inotifywait -e modify,delete /etc/unbound/named.cache
  146 root      0:01 apk update
  357 root      0:00 {debug.sh} /bin/sh /usr/sbin/debug.sh
  379 root      0:00 ps -ef
Try to run unbound...
        num-threads: 12
[1728637633] unbound[383:0] notice: Start of unbound 1.21.1.
        outgoing-range: 8192
#CNAUTO outgoing-range: 8192
#lowrmem        outgoing-range: 60
                        outgoing-num-tcp: 4096
#CNAUTO                 outgoing-num-tcp: 4096
                        incoming-num-tcp: 4096
#CNAUTO                 incoming-num-tcp: 4096
        msg-cache-size: 1500m
        msg-cache-slabs: 8
        num-queries-per-thread: 341
#CNAUTO num-queries-per-thread: 341
#lowrmem num-queries-per-thread: 30
        rrset-cache-size: 3000m
        rrset-cache-slabs: 8
[1728637633] unbound[383:0] notice: init module 0: cachedb
[1728637633] unbound[383:0] notice: Redis initialization
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: Connection to Redis established
[1728637633] unbound[383:0] notice: init module 1: iterator
RealCore:12
ulimit:1048576
*********************************************************************************

[INFO] TOP
*********************************************************************************

CPU:   3% usr  10% sys   0% nic  85% idle   0% io   0% irq   0% sirq
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  383   357 root     R     121m   0%   9  15% unbound -c /tmp/unbound_raw.conf -
  146   125 root     S    40324   0%   3   0% apk update
   56     1 root     S    22784   0%   2   0% redis-server unixsocket:/tmp/redis
  124     1 root     S     1696   0%   7   0% {watch_list.sh} /bin/sh /usr/sbin/
  125     1 root     S     1656   0%  10   0% {data_update.sh} /bin/sh /usr/sbin
    1     0 root     S     1644   0%   8   0% {init.sh} /bin/sh /usr/sbin/init.s
  357     0 root     S     1640   0%   4   0% {debug.sh} /bin/sh /usr/sbin/debug
  387   357 root     R     1636   0%   1   0% top -n1
  128     1 root     S     1628   0%   4   0% tail -f /dev/null
  388   357 root     S     1624   0%   8   0% grep %
  132   124 root     S     1084   0%  10   0% inotifywait -e modify,delete /etc/
   23     1 root     S      872   0%   9   0% crond
*********************************************************************************

[INFO] REDIS
*********************************************************************************

used_memory_human:1.11M
used_memory_rss_human:4.24M
used_memory_peak_human:1.11M
total_system_memory_human:23.48G
used_memory_lua_human:31.00K
used_memory_vm_total_human:63.00K
used_memory_scripts_human:181B
maxmemory_human:4.39G
20
*********************************************************************************

[TEST] IP ROUTE
*********************************************************************************

CN IP URL:
-
[1728637637] unbound[383:0] info: start of service (unbound 1.21.1).
xx.xx.xx.xx
--
xx.xx.xx.xx
CN RAW-IP URL:
xx.xx.xx.xx
------------------
Non-CN IP URL:
-
--
xx.xx.xx.xx
Non-CN RAW-IP URL:
xx.xx.xx.xx
-
xx.xx.xx.xx
--
xx.xx.xx.xx
---
xx.xx.xx.xx
------------------
IP INFO:
xx.xx.xx.xx
CN,undefined,undefined
ASN136188/NINGBO, ZHEJIANG Province, P.R.China.
HTTP/1.1 
Mozilla/5.0 Gecko/20100101 Firefox/120.0 https://github.com/kkkgo/PaoPaoDNS
Asia/Shanghai Time: 10/11/2024, 5:07:46 PM
[INFO] force_recurse_list
grep: /data/force_recurse_list.txt: No such file or directory
MOSDNS WHOAMI :
MOSDNS akahelp: "ns" "xx.xx.xx.xx"
MOSDNS 03k: xx.xx.xx.xx
UNBOUND WHOAMI:
UNBOUND akahelp: ;; communications error to 127.0.0.1#5301: connection refused
;; communications error to 127.0.0.1#5301: connection refused
;; communications error to 127.0.0.1#5301: connection refused
;; no servers could be reached

UNBOUND 03k: ;; communications error to 127.0.0.1#5301: connection refused
;; communications error to 127.0.0.1#5301: connection refused
;; communications error to 127.0.0.1#5301: connection refused
;; no servers could be reached

*********************************************************************************

[TEST] HIJACK
*********************************************************************************

;; communications error to 9.8.7.5#53: timed out
;; no servers could be reached

;; communications error to 9.8.7.6#53: timed out
;; no servers could be reached

HIJACK 127.0.0.1 = 58.217.249.177
*********************************************************************************

[TEST] DIG-CN [taobao]
*********************************************************************************

MOSDNS CN:
www.taobao.com.danuoyi.tbcache.com.
122.228.15.216
122.228.15.215
UNBOUND CN:
unbound_raw FAILED.
*********************************************************************************

[TEST] DIG-NOCN [youtube]

[kkkgo]

看起来是unbound进程死掉了，运行debug的话检测到没有unbound 进程会尝试运行看看有什么报错信息，但看起来运行并没有报错。
是运行固定的时间后就会无法连接？具体unbound 死掉的原因可能要看docker logs才知道。

[zreo0]

目前也还在排查中，正在压测看看什么情况下会挂掉。
主要是挂了之后一会儿就恢复了，不确定是怎么恢复的，两次都是在运行 debug.sh 之后就自己恢复了……所以当前只有一个已知信息是都是运行了一段时间，大约3h， pod 占用的内存不断增加，挂掉恢复之后内存释放了不少，正在跑看看是不是被杀了（限制了6G），但是看 pod 也没有发生过重启。
如果再次挂掉有什么优先需要获取的信息吗？

[kkkgo]

运行 debug.sh会尝试拉起挂掉的unbound进程所以恢复了。
你可以看看docker logs。
你限制的是6G吗？我看调试信息，他认为自己得到了16G以上的可用内存。

[zreo0]

是的，正好目前跑了一下，当前正好到临界值了

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O      PIDS
c30e302fe6df   PaoPaoDNS        1.18%     6GiB / 6GiB         99.99%    0B / 0B           102kB / 12MB   23

然后下一秒就是

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O      PIDS
c30e302fe6df   PaoPaoDNS        0.18%     37.29MiB / 6GiB       0.61%     0B / 0B           106kB / 12MB   11

然后这时候去使用 test.sh 就是上面的报错。

[kkkgo]

似乎是因为内核原因没有正确地识别到真实的可用内存。
你可以进入容器后cat /proc/meminfo看看容器认为自己得到了多少内存。
可以尝试给容器强制指定内存容量试试：
#129

[zreo0]

感谢。现在使用 lxcfs 进行了对应信息的挂载，目前容器内能够获取到配置时候限制的内存了：

docker exec 582a979a44a4 cat /proc/meminfo
MemTotal:        6291456 kB
MemFree:         5876976 kB
MemAvailable:    5876976 kB

现在准备继续压一下看看如果临界是否正常。
不过好奇一点，发生 OOM 之后的重启为什么没有正常拉起服务呢？OOM 之后我尝试手动 restart ，这样的操作是可以正常运行的。