Clear DNS configuration received from DHCP during networking reconfiguration in Linux. #13516
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liat-grozovik
approved these changes
Jan 29, 2023
|
@qiluo-msft could you please review or assign someone? |
kellyyeh
approved these changes
Jan 30, 2023
|
@oleksandrivantsiv PR conflicts with 202205 branch |
stepanblyschak
added a commit
to stepanblyschak/sonic-buildimage
that referenced
this pull request
Feb 3, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR sonic-net#13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
8 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Feb 6, 2023
…uration in Linux. (sonic-net#13516) - Why I did it fixes sonic-net#12907 When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared. - How I did it Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration. - How to verify it Ensure that the management interface has no static configuration. Check that /etc/resolv.conf has DNS configuration. Configure a static IP address on the management interface. Verify that /etc/resolv.conf has no DNS configuration. Remove the static IP address from the management interface. Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
|
Cherry-pick PR to 202211: #13686 |
oleksandrivantsiv
added a commit
to oleksandrivantsiv/sonic-buildimage
that referenced
this pull request
Feb 7, 2023
…uration in Linux. (sonic-net#13516) - Why I did it fixes sonic-net#12907 When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared. - How I did it Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration. - How to verify it Ensure that the management interface has no static configuration. Check that /etc/resolv.conf has DNS configuration. Configure a static IP address on the management interface. Verify that /etc/resolv.conf has no DNS configuration. Remove the static IP address from the management interface. Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
yxieca
pushed a commit
that referenced
this pull request
Feb 10, 2023
…uration in Linux. (#13516) (#13695) - Why I did it fixes #12907 When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared. - How I did it Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration. - How to verify it Ensure that the management interface has no static configuration. Check that /etc/resolv.conf has DNS configuration. Configure a static IP address on the management interface. Verify that /etc/resolv.conf has no DNS configuration. Remove the static IP address from the management interface. Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
liat-grozovik
pushed a commit
that referenced
this pull request
Feb 14, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR #13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
mssonicbld
pushed a commit
that referenced
this pull request
Feb 16, 2023
…uration in Linux. (#13516) - Why I did it fixes #12907 When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared. - How I did it Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration. - How to verify it Ensure that the management interface has no static configuration. Check that /etc/resolv.conf has DNS configuration. Configure a static IP address on the management interface. Verify that /etc/resolv.conf has no DNS configuration. Remove the static IP address from the management interface. Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
mssonicbld
added
Included in 202211 Branch
and removed
Approved for 202211 Branch
Created PR to 202211 Branch
labels
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Feb 17, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR sonic-net#13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Feb 21, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR sonic-net#13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
mssonicbld
pushed a commit
that referenced
this pull request
Feb 21, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR #13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
mssonicbld
pushed a commit
that referenced
this pull request
Feb 22, 2023
Go's runtime (and dockerd inherits this) uses own DNS resolver implementation by default on Linux. It has been observed that there are some DNS resolution issues when executing ```docker pull``` after first boot. Consider the following script: ``` admin@r-boxer-sw01:~$ while :; do date; cat /etc/resolv.conf; ping -c 1 harbor.mellanox.com; docker pull harbor.mellanox.com/sonic/cpu-report:1.0.0 ; sleep 1; done Fri 03 Feb 2023 10:06:22 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.99 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.989/5.989/5.989/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:57245->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:23 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.56 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.561/5.561/5.561/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:53299->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:24 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.78 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.783/5.783/5.783/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:55765->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:25 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=7.17 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.171/7.171/7.171/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:44877->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:26 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=5.66 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.656/5.656/5.656/0.000 ms Error response from daemon: Get "https://harbor.mellanox.com/v2/": dial tcp: lookup harbor.mellanox.com on [::1]:53: read udp [::1]:54604->[::1]:53: read: connection refused Fri 03 Feb 2023 10:06:27 AM UTC nameserver 10.211.0.124 nameserver 10.211.0.121 nameserver 10.7.77.135 search mtr.labs.mlnx labs.mlnx mlnx lab.mtl.com mtl.com PING harbor.mellanox.com (10.7.1.117) 56(84) bytes of data. 64 bytes from harbor.mtl.labs.mlnx (10.7.1.117): icmp_seq=1 ttl=53 time=8.22 ms --- harbor.mellanox.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 8.223/8.223/8.223/0.000 ms 1.0.0: Pulling from sonic/cpu-report 004f1eed87df: Downloading [===================> ] 19.3MB/50.43MB 5d6f1e8117db: Download complete 48c2faf66abe: Download complete 234b70d0479d: Downloading [=========> ] 9.363MB/51.84MB 6fa07a00e2f0: Downloading [==> ] 9.51MB/192.4MB 04a31b4508b8: Waiting e11ae5168189: Waiting 8861a99744cb: Waiting d59580d95305: Waiting 12b1523494c1: Waiting d1a4b09e9dbc: Waiting 99f41c3f014f: Waiting ``` While /etc/resolv.conf has the correct content and ping (and any other utility that uses libc's DNS resolution implementation) works correctly docker is unable to resolve the hostname and falls back to default [::1]:53. This started to happen after PR #13516 has been merged. As you can see from the log, dockerd is able to pick up the correct /etc/resolv.conf only after 5 sec since first try. This seems to be somehow related to the logic in Go's DNS resolver https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L385. There have been issues like that reported in docker like: - docker/cli#2299 - docker/cli#2618 - moby/moby#22398 Since this starts to happen after inclusion of resolvconf package by above mentioned PR and the fact I can't see any problem with that (ping, nslookup, etc. works) the choice is made to force dockerd to use cgo (libc) resolver. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
StormLiangMS
pushed a commit
to StormLiangMS/sonic-buildimage
that referenced
this pull request
Mar 1, 2023
…uration in Linux. (sonic-net#13516) - Why I did it fixes sonic-net#12907 When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared. - How I did it Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration. - How to verify it Ensure that the management interface has no static configuration. Check that /etc/resolv.conf has DNS configuration. Configure a static IP address on the management interface. Verify that /etc/resolv.conf has no DNS configuration. Remove the static IP address from the management interface. Verify that /etc/resolv.conf has DNS configuration retrieved form DHCP server.
yxieca
added a commit
to yxieca/sonic-buildimage
that referenced
this pull request
May 1, 2023
…reconfiguration in Linux. (sonic-net#13516) (sonic-net#13695)" This reverts commit d1fa414.
yxieca
added a commit
to yxieca/sonic-buildimage
that referenced
this pull request
May 1, 2023
…reconfiguration in Linux. (sonic-net#13516)" This reverts commit 5ef488f.
yxieca
added a commit
to yxieca/sonic-buildimage
that referenced
this pull request
May 1, 2023
…reconfiguration in Linux. (sonic-net#13516)" This reverts commit c7ecd92.
yxieca
added a commit
that referenced
this pull request
May 1, 2023
oleksandrivantsiv
added a commit
to oleksandrivantsiv/sonic-buildimage
that referenced
this pull request
May 30, 2023
…working reconfiguration in Linux. (sonic-net#13516)" (sonic-net#14902)" This reverts commit 72c52bc.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
fixes #12907
When the management interface IP address configuration changes from dynamic to static the DNS configuration (retrieved from the DHCP server) in /etc/resolv.conf remains uncleared. This leads to a DNS configuration pointing to the wrong nameserver. To make the behavior clear DNS configuration received from DHCP should be cleared.
How I did it
Use resolvconf package for managing DNS configuration. It is capable of tracking the source of DNS configuration and puts the configuration retrieved from the DHCP servers into a separate file. This allows the implementation of DNS configuration cleanup retrieved from DHCP during networking reconfiguration.
How to verify it
Which release branch to backport (provide reason below if selected)
Description for the changelog
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)