DNS Not Resolving under Network [CentOS8] #957

arpadgabor · 2020-03-22T02:08:45Z

Edit: There's a workaround in the comments below.

Ok so I have a fresh install of CentOS8, Docker, Docker compose. I'm trying to run Traefik. Created my network with docker create network proxy, attached it to my docker-compose file, ran my scripts, and it works. Kinda.

If I run a container with --net=host everything works fine. However, any other network I create does not resolve DNS

This is a bug report
This is a feature request
I searched existing issues before opening this one (yeah it's like 4AM already)

Expected behavior

$ docker exec -it traefik ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=55 time=3.983 ms
64 bytes from 1.1.1.1: seq=1 ttl=55 time=4.077 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.983/4.030/4.077 ms

Actual behavior

I repeat: If I run a container with --net=host everything works fine. However, any other network I create does not resolve DNS

$ docker exec -it traefik ping google.com
ping: bad address 'google.com

Steps to reproduce the behavior

I'm running a CentOS 8 VPS on Hetzner Cloud with the latest version of Docker (as of yesterday). The issue just "happened".

$ docker-compose -v
docker-compose version 1.25.4, build 8d51620a

Output of docker version:

$ docker -v
Docker version 19.03.8, build afacb8b

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 1
  Paused: 0
  Stopped: 4
 Images: 1
 Server Version: 18.09.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
 runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-80.11.2.el8_0.x86_64
 Operating System: CentOS Linux 8 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.755GiB
 Name: centos-4gb-nuremberg
 ID: DUXL:2E6W:WKB3:CWRG:7BUY:4VIN:YMXX:FRTH:L7YK:L5LY:X3D4:Y52L
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.)

Hetzner Cloud

The text was updated successfully, but these errors were encountered:

ask-and-go-to-the-blue · 2020-03-24T17:47:29Z

me too!

Output of uname -a :
Linux asusual.dev 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Wed Feb 5 02:00:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of iptables -L :

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Output of docker version :

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker run busybox nslookup google.com :

nslookup: write to '1.1.1.1': No route to host
;; connection timed out; no servers could be reached

nslookup: write to '8.8.8.8': No route to host

Output of docker run busybox ping 8.8.8.8 :

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=52 time=29.850 ms
64 bytes from 8.8.8.8: seq=1 ttl=52 time=30.712 ms

WARrior-Alex · 2020-04-30T09:18:49Z

Hi. Probably the reason is https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.0_release_notes/rhel-8_0_0_release#networking

I have the same issue at Fedora 32:
https://fedoraproject.org/wiki/Changes/firewalld_default_to_nftables#Scope

zx1986 · 2020-05-12T03:54:25Z

me too! same problem on CentOS 8, all my docker-compose containers are with a dead Internet.

lfiraza · 2020-05-12T07:31:49Z

fast workaround
sudo nano /etc/firewalld/firewalld.conf
in config file please change
FirewallBackend=nftables
on
FirewallBackend=iptables
save this changes and reload firewalld
sudo systemctl restart firewalld.service

it should help you work with docker but I advise rollback when docker will fix this

ask-and-go-to-the-blue · 2020-05-12T08:41:40Z

@lfiraza
Wow. Good!
Thank you Very very very very * 1000 much.

cassiamani · 2020-05-13T14:39:56Z

@lfiraza thanks dude, it's worked for me.

arpadgabor · 2020-05-14T14:15:20Z

@lfiraza added a link to your comment in the OP, in case anymore people see this. I switched back to centos 7 in the meantime.

l3lackMegas · 2020-05-25T07:37:38Z

@lfiraza That's work! Thank you very much. <3

edkepex · 2020-06-04T06:48:29Z

@lfiraza thanks friend, this save my day. I spent almost 24 hrs on this. Finally I came here and this works

otidh · 2020-06-27T06:07:53Z

@lfiraza Thanks very much! It works!!

matheenl · 2020-06-29T13:43:28Z

@lfiraza Thank you very much. It works for me as well. I was having issues with running Terraform from a docker container running on centos 8 host. Tried different steps including adding custom DNS, adding docker interface to trusted on firewall... but only the solution provided by you worked. Now I can run Terraform commands from my docker container....

Angeluz-07 · 2020-07-01T01:41:50Z

@lfiraza thank you. I couldn't build a ubuntu18 docker image because I was getting temporary failure in name resolution error , but your solution worked.

carlosemidio · 2020-07-05T13:08:11Z

@lfiraza Thanks very much. You saved my day!

full diff: moby/libnetwork@2e24aed...9e99af2 - moby/libnetwork#2548 Add docker interfaces to firewalld docker zone - fixes docker/for-linux#957 DNS Not Resolving under Network [CentOS8] - fixes moby/libnetwork#2496 Port Forwarding does not work on RHEL 8 with Firewalld running with FirewallBackend=nftables - store.getNetworksFromStore() remove unused error return - moby/libnetwork#2554 Fix 'failed to get network during CreateEndpoint' - fixes/addresses docker/for-linux#888 failed to get network during CreateEndpoint - moby/libnetwork#2558 [master] bridge: disable IPv6 router advertisements - moby/libnetwork#2563 log error instead if disabling IPv6 router advertisement failed - fixes docker/for-linux#1033 Shouldn't be fatal: Unable to disable IPv6 router advertisement: open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

jeffvc · 2020-07-09T00:44:42Z

Yo, seriously, thank you, I think I tried everything else till I stumbled on this. You should like ... work for NASA or SpaceX :)

junewerner · 2020-07-10T22:16:55Z

@lfiraza THANK YOU. Took forever to find this.

wswind · 2020-07-13T15:37:00Z

lfiraza's workaround works fine.
And there are two more ways:

(1) just use masquerade will make this work (recommand)

firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --reload

from robos's comment in <https://linuxconfig.org/how-to-install-docker-in-rhel-8 >

(2)set docker0 to trusted (this will open all the ports published by docker)

firewall-cmd --zone=trusted --add-interface=docker0 --permanent
firewall-cmd --reload

from: https://forums.centos.org/viewtopic.php?f=54&t=74270

full diff: moby/libnetwork@2e24aed...9e99af2 - moby/libnetwork#2548 Add docker interfaces to firewalld docker zone - fixes docker/for-linux#957 DNS Not Resolving under Network [CentOS8] - fixes moby/libnetwork#2496 Port Forwarding does not work on RHEL 8 with Firewalld running with FirewallBackend=nftables - store.getNetworksFromStore() remove unused error return - moby/libnetwork#2554 Fix 'failed to get network during CreateEndpoint' - fixes/addresses docker/for-linux#888 failed to get network during CreateEndpoint - moby/libnetwork#2558 [master] bridge: disable IPv6 router advertisements - moby/libnetwork#2563 log error instead if disabling IPv6 router advertisement failed - fixes docker/for-linux#1033 Shouldn't be fatal: Unable to disable IPv6 router advertisement: open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 219e7e7ddcf5f0314578d2a517fc0832f03622c1 Component: engine

jeffvc · 2020-07-13T20:21:55Z

lfiraza's workaround works fine.
And there are two more ways:

(1) just use masquerade will make this work
firewall-cmd --zone=public  --add-masquerade --permanent
firewall-cmd --reload
from comment in <https://linuxconfig.org/how-to-install-docker-in-rhel-8 >

(2)set docker0 to trusted
firewall-cmd --zone=trusted --add-interface=docker0 --permanent
firewall-cmd --reload
from: https://forums.centos.org/viewtopic.php?f=54&t=74270

Confirmed that firewall-cmd --zone=public --add-masquerade --permanent works on CentOS 8. Thanks for this fix, I wanted to mess around with nftables since that apparently is the standard going forward. Thanks for the two additional fixes!

lfiraza · 2020-07-14T06:21:52Z

@wswind @jeffvc
As I rememebr it dosen't work for docker network creating by docker-compose

wswind · 2020-07-15T09:12:09Z

@wswind @jeffvc
As I rememebr it dosen't work for docker network creating by docker-compose

@lfiraza Thanks for this information.

I checked the two fixes with busybox container .
--add-interface=docker0 can't work with docker-compose
masquerade works fine with both "docker" and "docker-compose"

Here is how I test this:

this command could be useful with docker test, both ways work fine

docker run busybox nslookup <somesite.com>

I tested docker-compose with a simple docker-compose.yml:

version: "3"
services:
  test:
    image: busybox

Steps:

1.clean the tow fixes

firewall-cmd --zone=public --remove-masquerade --permanent
firewall-cmd --zone=trusted --remove-interface docker0 --permanent
firewall-cmd --reload

2.apply one of the fixes

firewall-cmd --zone=public --add-masquerade --permanent
#or
firewall-cmd --zone=trusted --add-interface=docker0 --permanent
firewall-cmd --reload

3.use docker-compose to run busybox nslookup

docker-compose run test nslookup <somesite.com>

My environment:

docker-compose version 1.26.2
docker version 19.03.12
CentOS 8.2.2004 (Core)

Kwaiyu · 2020-08-14T03:45:25Z

在保存更改并重新加载firewalld时更改
sudo nano /etc/firewalld/firewalld.conf
配置文件的快速解决方法
FirewallBackend=nftables

FirewallBackend=iptables

sudo systemctl restart firewalld.service

它应该可以帮助您使用docker，但我建议在docker提供解决方案时回滚此设置。

fast workaround
sudo nano /etc/firewalld/firewalld.conf
in config file change
FirewallBackend=nftables
on
FirewallBackend=iptables
save change and reload firewalld
sudo systemctl restart firewalld.service

it should help you work with docker but I advise rollback this setting when docker provides solution.

Thank you very much for solving this issue, Centos8 has caused me a lot of trouble, I don't know why use nftables instead of iptables? why use dnf instead of yum?

EG-easy · 2020-09-12T20:49:38Z

uhyooooooooooooooooo!
THANK YOU!

cesirx · 2020-10-29T17:10:00Z

Same problem here.

After some research I found that DNS requests from the docker to my DNS server where arriving with the internal docker ip (172.16.0.2) as source address (instead of using the ip address of the docker host network interface).

In my case, SNAT was failing only for DNS traffic (could ping 8.8.8.8 from inside the container).

Proposed solutions did work for me too.

Many thanks!

gstanden · 2020-11-15T14:53:22Z

@lfiraza a lifesaver, THANK YOU!! FYI this also works to fix broken LXC container DNS name resolution and networking inside LXC containers on Linux 8 distros for LXC as well when containers are using iptables rules with physical interfaces. Symptoms of my LXC containers on Oracle Linux 8 before the FirewallBackend=iptables fix were ping of external internet addresses from inside LXC container was working but DNS resolution from inside container was not working.

As noted above by other commenters, there are ways to make this work without disabling nftables. I have tried them with LXC containers and they do work too.

moby/libnetwork#2496

codegraoully · 2021-01-21T11:08:50Z

fast workaround
sudo nano /etc/firewalld/firewalld.conf
in config file change
FirewallBackend=nftables
on
FirewallBackend=iptables
save change and reload firewalld
sudo systemctl restart firewalld.service

it should help you work with docker but I advise rollback this setting when docker provides solution.

Same issue, same resolving !

Thanks a lot!

JakkuSakura · 2021-01-24T10:12:16Z

What to do on MacOS?

shonoru mentioned this issue May 12, 2020

cgroups: cannot found cgroup mount destination: unknown #219

Open

3 tasks

arkodg mentioned this issue May 28, 2020

Add docker interfaces to firewalld docker zone moby/libnetwork#2548

Merged

jadbaz mentioned this issue Jun 9, 2020

Upgrade fails with KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"} getsentry/self-hosted#515

Closed

thaJeztah mentioned this issue Jul 8, 2020

vendor: libnetwork 9e99af28df21367340c95a3863e31808d689c92a moby/moby#41189

Merged

tiborvass closed this as completed in moby/moby#41189 Jul 13, 2020

herrib mentioned this issue Jul 28, 2020

dl-cdn.alpinelinux.org errors gliderlabs/docker-alpine#386

Closed

doludizgin mentioned this issue Aug 4, 2020

Getting errors on kibana and logstash sherifabdlnaby/elastdocker#18

Closed

thaJeztah mentioned this issue Aug 31, 2020

[19.03 backport] Add docker interfaces to firewalld docker zone moby/libnetwork#2575

Closed

vsotirasqmplus mentioned this issue Oct 9, 2020

docker-compose networking issue cannot connect to the database server moodlehq/moodle-docker#140

Closed

onepushmain mentioned this issue Oct 11, 2020

Temporary error (try again later) gliderlabs/docker-alpine#334

Open

Moisey777 mentioned this issue Oct 27, 2020

Centos 8: ошибка при сборке контейера GolosChain/golosio-launching#12

Closed

SRAPSpencer mentioned this issue Apr 15, 2022

Broken redirect after Azure SSO integration SecurityRiskAdvisors/VECTR#172

Closed

ytooyama mentioned this issue Jun 27, 2022

Trivy Scan does not work (Failed to download vulnerability DB) goharbor/harbor#17061

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS Not Resolving under Network [CentOS8] #957

DNS Not Resolving under Network [CentOS8] #957

arpadgabor commented Mar 22, 2020 •

edited

Loading

ask-and-go-to-the-blue commented Mar 24, 2020 •

edited

Loading

WARrior-Alex commented Apr 30, 2020

zx1986 commented May 12, 2020

lfiraza commented May 12, 2020 •

edited

Loading

ask-and-go-to-the-blue commented May 12, 2020

cassiamani commented May 13, 2020

arpadgabor commented May 14, 2020

l3lackMegas commented May 25, 2020 •

edited

Loading

edkepex commented Jun 4, 2020

otidh commented Jun 27, 2020

matheenl commented Jun 29, 2020

Angeluz-07 commented Jul 1, 2020

carlosemidio commented Jul 5, 2020

jeffvc commented Jul 9, 2020

junewerner commented Jul 10, 2020

wswind commented Jul 13, 2020 •

edited

Loading

jeffvc commented Jul 13, 2020

lfiraza commented Jul 14, 2020

wswind commented Jul 15, 2020 •

edited

Loading

Kwaiyu commented Aug 14, 2020

EG-easy commented Sep 12, 2020

cesirx commented Oct 29, 2020

gstanden commented Nov 15, 2020 •

edited

Loading

codegraoully commented Jan 21, 2021

JakkuSakura commented Jan 24, 2021

DNS Not Resolving under Network [CentOS8] #957

DNS Not Resolving under Network [CentOS8] #957

Comments

arpadgabor commented Mar 22, 2020 • edited Loading

Expected behavior

Actual behavior

Steps to reproduce the behavior

ask-and-go-to-the-blue commented Mar 24, 2020 • edited Loading

WARrior-Alex commented Apr 30, 2020

zx1986 commented May 12, 2020

lfiraza commented May 12, 2020 • edited Loading

ask-and-go-to-the-blue commented May 12, 2020

cassiamani commented May 13, 2020

arpadgabor commented May 14, 2020

l3lackMegas commented May 25, 2020 • edited Loading

edkepex commented Jun 4, 2020

otidh commented Jun 27, 2020

matheenl commented Jun 29, 2020

Angeluz-07 commented Jul 1, 2020

carlosemidio commented Jul 5, 2020

jeffvc commented Jul 9, 2020

junewerner commented Jul 10, 2020

wswind commented Jul 13, 2020 • edited Loading

jeffvc commented Jul 13, 2020

lfiraza commented Jul 14, 2020

wswind commented Jul 15, 2020 • edited Loading

Kwaiyu commented Aug 14, 2020

EG-easy commented Sep 12, 2020

cesirx commented Oct 29, 2020

gstanden commented Nov 15, 2020 • edited Loading

codegraoully commented Jan 21, 2021

JakkuSakura commented Jan 24, 2021

arpadgabor commented Mar 22, 2020 •

edited

Loading

ask-and-go-to-the-blue commented Mar 24, 2020 •

edited

Loading

lfiraza commented May 12, 2020 •

edited

Loading

l3lackMegas commented May 25, 2020 •

edited

Loading

wswind commented Jul 13, 2020 •

edited

Loading

wswind commented Jul 15, 2020 •

edited

Loading

gstanden commented Nov 15, 2020 •

edited

Loading