Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pod in minikube can't access other services or connect to the internet #8949

Closed
koji117 opened this issue Aug 10, 2020 · 10 comments · Fixed by #10985
Closed

pod in minikube can't access other services or connect to the internet #8949

koji117 opened this issue Aug 10, 2020 · 10 comments · Fixed by #10985
Labels
kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@koji117
Copy link

koji117 commented Aug 10, 2020

I cannot get my services to communicate each other.

After I did kubectl exec -it my-pod bash and execute ping and curl, I found that I cannot access other services and also have no access to the internet.

In my /etc/resolve.conf inside my pod, I have

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local dscd-mno.jpe2b.dcnw.rakuten
options ndots:5

So it seems to be pointing at kube-dns (10.96.0.10) but when I checked the addon, I dont have kube-dns enabled.

  1. minikube start --vm-driver=virtualbox
  2. minikube addons list | grep dns
|-----------------------------|----------|--------------|
|         ADDON NAME          | PROFILE  |    STATUS    |
|-----------------------------|----------|--------------|
| ambassador                  | minikube | disabled     |
| dashboard                   | minikube | disabled     |
| default-storageclass        | minikube | enabled ✅   |
| efk                         | minikube | disabled     |
| freshpod                    | minikube | disabled     |
| gcp-auth                    | minikube | disabled     |
| gvisor                      | minikube | disabled     |
| helm-tiller                 | minikube | disabled     |
| ingress                     | minikube | disabled     |
| ingress-dns                 | minikube | disabled     |
| istio                       | minikube | disabled     |
| istio-provisioner           | minikube | disabled     |
| kubevirt                    | minikube | disabled     |
| logviewer                   | minikube | disabled     |
| metallb                     | minikube | disabled     |
| metrics-server              | minikube | disabled     |
| nvidia-driver-installer     | minikube | disabled     |
| nvidia-gpu-device-plugin    | minikube | disabled     |
| olm                         | minikube | disabled     |
| pod-security-policy         | minikube | disabled     |
| registry                    | minikube | disabled     |
| registry-aliases            | minikube | disabled     |
| registry-creds              | minikube | disabled     |
| storage-provisioner         | minikube | enabled ✅   |
| storage-provisioner-gluster | minikube | disabled     |
|-----------------------------|----------|--------------|
  1. minikube addons enable kube-dns
💣  enable failed: run callbacks: kube-dns is not a valid addon
😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose
@priyawadhwa priyawadhwa added the kind/support Categorizes issue or PR as a support question. label Aug 12, 2020
@neargle
Copy link

neargle commented Sep 9, 2020

I got a same issue like this in Microsoft Windows 10 10.0.18362 Build 18362.
Minikube version info:

minikube version: v1.13.0
commit: 0c5e9de4ca6f9c55147ae7f90af97eff5befef5f-dirty

@priyawadhwa
Copy link

Hey @neargle thanks for opening this issue. kube-dns is not available as an addon in minikube, we use coredns which is automatically enabled by kubernetes now.

It looks like there's some networking issue if you can't connect to the internet.

Could you provide the output of

  • minikube ssh -- ping -c 4 google.com
  • kubectl describe deploy/coredns -n kube-system
  • minikube ssh -- cat /etc/resolv.conf

to help with debugging?

@priyawadhwa priyawadhwa added the triage/needs-information Indicates an issue needs more information in order to work on it. label Sep 9, 2020
@priyawadhwa priyawadhwa changed the title kube-dns is not a valid addon pod in minikube can't access other services or connect to the internet Sep 9, 2020
@neargle
Copy link

neargle commented Sep 10, 2020

@priyawadhwa Thx for reply.

$ cat /etc/resolv.conf

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.0.2.3

$ ping google.com -c 4

PING google.com (216.58.220.206): 56 data bytes
64 bytes from 216.58.220.206: seq=0 ttl=111 time=8.963 ms
64 bytes from 216.58.220.206: seq=1 ttl=111 time=5.990 ms
64 bytes from 216.58.220.206: seq=2 ttl=111 time=5.739 ms
64 bytes from 216.58.220.206: seq=3 ttl=111 time=5.534 ms

--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 5.534/6.556/8.963 ms

This is the output of kubectl describe deploy/coredns -n kube-system:

kubectl describe deploy/coredns -n kube-system
Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Tue, 08 Sep 2020 14:13:32 +0800
Labels:                 k8s-app=kube-dns
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       k8s.gcr.io/coredns:1.7.0
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-f9fd979d6 (1/1 replicas created)
Events:          <none>

I add log in coredns configmap:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    log\n    errors\n    health {\n       lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n       pods insecure\n       fallthrough in-addr.arpa ip6.arpa\n       ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n       max_concurrent 1000\n    }\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"creationTimestamp":"2020-09-08T06:13:32Z","name":"coredns","namespace":"kube-system","resourceVersion":"205","selfLink":"/api/v1/namespaces/kube-system/configmaps/coredns","uid":"d61d3477-61fb-4a95-8dd4-e5c490e93393"}}
  creationTimestamp: "2020-09-08T06:13:32Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data: {}
    manager: kubeadm
    operation: Update
    time: "2020-09-08T06:13:32Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:Corefile: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2020-09-09T06:52:33Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "90799"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: d61d3477-61fb-4a95-8dd4-e5c490e93393

And all log since yesterday below:

➜  ~ kubectl logs coredns-f9fd979d6-xgg5l -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
[INFO] 127.0.0.1:53665 - 43653 "HINFO IN 2120293140971769503.5099434421635548354. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.009732407s

@priyawadhwa
Copy link

Hey @neargle looks like there's an issue with the /etc/hosts file -- we write to that file during minikube start and after a successful start my file looks like this:

$ minikube ssh -- cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	minikube
192.168.64.1	host.minikube.internal
192.168.64.4	control-plane.minikube.internal

could you try upgrading to minikube 1.13.1 and see if that resolves the issue? I know we've done some work around /etc/hosts in the past couple months

@priyawadhwa
Copy link

Hey @neargle I'm going to go ahead and close this issue as it's been quiet for a few weeks. If you're still seeing this issue, feel free to reopen at any time by commenting /reopen!

@cheloim
Copy link

cheloim commented Oct 23, 2020

Sorry I open it up again as soon as you closed it. But i encounter this same error at this moment. I'm using minikube version v1.14.0 with Libvirt on Fedora 32 and no DNS resolution.

using dnsutils image.

#nslookup google.com.ar
;; connection timed out; no servers could be reached.

cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

I verified dns Service ClusterIP is 10.96.0.10 and endpoint is running and svc pointed to endpoints IP.

minikube -p k8s-workshop ssh -- cat /etc/hosts
127.0.0.1       localhost
127.0.1.1 k8s-workshop
192.168.39.1    host.minikube.internal
192.168.39.200  control-plane.minikube.internal
# k get svc kube-dns -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   24h
# k get ep -n kube-system -l k8s-app=kube-dns
NAME       ENDPOINTS                                     AGE
kube-dns   172.17.0.6:53,172.17.0.6:9153,172.17.0.6:53   24h
# k describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.17.0.6:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.17.0.6:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         172.17.0.6:9153
Session Affinity:  None
Events:            <none>

@cheloim
Copy link

cheloim commented Oct 23, 2020

Ok, so I keep making test. I deleted my k8s-workshop cluster so I can try it in stock configurations.

With single node everything just works fine, with and without '-p' parameter

/ # nslookup kubernetes
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

When I try to create a multi node cluster, in this case with 3, no DNS resolution.

/ # nslookup kubernetes
;; connection timed out; no servers could be reached

/ # Session ended, resume using 'kubectl attach dnsutils -c dnsutils -i -t' command when the pod is running
k get nodes
NAME           STATUS   ROLES    AGE    VERSION
minikube       Ready    master   145m   v1.19.2
minikube-m02   Ready    <none>   143m   v1.19.2
minikube-m03   Ready    <none>   142m   v1.19.2

@shubhamdixit863
Copy link

minikube addons enable coredns ,I was able to solve my problem with the help of this . But still can't communicate between two container in different pods

@dxGuan
Copy link

dxGuan commented Oct 29, 2020

I am facing the same problem with minikube version: v1.13.1

  • minikube ssh -- ping -c 4 kubernetes.io
PING kubernetes.io (147.75.40.148): 56 data bytes
64 bytes from 147.75.40.148: seq=0 ttl=50 time=221.348 ms
64 bytes from 147.75.40.148: seq=1 ttl=50 time=220.647 ms
64 bytes from 147.75.40.148: seq=2 ttl=50 time=221.285 ms
64 bytes from 147.75.40.148: seq=3 ttl=50 time=221.470 ms

--- kubernetes.io ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 220.647/221.187/221.470 ms
  • kubectl describe deploy/coredns -n kube-system
Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Wed, 28 Oct 2020 11:20:21 +0800
Labels:                 k8s-app=kube-dns
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-6c76c8bb89 (1/1 replicas created)
Events:          <none>
  • minikube ssh -- cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.122.1

@infa-knandi
Copy link

infa-knandi commented Nov 26, 2020

Try this it got solved for me

kubernetes/kubernetes#21613 (comment)

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Thank me later!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants