N6 internet connectivity

[amrbaraka]

First, thank you for the published deployment & configuration details, following the instructions I've been able to deploy the UPF successfully on minikube on top of AWS EC2 ubuntu.

Now I'm facing the following issue:

First, I tried to use ping to test N6 connectivity towards the internet, but apparently iputils-ping is not part of UPF image.

So I tried apt-get update, and that's when I realized that internet connectivity is NOK:

You can see below that all interfaces are UP:

IP Routes:

I also removed the secondary default route manually "to keep only n6 as default route" but still internet reacheability failed:

Knowing that, the UPF is able to reach its n6 designated gw 192.168.49.1 (which is the IP of minikube bridge created on AWS ec2)

I verified this via ping from the bridge interface on ec2:

Any ideas what could be going wrong?

Thanks in advance for your help & support.

Br,
Amr

[amrbaraka]

So I've realized that it's a DNS issue.
I've installed an alpine container with eth0 calico and n6 network attachment., where I've noticed that ping to the internet is working only towards direct IPs and fails for domain names.
/etc/resolve.conf points to kube-dns pod cluster IP:

/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search free5gc.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal
options ndots:5

And this is the coredns configmap:
<<<
kubectl -n kube-system get cm coredns -o yaml
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
hosts {
192.168.49.1 host.minikube.internal
fallthrough
}
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
kind: ConfigMap

When I changed the nameserver to 8.8.8.8 inside the alpine test container, the DNS resolution succeeds 50% of the time,,
which is really strange,,

I also checked IP tables and added this rule to both minikube container and the ec2 instance iptables:
sudo iptables -I DOCKER-USER -j ACCEPT

but still no change.

Here's a tcpdump during DNS query failure:

/ # tcpdump -v -i n6
tcpdump: listening on n6, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:51:25.879423 IP (tos 0x0, ttl 64, id 61996, offset 0, flags [DF], proto UDP (17), length 56)
alpine.56107 > dns.google.53: 40095+ A? medium.com. (28)
23:51:25.879467 IP (tos 0x0, ttl 64, id 61997, offset 0, flags [DF], proto UDP (17), length 56)
alpine.56107 > dns.google.53: 41171+ AAAA? medium.com. (28)
23:51:28.380764 IP (tos 0x0, ttl 64, id 61998, offset 0, flags [DF], proto UDP (17), length 56)
alpine.56107 > dns.google.53: 40095+ A? medium.com. (28)
23:51:28.380823 IP (tos 0x0, ttl 64, id 61999, offset 0, flags [DF], proto UDP (17), length 56)
alpine.56107 > dns.google.53: 41171+ AAAA? medium.com. (28)
23:51:28.456841 IP (tos 0x0, ttl 64, id 55706, offset 0, flags [DF], proto UDP (17), length 66)
192.168.49.101.46484 > dns.google.53: 4320+ PTR? 8.8.8.8.in-addr.arpa. (38)
23:51:28.458253 IP (tos 0x0, ttl 117, id 29601, offset 0, flags [none], proto UDP (17), length 90)
dns.google.53 > 192.168.49.101.46484: 4320 1/0/0 8.8.8.8.in-addr.arpa. PTR dns.google. (62)
23:51:30.898029 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.49.1 tell 192.168.49.101, length 28
23:51:30.898392 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.49.1 is-at 02:42:08:b2:fc:15 (oui Unknown), length 28
23:51:30.974570 IP (tos 0x0, ttl 64, id 55851, offset 0, flags [DF], proto UDP (17), length 73)
192.168.49.101.34128 > dns.google.53: 10314+ PTR? 101.49.168.192.in-addr.arpa. (45)
23:51:30.975529 IP (tos 0x0, ttl 54, id 62126, offset 0, flags [none], proto UDP (17), length 73)
dns.google.53 > 192.168.49.101.34128: 10314 NXDomain 0/0/0 (45)
23:51:30.978078 IP (tos 0x0, ttl 64, id 55852, offset 0, flags [DF], proto UDP (17), length 71)
192.168.49.101.37200 > dns.google.53: 4779+ PTR? 1.49.168.192.in-addr.arpa. (43)
23:51:30.979422 IP (tos 0x0, ttl 54, id 54314, offset 0, flags [none], proto UDP (17), length 71)
dns.google.53 > 192.168.49.101.37200: 4779 NXDomain 0/0/0 (43)
23:51:33.714016 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.49.101 tell 192.168.49.1, length 28
23:51:33.714052 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.49.101 is-at ae:bd:e6:be:28:d8 (oui Unknown), length 28

192.168.49.101 is the container n6 interface IP
192.168.49.1 is the minikube bridge interface IP which is the gw for the n6 interface.

Any ideas what's causing DNS resolution failure from the pods ?

[amrbaraka]

So, I've added these entries to /etc/hosts:
91.189.91.38 archive.ubuntu.com
91.189.91.38 security.ubuntu.com

The connection worked towards the internet, and I was able to install iputils-ping and ping worked:

root@free5gc-upf-upf-589c67bf5-9lgx8:/free5gc/upf# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=1.39 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=2.45 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=111 time=2.55 ms

Has this DNS issue been seen/reported before?

It might be a minikube issue, there seem to be some similar issues reported earlier:
kubernetes/minikube#8949

[rveerama1]

@amrbaraka is N6 internet connectivity working now for you now? I am also stuck at N6 internet connectivity issue.

[amrbaraka]

@amrbaraka is N6 internet connectivity working now for you now? I am also stuck at N6 internet connectivity issue.

@rveerama1 is it the same symptoms as in my description ? i.e ping command from inside the pod is not available and DNS resolution failure when executing apt-get update,,, please also keep in mind that I was able to ping the n6 interface from it's default gw (which is the IP of minikube bridge created on AWS ec2)

if yes, then you can append the below entries to /etc/hosts of the UPF:
echo '91.189.91.38 archive.ubuntu.com' >> /etc/hosts
echo '91.189.91.38 security.ubuntu.com' >> /etc/hosts

Afterwards, I was able to execute apt-get update and install iputils-ping pkg and I was able to test the ping towards the internet,, which worked fine in case of IPs, but still failed for domain names, so I installed vim and set 8.8.8.8 as my DNS server under /ets/resolv.conf and it worked.

Br,
Amr

[rveerama1]

@amrbaraka thanks for the reply. I reported my issue #62.

I could do "apt update" and "apt install " inside UPF. but

"kubectl --namespace sim exec -it $POD_NAME -- ping -c 1 -I uesimtun0 www.google.com" doesn't work.

I tried pinging from inside UPF too.

root@test-free5gc-upf-upf-6485b99bf9-8p6qr:/free5gc/upf# ping -I n6 -c 1 www.google.com
PING www.google.com (172.217.169.36) from 192.168.0.3 n6: 56(84) bytes of data.

--- www.google.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

[amrbaraka]

@rveerama1 I've just suggested a modification in your config in issue #62
hope it works!

Br,
Amr

[rveerama1]

@amrbaraka thank you.

[amrbaraka]

closing the issue.