Processes unavailable by service: port

[ghost]

Description

Can not access to container port via service on local OCP.

Reproduction Steps

1. Run a local OCP (oc cluster up) or MiniShift
2. Create the following objects via OCP console:

• Create a project;
• Click Add to Project
• Click Import YAML/JSON
• Paste the following YAML and click Create

Objects to import

---
kind: List
items:
-
  apiVersion: v1
  kind: Pod
  metadata:
    name: tomcat-pod
    labels:
      app: tomcat
  spec:
    containers:
      -
        image: sleshchenko/webapp
        name: tomcat-container
        ports:
          -
            containerPort: 8080
            protocol: TCP
      -
        image: eclipse/ubuntu_jdk8
        name: requester
-
  apiVersion: v1
  kind: Service
  metadata:
    name: tomcat
  spec:
    ports:
      - name: tomcat
        port: 8080
        protocol: TCP
        targetPort: 8080
    selector:
      app: tomcat

3. Open terminal in the newly created pod in requester container.
4. Try to access tomcat via service. Execute curl tomcat:8080
   Expected: Response HELLO
   Actual: Request hung up.

Note that it works fine on our OSD and OSO too.

Note that it works fine if containers are in separated pods.

Containers in separate pods

---
kind: List
items:
-
  apiVersion: v1
  kind: Pod
  metadata:
    name: tomcat-pod
    labels:
      app: tomcat
  spec:
    containers:
      -
        image: sleshchenko/webapp
        name: tomcat-container
        ports:
          -
            containerPort: 8080
            protocol: TCP
-
  apiVersion: v1
  kind: Pod
  metadata:
    name: requester
  spec:
    containers:
      -
        image: eclipse/ubuntu_jdk8
        name: requester
-
  apiVersion: v1
  kind: Service
  metadata:
    name: tomcat
  spec:
    ports:
      - name: tomcat
        port: 8080
        protocol: TCP
        targetPort: 8080
    selector:
      app: tomcat

OCP version:
3.7.0

Fedora 27. Firewalld is off.

The process is unavailable even when there's just one pod. Say I run tomcat on port 8080, and can curl it from inside a container as localhost:8080 but not tomcat:8080

[garagatyi]

What can I see is that access to pod from the inside of this pod by the service IP doesn't work for some reason. It works neither from the container that exposes port nor from other containers of the same pod.

[php-coder]

@garagatyi This could be caused by improper configured DNS. I'd start with checking /etc/resolv.conf inside of a pod.

This instruction also could be helpful: https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#debugging-dns-resolution

Hope this helps!

[knobunc]

@garagatyi -- Can you grab the iptables output from the node running openshift? And give me the service IP for the service you are looking at? Thanks.

[garagatyi]

@php-coder this doesn't look like a DNS resolution issue since DNS gets resolved to IP of a service. But this IP is not responding.And it is not responding from the inside of the POD, but is responding from other PODs and my host.

@knobunc sure. BTW it is an OCP running inside of Docker containers on my dev fedora 25. And we have the same situation on other dev fedoras and also on a CI (containerized OCP too).
Service IP is: 172.30.64.152

Unfold iptables --list output

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
KUBE-NODEPORT-NON-LOCAL  all  --  anywhere             anywhere             /* Ensure that non-local NodePort traffic can flow */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
OUTPUT_direct  all  --  anywhere             anywhere            

Chain DOCKER (3 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:webcache
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:irdmi

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 
FWDI_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 
FWDO_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_FedoraWorkstation (2 references)
target     prot opt source               destination         
FWDI_FedoraWorkstation_log  all  --  anywhere             anywhere            
FWDI_FedoraWorkstation_deny  all  --  anywhere             anywhere            
FWDI_FedoraWorkstation_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain FWDI_FedoraWorkstation_allow (1 references)
target     prot opt source               destination         

Chain FWDI_FedoraWorkstation_deny (1 references)
target     prot opt source               destination         

Chain FWDI_FedoraWorkstation_log (1 references)
target     prot opt source               destination         

Chain FWDO_FedoraWorkstation (2 references)
target     prot opt source               destination         
FWDO_FedoraWorkstation_log  all  --  anywhere             anywhere            
FWDO_FedoraWorkstation_deny  all  --  anywhere             anywhere            
FWDO_FedoraWorkstation_allow  all  --  anywhere             anywhere            

Chain FWDO_FedoraWorkstation_allow (1 references)
target     prot opt source               destination         

Chain FWDO_FedoraWorkstation_deny (1 references)
target     prot opt source               destination         

Chain FWDO_FedoraWorkstation_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 
IN_FedoraWorkstation  all  --  anywhere             anywhere            [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_FedoraWorkstation (2 references)
target     prot opt source               destination         
IN_FedoraWorkstation_log  all  --  anywhere             anywhere            
IN_FedoraWorkstation_deny  all  --  anywhere             anywhere            
IN_FedoraWorkstation_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain IN_FedoraWorkstation_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:blackjack:65535 ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpts:blackjack:65535 ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-ns ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-dgm ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     udp  --  anywhere             224.0.0.251          udp dpt:mdns ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http ctstate NEW

Chain IN_FedoraWorkstation_deny (1 references)
target     prot opt source               destination         

Chain IN_FedoraWorkstation_log (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-NODEPORT-NON-LOCAL (1 references)
target     prot opt source               destination         

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination         

[garagatyi]

Thanks to @gorkem I discovered this issue. Which led me to the idea that hairpin mode might be misconfigured. I haven't found how to check it.
@knobunc do you know the way how to check it?

[garagatyi]

Yep. It is because of hairpin mode. I am able to access the pod by its service after I exec'd into openshift container and put 1 to /sys/devices/virtual/net/veth*/brport/hairpin_mode.
So now I'm investigating how to fix setting of the hairpin in OCP.

[garagatyi]

Looks like this issue is a duplicate of #14031

[knobunc]

Can you see if your bridge is in promiscuous mode:
ip link show docker0

And can you see if your node logs mention "Hairpin mode" please?

[garagatyi]

ip link show docker0

5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:26:44:8f:4b brd ff:ff:ff:ff:ff:ff

docker logs origin 2>&1 | grep -i hair

W0118 19:46:33.822287   25899 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0118 19:46:33.822322   25899 kubelet.go:507] Hairpin mode set to "hairpin-veth"

If provided logs are not what you were asking for, please, let me know how I can get those logs.

[TimMoran]

@knobunc This ticket seems to have gone stale and is in the process of being escalated. Is the logs that @garagatyi provided sufficient?

Thanks for looking back at this. The priority has raised on it.

[skabashnyuk]

Comment from Ben Bennett 2018-03-01 13:19:04 EST

This is because you have no networking plugin and so nothing is setting the hairpin mode on the > bridge. Please try using the kubenet networking plugin.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[ghost]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-merge-robot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-merge-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.