Che fails on version 1.4.1

[gorkem]

Che has a service named che-host that has a route configured. When the service is trying to reach itself through the internal names it fails, public route works fine. When I ssh into the pod and try to curl to http://che-host it fails after a long wait where as http://localhost and with internal ip address works fine.

Also when I enable the logs I see lines like follow for every pod.
1610 docker_sandbox.go:263] Couldn't find network status for eclipse-che/che-3-deploy through plugin: invalid network status for

We also did a minishift ssh and compared /etc/resolv.conf with 1.3.1 and 1.4.1 only has a single nameserver entry where as 1.3.1 also includes and entry for search.

@amisevsk Anything else I am missing?

[gbraad]

Interesting, I'll have a look. BTW, did you use the B2D or CentOS image?

[gorkem]

I seem to have b2d

[gbraad]

I did a quick run of the versions you mentioned, using B2D, but I never get an entry for 'search' in /etc/resolv.conf

• Minishift v.1.4.1
  • OpenShift v3.6 - nameserver 192.168.122.1
  • OpenShift v1.5.1 - nameserver 192.168.122.1
• Minishift v1.3.1
  • OpenShift v1.5.1 - nameserver 192.168.122.1

With CentOS (v1.3.1 - OS v1.5.1)

# Generated by NetworkManager
nameserver 192.168.122.1
nameserver 192.168.42.1

With CentOS (v1.4.1 - OS v3.6.0)

# Generated by NetworkManager
nameserver 192.168.122.1
nameserver 192.168.42.1

Which is as expected...

[gorkem]

Hmmm. I do not have those entries anymore on my 1.3.1 either

[amisevsk]

I've managed to narrow this down somewhat.

Running minishift 1.4.1 and Openshift version

OpenShift Master: v3.6.0+c4dd4cf
Kubernetes Master: v1.6.1+5115d708d7

I am able to reproduce the issue. Running minishift 1.4.1 and OpenShift v1.5.1 it does not occur.

However, the issue is actually that pods cannot resolve their own service or service's clusterIP. Starting a second pod, I can curl che-host without issue, but from within che host I cannot. There are no real networking issues except that pods cannot access their own services -- localhost and external work fine.

After a bit of digging, I came across this section of kubernetes documentation that seems related.

@gbraad Is there a setting that has changed between 1.5.1 and 3.6? I don't see this issue on OpenShift Online, running

OpenShift Master: v3.6.173.0.5 (online version 3.5.0.20)
Kubernetes Master: v1.6.1+5115d708d7

[gbraad]

Is there a setting that has changed between 1.5.1 and 3.6?

I do not have enough visibility on this, but hopefully @csrwng or @bparees knows more about this. This could very well be related to how oc cluster up sets up the configuration. You could try this with a new VM and just running oc cluster up. This would exclude any of our configuration that happens.

[csrwng]

We've seen this issue in cluster up before -- see openshift/origin#12111
There were 2 different problems we found with code,

1. /sys/devices/virtual/net was not getting mounted into the origin container as read/write and the kubelet was not able to set the hairpin mode on the docker bridge
2. /var/lib/docker was hardcoded as the host's docker directory and in some machines that wasn't the real docker directory

If either of these is still the cause, one thing you can try to solve the issue is running:

ifconfig docker0 promisc

on the minishift vm

[amisevsk]

@csrwng It looks like your command solved the issue for me.

Regarding the problems you listed,

1. /sys/devices/virtual/net seems to be mounted RW in the origin container:

            {
                "Source": "/sys/devices/virtual/net",
                "Destination": "/sys/devices/virtual/net",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },

2. I'm not sure how to check, but /var/lib/docker is a symlink to /mnt/sda1/var/lib/docker, which seems to be correct.

We've figured out a workaround for our issue (using localhost instead of the service), but I would still like to know what's going wrong.

[gbraad]

I would still like to know what's going wrong.

So do I, as we might have to provide this as a known issue. Especially since the PR openshift/origin#12744 seems to have been available since Feb 1st. And therefore this should have been observed since v1.5.1?

[csrwng]

@gbraad if you tell minishift v1.4.1 to run origin version v3.6, does it obtain the v3.6 oc client to run 'oc cluster up'? or does it use the one bundled in the image? If the latter, what version of the 'oc' binary is included in the minishift v1.4.1 image?

[praveenkumar]

run origin version v3.6, does it obtain the v3.6 oc client to run 'oc cluster up'?

@csrwng Yes it obtain 3.6 oc client to deploy 3.6 cluster.

[LalatenduMohanty]

@csrwng I am confused. I guess Origin 3.6 oc binary has the fix i.e. openshift/origin#12744 right?

[csrwng]

Yes it should have the fix

[praveenkumar]

@gorkem can you try out it with minishift 1.5.0 without any workaround and let us know if it still fail because here we are using 3.6.0 as default openshift version.

[coolbrg]

Any update @gorkem ? Have you got chance to try minishift 1.5.0 ?

[gorkem]

I was able to use it with minishift 1.5.1 without any hacks.

[coolbrg]

Thanks, @gorkem for confirming. We are closing this issue now. If you facing anything please feel free to open the new issue.