Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] #9444

csrwng · 2016-06-20T20:52:03Z

Test flake: kubectl expose test fails while waiting for redis container

Version

v1.3.0-alpha.1-380-g4965f56

Steps To Reproduce

Run extended tests core

Current Result

Extended test failure:

• Failure [111.412 seconds]
[k8s.io] Kubectl client
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/framework/framework.go:505
  [k8s.io] Kubectl expose
  /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/framework/framework.go:505
    should create services for rc [Conformance] [It]
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/kubectl.go:798

    Jun 20 15:08:44.916: No pods matched the filter.

Expected Result

Test passes

Additional Information

Times out waiting for redis RC:

STEP: creating Redis RC
Jun 20 15:07:14.515: INFO: namespace e2e-tests-kubectl-h1cxa
Jun 20 15:07:14.515: INFO: Running '/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/kubectl --server=https://172.18.14.213:8443 --kubeconfig=/tmp/openshift/openshift/test-extended/core/openshift.local.config/master/admin.kubeconfig create -f /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/testing-manifests/kubectl/redis-master-controller.json --namespace=e2e-tests-kubectl-h1cxa'
Jun 20 15:07:14.902: INFO: stderr: ""
Jun 20 15:07:14.902: INFO: stdout: "replicationcontroller \"redis-master\" created\n"
STEP: Waiting for Redis master to start.
Jun 20 15:07:15.968: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:07:15.968: INFO: Found 0 / 1
Jun 20 15:08:38.908: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:38.908: INFO: Found 0 / 1
Jun 20 15:08:39.942: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:39.942: INFO: Found 0 / 1
Jun 20 15:08:40.905: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:40.905: INFO: Found 0 / 1
Jun 20 15:08:41.905: INFO: Selector matched 1 pods for map[app:redis]
...
Jun 20 15:08:41.905: INFO: Found 0 / 1
Jun 20 15:08:42.905: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:42.905: INFO: Found 0 / 1
Jun 20 15:08:43.905: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:43.905: INFO: Found 0 / 1
Jun 20 15:08:44.905: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:44.905: INFO: Found 0 / 1
Jun 20 15:08:44.910: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:44.910: INFO: Found 0 / 1
Jun 20 15:08:44.910: INFO: WaitFor completed with timeout 1m30s.  Pods found = 0 out of 1
Jun 20 15:08:44.916: INFO: Selector matched 1 pods for map[app:redis]
Jun 20 15:08:44.916: INFO: No pods matched the filter.

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2306/

The text was updated successfully, but these errors were encountered:

ncdc · 2016-06-21T13:28:17Z

The openshift log was truncated at 20MB and doesn't contain anything related to the failing test case. The Jenkins console has this as the last event related to the pod:

Jun 20 15:08:44.921: INFO: At {2016-06-20 15:07:35 -0400 EDT} - event for redis-master-h8d1a: {kubelet 172.18.14.213} Pulling: pulling image "gcr.io/google_containers/redis:e2e"

I'm guessing this is a gcr.io flake but without logs, I can't say for sure. Please reopen if this happens again, and maybe we'll have more useful logs.

0xmichalis · 2016-06-24T09:54:14Z

Once again: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2548/consoleFull

ncdc · 2016-06-24T11:08:18Z

This is what I suspected - pulling the image is just taking too long.

Jun 24 05:02:01.070: INFO: stdout: "replicationcontroller \"redis-master\" created\n"
STEP: Waiting for Redis master to start.
[...]
Jun 24 05:03:31.077: INFO: WaitFor completed with timeout 1m30s.  Pods found = 0 out of 1

Jun 24 05:03:31.084: INFO: At {2016-06-24 05:02:01 -0400 EDT} - event for redis-master: {replication-controller } SuccessfulCreate: Created pod: redis-master-ukphz
Jun 24 05:03:31.084: INFO: At {2016-06-24 05:02:01 -0400 EDT} - event for redis-master-ukphz: {default-scheduler } Scheduled: Successfully assigned redis-master-ukphz to 172.18.12.236
Jun 24 05:03:31.084: INFO: At {2016-06-24 05:02:16 -0400 EDT} - event for redis-master-ukphz: {kubelet 172.18.12.236} Pulling: pulling image "gcr.io/google_containers/redis:e2e"
Jun 24 05:03:31.084: INFO: At {2016-06-24 05:03:29 -0400 EDT} - event for redis-master-ukphz: {kubelet 172.18.12.236} Pulled: Successfully pulled image "gcr.io/google_containers/redis:e2e"
Jun 24 05:03:31.084: INFO: At {2016-06-24 05:03:30 -0400 EDT} - event for redis-master-ukphz: {kubelet 172.18.12.236} Created: Created container with docker id 707e37a9807e
Jun 24 05:03:31.085: INFO: At {2016-06-24 05:03:30 -0400 EDT} - event for redis-master-ukphz: {kubelet 172.18.12.236} Started: Started container with docker id 707e37a9807e

Upstream has taken to pre-pulling all the images used in tests to avoid situations like this.

ncdc · 2016-06-24T11:22:22Z

See https://github.com/kubernetes/kubernetes/blob/ee7ca66dbad52b453bede0da5d1f8b10f972c360/cluster/saltbase/salt/e2e-image-puller/e2e-image-puller.manifest

Looks like that is installed via salt. We could modify our test scripts to do something similar.

Also see e.g. https://github.com/kubernetes/kubernetes/blob/dd4dae4a57129c40d4496ca6ad37ee3a4fa045fa/test/e2e/e2e.go#L128

ncdc · 2016-06-24T11:29:51Z

@derekwaynecarr do you think we should just try to run the upstream image puller manifest in origin's e2e test?

derekwaynecarr · 2016-06-24T16:26:50Z

👍 on pre-pulling e2e images

ncdc · 2016-06-24T17:58:03Z

Ok, we can't reuse the upstream e2e-image-puller pod as is, at least not on Fedora 24, because our /usr/bin/docker is dynamically linked, and trying to run it bind-mounted into a busybox container results in unresolved shared libraries. I modified the manifest to use fedora:24 and everything is pulling as it should be. We just need to decide how we want to approach this. I assume we don't have the e2e-image-puller.manifest file available by default, so maybe we'll need to download it, run sed to change the image, and then create it.

It's also pulling the images serially, which might not be optimal from a timing perspective. It took about 6.5 minutes on my 50Mbps FIOS connection.

ncdc · 2016-06-24T18:00:11Z

It also looks like it's over 5GB of image data, according to docker info.

bparees · 2016-06-29T22:15:39Z

hit again here:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2775/consoleFull

ncdc · 2016-06-29T22:16:58Z

We have a wip pr to pre pull

On Wednesday, June 29, 2016, Ben Parees notifications@github.com wrote:

hit again here:

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2775/consoleFull

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#9444 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAABYskRtOkCxMGVe0BsztILQDGKySBgks5qQu6TgaJpZM4I6HLk
.

mfojtik · 2016-06-30T10:46:52Z

and again here: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2793/

0xmichalis · 2016-06-30T11:23:11Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2794/consoleFull

bparees · 2016-06-30T13:06:02Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2789/consoleFull

ncdc · 2016-06-30T13:13:04Z

No need to keep linking 😄

bparees · 2016-06-30T13:16:11Z

@ncdc sorry for the spam, i forgot this one was actually understood/being fixed. :)

bparees · 2016-07-02T04:05:41Z

We have a wip pr to pre pull

is that wip PR linked to this issue anywhere? or it's still upstream?

ncdc · 2016-07-02T11:45:02Z

#9622

On Saturday, July 2, 2016, Ben Parees notifications@github.com wrote:

We have a wip pr to pre pull

is that wip PR linked to this issue anywhere? or it's still upstream?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#9444 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAABYnEOF3pucsG5B9zZqly4olrpEOD4ks5qReOdgaJpZM4I6HLk
.

bparees · 2016-07-02T13:01:32Z

@ncdc thanks. i took the liberty of updating that PR to indicated it'll fix this issue.

csrwng added area/tests kind/test-flake Categorizes issue or PR as related to test flakes. labels Jun 20, 2016

csrwng mentioned this issue Jun 20, 2016

cluster up: prevent start without a writeable KUBECONFIG #9381

Merged

danmcp added the priority/P2 label Jun 21, 2016

danmcp assigned ncdc Jun 21, 2016

ncdc closed this as completed Jun 21, 2016

0xmichalis reopened this Jun 24, 2016

0xmichalis mentioned this issue Jun 24, 2016

Trigger controller with caches #9485

Merged

0xmichalis mentioned this issue Jun 24, 2016

oc: restore legacy behavior for deploy --latest #9501

Merged

ncdc assigned aveshagarwal and unassigned ncdc Jun 24, 2016

This was referenced Jun 25, 2016

k8s framework test flake "should create services for rc" #9549

Closed

deploy: fix initial image change deployments #9539

Merged

bparees mentioned this issue Jun 30, 2016

bump(github.com/openshift/source-to-image): 392ce24366d6e8c69fac61fcf… #9625

Closed

marun mentioned this issue Jul 1, 2016

dind: enable overlay storage #9661

Merged

deads2k mentioned this issue Jul 1, 2016

make dockercfg secrets handle multiple urls including service DNS #9306

Merged

aveshagarwal mentioned this issue Jul 2, 2016

[WIP] Pre pull images before running tests #9622

Closed

csrwng mentioned this issue Jul 7, 2016

Fix race in imageprogress test #9736

Merged

bparees mentioned this issue Jul 8, 2016

Ensure client can delete a buildconfig with no associated builds, eve… #9657

Merged

danmcp added the component/kubernetes label Sep 7, 2016

ncdc mentioned this issue Nov 7, 2016

should be able to retrieve and filter logs flake #11776

Closed

bparees changed the title ~~extended test flake: kubectl expose conformance test fails with a timeout waiting for redis~~ Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] Dec 14, 2016

bparees mentioned this issue Dec 14, 2016

Updated mysql templates to use mysql5.7 and perl5.24, add both to image-stream templates #12126

Merged

smarterclayton closed this as completed Jan 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] #9444

Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] #9444

csrwng commented Jun 20, 2016

ncdc commented Jun 21, 2016

0xmichalis commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016

derekwaynecarr commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016 •

edited

Loading

bparees commented Jun 29, 2016

ncdc commented Jun 29, 2016

mfojtik commented Jun 30, 2016

0xmichalis commented Jun 30, 2016

bparees commented Jun 30, 2016

ncdc commented Jun 30, 2016

bparees commented Jun 30, 2016

bparees commented Jul 2, 2016

ncdc commented Jul 2, 2016

bparees commented Jul 2, 2016

Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] #9444

Extended.[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] #9444

Comments

csrwng commented Jun 20, 2016

Version

Steps To Reproduce

Current Result

Expected Result

Additional Information

ncdc commented Jun 21, 2016

0xmichalis commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016

derekwaynecarr commented Jun 24, 2016

ncdc commented Jun 24, 2016

ncdc commented Jun 24, 2016 • edited Loading

bparees commented Jun 29, 2016

ncdc commented Jun 29, 2016

mfojtik commented Jun 30, 2016

0xmichalis commented Jun 30, 2016

bparees commented Jun 30, 2016

ncdc commented Jun 30, 2016

bparees commented Jun 30, 2016

bparees commented Jul 2, 2016

ncdc commented Jul 2, 2016

bparees commented Jul 2, 2016

ncdc commented Jun 24, 2016 •

edited

Loading