Jenkins flake on pinging /healthz

[stevekuznetsov]

Seen here:

ERROR: gave up waiting for https://127.0.0.1:28443/healthz

!!! Error in hack/test-cmd.sh:362
    'return 1' exited with status 1
Call stack:
    1: hack/test-cmd.sh:362 main(...)
Exiting with status 1

[miminar]

Can also be seen in #5141, #5578. #5819, #5932, #5971, #5997, #6020, #6063.

[php-coder]

In my case extended tests failed with the same error. I have found the problem, there will be running another instance of openshift:

[INFO] Scan of OpenShift related processes already up via ps -ef    | grep openshift : 
root      8392 16944  0 13:25 pts/0    00:00:00 sudo /data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/openshift start --public-master=localhost --volume-dir=/opt/openshift
root      8396  8392 22 13:25 pts/0    00:00:08 /data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/openshift start --public-master=localhost --volume-dir=/opt/openshift
vagrant   8668  8476  0 13:26 pts/0    00:00:00 grep openshift
[INFO] Starting OpenShift server
[INFO] OpenShift server start at: 
Tue Dec  1 13:26:19 UTC 2015
ERROR: gave up waiting for https://10.0.2.15:8443/healthz

!!! Error in ./test/extended/../../hack/util.sh:364
    'return 1' exited with status 1
Call stack:
    1: ./test/extended/../../hack/util.sh:364 start_os_server(...)
    2: ./test/extended/core.sh:118 main(...)
Exiting with status 1

HTH

[stevekuznetsov]

This is a common problem when running locally, yes. I do not think that this is the issue on Jenkins, however.

[knobunc]

Hitting this too :-(

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/7931/consoleText

[INFO] Scan of OpenShift related processes already up via ps -ef | grep openshift :
ec2-user 26025 26024 0 15:40 ? 00:00:00 /bin/bash /data/src/github.com/openshift/origin/hack/update-generated-swagger-spec.sh _output/verify-generated-swagger-spec
ec2-user 26120 26025 0 15:40 ? 00:00:00 grep openshift
[INFO] Starting OpenShift server
[INFO] OpenShift server start at:
Thu Dec 17 15:40:37 EST 2015
ERROR: gave up waiting for https://127.0.0.1:38443/healthz

!!! Error in /data/src/github.com/openshift/origin/hack/../hack/util.sh:364
'return 1' exited with status 1
Call stack:
1: /data/src/github.com/openshift/origin/hack/../hack/util.sh:364 start_os_master(...)
2: /data/src/github.com/openshift/origin/hack/update-generated-swagger-spec.sh:55 main(...)
Exiting with status 1

[liggitt]

https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_origin/4432/

[liggitt]

https://ci.openshift.redhat.com/jenkins/job/devenv_ami/3144/console

[stevekuznetsov]

@pweil- are you still looking into this? The LDAP extended run hits this pretty often, and there's never seemingly any other OpenShift-related processes up at the time. Seems like a deeper failure of the API server init

[pweil-]

no, not at the moment. There was an update in the timeout and more debugging added so we closed my debug PR #6171 (comment).

[stevekuznetsov]

Hm. We're not seeing it terribly often on the PR job as things fail in the verify/unit test phase often, but there is still some prevalence of this issue. On the LDAP job, there is very little that could flake before this so we're seeing it more often. Doesn't seem as though the fixes you mention did much to alleviate the problem.

[stevekuznetsov]

Seen this three times in a row on the LDAP job.

[stevekuznetsov]

@pweil- continuing to see this -- the LDAP job is up to around 20% failure rate due to this issue alone.

[smarterclayton]

The LDAP job needs to get david's etcd fix thing.