jenkins in minishift sample project fails to start and run builds

[goraxe]

The latest version of the openshift-sync plugin, seems to have broken starting and launching builds.

The following error is seen in the jenkins log

SEVERE: Failed to load initial Builds: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/myproject/builds?fieldSelector=status%3DNew. Message: No field label conversion function found for version: build.openshift.io/v1. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=No field label conversion function found for version: build.openshift.io/v1, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).
--
  | io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/myproject/builds?fieldSelector=status%3DNew. Message: No field label conversion function found for version: build.openshift.io/v1. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=No field label conversion function found for version: build.openshift.io/v1, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
  | at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
  | at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
  | at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
  | at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:97)
  | at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
  | at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
  | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  | at java.lang.Thread.run(Thread.java:748)

dropping down to version 0.1.28 resolved the issue.

openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

related issue -> openshift/origin#16501

[gabemontero]

Highlighting the crux of the above expcetion for readability:

Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/myproject/builds?fieldSelector=status%3DNew. Message: No field label conversion function found for version: build.openshift.io/v1. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=No field label conversion function found for version: build.openshift.io/v1, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).

The No field label conversion function found at least at various times has been reported by k8s.io/apimachinery/pkg/runtime/scheme.go

And after a refresh rebase/rebuild from openshift/origin:lates, and then performing something analogous via oc with --loglevel=8, the URI is the more familiar form:

oapi/v1/namespaces/myproject/builds

vs.

apis/build.openshift.io/v1/namespaces/myproject/builds

@deads2k @bparees - I know some reorganization has been going on, but don't know the precise status of where we are at / what should be supported. Based on the details, any inferences from the above details?

@jstrachan - we have started seeing this after bumping to 2.6.2 of fabric8, and going back to 2.3.1 seems to have resolved it. Does this look like something you have run into? (Apologies, don't know where to look to scan for any recent fabric8 bugs/fixes, etc.).

[bparees]

Looks like the client went to the groups api and got a group api object type back and then didn't know what to do with it. I know @jstrachan was adding groups support to the fabric client but i don't know the state of it. As to why it went to that api instead of the legacy api, probably something related to discovery but not sure of the details.

[bparees]

Also not sure why our extended tests/plugin tests didn't turn up this problem when we moved up to the newer fabric8 client?

[gabemontero]

On Sun, Sep 24, 2017 at 11:15 PM Ben Parees ***@***.***> wrote: Also not sure why our extended tests/plugin tests didn't turn up this problem when we moved up to the newer fabric8 client?

I am wondering that as well. I can only speculate something external changed some time after we bumped fabric. I plan to try some different combos on Monday and see what I can discern.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#173 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADbadOsk-wBHpj_P44qtroxIL7iZNQfkks5slxq6gaJpZM4PhWfU> .

[bparees]

I am wondering that as well. I can only speculate something external
changed some time after we bumped fabric.

i haven't seen any pipeline test failures in the nightly builds runs the last couple days.

[gabemontero]

On Mon, Sep 25, 2017 at 12:10 AM Ben Parees ***@***.***> wrote: I am wondering that as well. I can only speculate something external changed some time after we bumped fabric. i haven't seen any pipeline test failures in the nightly builds runs the last couple days.

maybe restarting jenkins has something to do with it Was planning to throw that in the mix

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#173 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADbadKmujF5MtXBkJUxbbTwEVt6vyG0hks5slye7gaJpZM4PhWfU> .

[deads2k]

The No field label conversion function found at least at various times has been reported by k8s.io/apimachinery/pkg/runtime/scheme.go

This was broken in 3.6, but was recently (last week), fixed in 3.7.

[bparees]

This was broken in 3.6, but was recently (last week), fixed in 3.7.

are we going to fix it in 3.6? if not, what are clients supposed to do?

[gabemontero]

Bookkeeping: this has also been reported in https://issues.jenkins-ci.org/browse/JENKINS-47093

[gabemontero]

@deads2k - is openshift/origin@82c48ea the fix you are referring to?

[deads2k]

@deads2k - is openshift/origin@82c48ea the fix you are referring to?

at a glance:

1. Create default storage selection functions origin#15594
2. create default storage selection functions kubernetes/kubernetes#50019
3. default field selectors origin#15864
4. default field selectors kubernetes/kubernetes#51197
5. add legacy field selector for names origin#15963
6. fix the route field selectors origin#16305
7. fix remaining field selectors  origin#16327

The surgery is deep, all the way down to REST Storage. @liggitt and I hadn't discussed a different set of fixes to attempt to backport, but those are unlikely.

[gabemontero]

Wow - sure is deep.

Not having digested all those different fixes yet, from the perspective of the fabric8 java client to openshift, and how it might manage to support both 3.6(with the bug) and 3.7(without the bug), is there some sort of query they should make prior to choosing between apis/build.openshift.io/v1/namespaces/myproject/builds and oapi/v1/namespaces/myproject/builds?

Or is the only recourse to retry using the old form based on the particulars of the http response to the new form?

[gabemontero]

@bparees - re: why we did not see this in the PR testing .... to confirm the prior implication, we bumped the fabric8 version after the last of the pulls @deads2k listed got merged

so we tested against a version of openshift with the fix.

But users testing against 3.6 or early 3.7 alphas don't have the fix

[deads2k]

are we going to fix it in 3.6? if not, what are clients supposed to do?

opened openshift/origin#16541 for you.

[gabemontero]

Thanks @deads2k

[gabemontero]

I've submitted v0.1.30 to jenkins update center ... will update when it is available

[gabemontero]

v0.1.30 is available .... updating the openshift jenkins centos as I type

[gabemontero]

openshift/jenkins#387 will update the openshift jenkins centos image with v0.1.30 of the sync plugin