kubeless on Openshift/Minishift 3.6 #390

jberggg · 2017-10-20T10:09:35Z

I try to run kubeless on the latest Minishift release:

oc v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
features: Basic-Auth

Server https://192.168.64.5:8443
openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

For kubeless version 0.2.3 there seems to be a compatibility issue with Openshift. As far as I understood kubeless uses a new Custom Resource Definition (of Kubernetes v1.7.0+, according to the release notes) from version 0.2.0 on. This probably breaks compatibility with Openshift because version 3.6 is based on Kubernetes 1.6. I run into this issue:

➜ ~ oc create -f https://github.com/kubeless/kubeless/releases/download/v0.2.3/kubeless-rbac-v0.2.3.yaml
clusterrole "kubeless-controller-deployer" created
clusterrolebinding "kubeless-controller-deployer" created
service "broker" created
statefulset "kafka" created
service "kafka" created
statefulset "zoo" created
deployment "kubeless-controller" created
serviceaccount "controller-acct" created
service "zoo" created
service "zookeeper" created
error: unable to recognize "https://github.com/kubeless/kubeless/releases/download/v0.2.3/kubeless-rbac-v0.2.3.yaml": no matches for apiextensions.k8s.io/, Kind=CustomResourceDefinition

I can deploy the 0.1.0 kubeless version on the Openshift cluster without errors. However, if I then run a kubeless function ls command I get an ...server could not find... error. Same if I try to deploy something with kubeless.

➜  oc create -f https://github.com/kubeless/kubeless/releases/download/v0.1.0/kubeless-rbac-v0.1.0.yaml
deployment "kubeless-controller" created
clusterrole "kubeless-controller-deployer" created
service "kafka" created
service "zoo" created
service "zookeeper" created
statefulset "zoo" created
serviceaccount "controller-acct" created
clusterrolebinding "kubeless-controller-deployer" created
service "broker" created
statefulset "kafka" created
thirdpartyresource "function.k8s.io" created

➜  ./kubeless_1.0 version
Kubeless version: v0.1.0 (a1e1fab)
➜  ./kubeless_1.0 function ls
FATA[0000] the server could not find the requested resource (get functions.k8s.io)

Does anyone know if/how I can make kubeless work on a minishift instance? Thank you for your help!

The text was updated successfully, but these errors were encountered:

jbianquetti-nami · 2017-10-20T10:57:43Z

Kubeless versions prior to 0.2.0 must work for you: still uses TPR.

Can you share logs of kubeless-controller?

jberggg · 2017-10-20T12:03:07Z

Thank you for your fast reply.

When I deploy kubeless-rbac-v0.1.0.yaml on Minishift kubeless-controller prints following log message over and over again:
E1020 11:57:31.204827 1 reflector.go:201] github.com/kubeless/kubeless/pkg/controller/controller.go:119: Failed to list *spec.Function: User "system:serviceaccount:kubeless:controller-acct" cannot list all functions.k8s.io in the cluster

sebgoa · 2017-10-20T12:43:32Z

@jberggg kubeless was only test on openshift 1.5.1 which was the release prior to 3.6.

3.6 does not include TPR afaik, and 3.7 does not include CRD yet.

so you need to use minishift 1.5.0 (which has TPR) and downgrade to kubeless 0.1.0

The version numbers are confusing, the support of TPR and CRD is not yet well advertized in openshift so I know this is very confusing.

We probably need to remove the openshift manifest until an openshift release that supports CRD.

sebgoa · 2017-10-21T08:17:10Z

I checked with the openshift folks, 3.7 should come out late november and it should have CRD.

So we should be able to run kubeless on openshift 3.7 just before kubecon :)

sebgoa · 2017-10-23T08:03:42Z

I am going to leave this open and as soon as openshift 3.7 is released we should validate our release on it.

jberggg · 2017-10-23T08:04:07Z

Thank you for the investigation and the update :)

ccll · 2017-11-01T09:27:34Z

Openshift released a RC version v3.7.0-rc.0 several days ago, I've tried to run kubeless on top of that with minishift.

$ minishift start --openshift-version=v3.7.0-rc.0
...

$ eval $(minishift oc-env)
...

$ oc version
oc v3.7.0-rc.0+e92d5c5
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.99.147:8443
openshift v3.7.0-rc.0+e92d5c5
kubernetes v1.7.6+a08f5eeb62

$ oc new-project kubeless
...

$ oc create -f https://github.com/kubeless/kubeless/releases/download/v0.2.3/kubeless-openshift-v0.2.3.yaml
clusterrole "kubeless-controller-deployer" created
clusterrolebinding "kubeless-controller-deployer" created
service "broker" created
statefulset "kafka" created
service "kafka" created
statefulset "zoo" created
deployment "kubeless-controller" created
serviceaccount "controller-acct" created
customresourcedefinition "functions.k8s.io" created
service "zoo" created
service "zookeeper" created

All looks good, the CustomResourceDefinition is there:

$ oc get crd
NAME               AGE
functions.k8s.io   48s

$ oc describe crd functions.k8s.io
Name:		functions.k8s.io
Namespace:	
Labels:		<none>
Annotations:	<none>
API Version:	apiextensions.k8s.io/v1beta1
Kind:		CustomResourceDefinition
Metadata:
  Creation Timestamp:	2017-11-01T09:20:04Z
  Resource Version:	4463
  Self Link:		/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/functions.k8s.io
  UID:			d8902eb6-bee5-11e7-ab25-eaf83f141d96
Spec:
  Group:	k8s.io
  Names:
    Kind:	Function
    List Kind:	FunctionList
    Plural:	functions
    Singular:	function
  Scope:	Namespaced
  Version:	v1
Status:
  Accepted Names:
    Kind:	Function
    List Kind:	FunctionList
    Plural:	functions
    Singular:	function
  Conditions:
    Last Transition Time:	<nil>
    Message:			no conflicts found
    Reason:			NoConflicts
    Status:			True
    Type:			NamesAccepted
    Last Transition Time:	2017-11-01T09:20:04Z
    Message:			the initial names have been accepted
    Reason:			InitialNamesAccepted
    Status:			True
    Type:			Established
Events:				<none>

But the kubeless-controller failed to deploy functions, the logs:

time="2017-11-01T09:23:53Z" level=error msg="Error processing default/hello (giving up): configmaps \"hello\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function" pkg=controller 
E1101 09:23:53.325926       1 controller.go:166] configmaps "hello" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function

Is this problem rooted in Openshift or Kubeless?

sebgoa · 2017-11-01T13:54:22Z

It seems to be an RBAC issue, we have not tried yet on openshift 3.7.

Can you run oc api-versions and check that the endpoint k8s.io/v1 is present ?

ccll · 2017-11-02T01:08:31Z

@sebgoa Sure, I suppose you meant kubectl api-versions?

$ oc api-versions
Error: unknown command "api-versions" for "oc"
Run 'oc --help' for usage.

$ kubectl api-versions
admissionregistration.k8s.io/v1alpha1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps.openshift.io/v1
apps/v1beta1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
authorization.openshift.io/v1
autoscaling/v1
batch/v1
batch/v2alpha1
build.openshift.io/v1
certificates.k8s.io/v1beta1
extensions/v1beta1
image.openshift.io/v1
k8s.io/v1
network.openshift.io/v1
networking.k8s.io/v1
oauth.openshift.io/v1
policy/v1beta1
project.openshift.io/v1
quota.openshift.io/v1
rbac.authorization.k8s.io/v1beta1
route.openshift.io/v1
security.openshift.io/v1
storage.k8s.io/v1
storage.k8s.io/v1beta1
template.openshift.io/v1
user.openshift.io/v1
v1

ccll · 2017-11-02T07:47:26Z

kubectl api-verions on a GKE 1.8.1 cluster:

$ kubectl api-versions
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
extensions/v1beta1
k8s.io/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

GKE 1.7.8:

$ kubectl api-versions
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
batch/v1
certificates.k8s.io/v1beta1
extensions/v1beta1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

jbianquetti-nami · 2017-11-30T12:00:52Z

Openshift just released v3.7.0 stable today. I will give a try once the binary builds appear for 3.7.0 https://github.com/openshift/origin/releases

Error starting the cluster: Error attempting to download and cache oc: Cannot get binary 'oc' in version v3.7.0 for the target environment Darwin

jbianquetti-nami · 2017-12-01T10:07:02Z

I found several issues with trying to deploy kubeless v0.3.0 over Openshift v3.7.0.

$ kubeless function ls hello
NAME 	NAMESPACE	HANDLER    	RUNTIME  	TYPE	TOPIC	DEPENDENCIES	STATUS
hello	kubeless 	hello.hello	python2.7	HTTP	     	            	MISSING: Check controller logs

Looking at the kubeless-controller logs, we can see:

time="2017-12-01T09:51:42Z" level=error msg="Error processing kubeless/hello (giving up): configmaps \"hello\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function" pkg=controller
E1201 09:51:42.858149       1 controller.go:167] configmaps "hello" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function

So the problem still persists on v3.7.0 and seems to be related to the format of the CRD: "functions.k8s.io"

andresmgot · 2017-12-01T10:38:55Z

It seems that we can tackle that specific issue specifying different parameters when defining the OwnerReference:

	or := []metav1.OwnerReference{
		{
			Kind:               "Function",
			APIVersion:         "k8s.io",
			Name:               funcObj.Metadata.Name,
			UID:                funcObj.Metadata.UID,
			BlockOwnerDeletion: &t,
		},
	}

@ngtuna do you know if we can change the Kind and APIVersion to something else?

arjunrao87 · 2018-01-09T16:08:25Z

@andresmgot We are having issues with Kubeless deployment over Openshift too. Its a slightly different error but I think its all linked to the same issue

oc logs kubeless-controller-1616540152-qwx6h
time="2018-01-09T14:57:57Z" level=info msg="Starting kubeless controller" pkg=controller
time="2018-01-09T14:57:57Z" level=info msg="Kubeless controller synced and ready" pkg=controller
time="2018-01-09T15:17:05Z" level=info msg="Processing change to Function default/get-python" pkg=controller
time="2018-01-09T15:17:05Z" level=error msg="Function can not be created/updated: configmaps \"get-python\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function" pkg=controller
time="2018-01-09T15:17:05Z" level=error msg="Error processing default/get-python (will retry): configmaps \"get-python\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function" pkg=controller
E0109 15:24:38.959439       1 controller.go:167] configmaps "get-python" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion k8s.io Kind Function: allowed by openshift authorizer, no matches for /, Kind=Function

We are running Openshift 3.7 and kubeless-openshift-v0.2.4.yaml

andresmgot · 2018-01-09T16:44:24Z

@arjunrao87 it seems that it is an issue related to the APIVersion that we are using. It is k8s.io while it should be kubeless specific (kubeless.io). We are working on changing it (apart from several other things) in this PR #537. Once we change it the issue should go away. We will give Openshift 3.7 a try once that PR is merged.

cc/ @murali-reddy

arapulido · 2018-02-09T08:49:31Z

We fixed Openshift 3.7 compatibility here: #580

@jberggg Can you test with the latest master and let us know if it works for you, please?

Thanks!

sebgoa added the enhancement label Oct 21, 2017

ccll mentioned this issue Nov 2, 2017

Unable to deploy 'kubeless' functions in v3.7.0 cluster openshift/origin#17144

Closed

aland-zhang mentioned this issue Mar 1, 2018

v0.7.0 not support OpenShift: configmaps "rook-config-override" is forbidden: cannot set blockOwnerDeletion rook/rook#1533

Closed

sebgoa closed this as completed Apr 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeless on Openshift/Minishift 3.6 #390

kubeless on Openshift/Minishift 3.6 #390

jberggg commented Oct 20, 2017

jbianquetti-nami commented Oct 20, 2017 •

edited

Loading

jberggg commented Oct 20, 2017 •

edited

Loading

sebgoa commented Oct 20, 2017

sebgoa commented Oct 21, 2017

sebgoa commented Oct 23, 2017

jberggg commented Oct 23, 2017 •

edited

Loading

ccll commented Nov 1, 2017

sebgoa commented Nov 1, 2017

ccll commented Nov 2, 2017

ccll commented Nov 2, 2017 •

edited

Loading

jbianquetti-nami commented Nov 30, 2017 •

edited

Loading

jbianquetti-nami commented Dec 1, 2017

andresmgot commented Dec 1, 2017

arjunrao87 commented Jan 9, 2018

andresmgot commented Jan 9, 2018

arapulido commented Feb 9, 2018

kubeless on Openshift/Minishift 3.6 #390

kubeless on Openshift/Minishift 3.6 #390

Comments

jberggg commented Oct 20, 2017

jbianquetti-nami commented Oct 20, 2017 • edited Loading

jberggg commented Oct 20, 2017 • edited Loading

sebgoa commented Oct 20, 2017

sebgoa commented Oct 21, 2017

sebgoa commented Oct 23, 2017

jberggg commented Oct 23, 2017 • edited Loading

ccll commented Nov 1, 2017

sebgoa commented Nov 1, 2017

ccll commented Nov 2, 2017

ccll commented Nov 2, 2017 • edited Loading

jbianquetti-nami commented Nov 30, 2017 • edited Loading

jbianquetti-nami commented Dec 1, 2017

andresmgot commented Dec 1, 2017

arjunrao87 commented Jan 9, 2018

andresmgot commented Jan 9, 2018

arapulido commented Feb 9, 2018

jbianquetti-nami commented Oct 20, 2017 •

edited

Loading

jberggg commented Oct 20, 2017 •

edited

Loading

jberggg commented Oct 23, 2017 •

edited

Loading

ccll commented Nov 2, 2017 •

edited

Loading

jbianquetti-nami commented Nov 30, 2017 •

edited

Loading