Impossible to annotate Kibana Service

[xkobal]

Bug Report

What did you do?
I want to configure Google IAP (https://cloud.google.com/iap/docs/enabling-kubernetes-howto) to secure my Kibana installation and be able to open it on Internet without risk.
I need to add a BackendConfig to enable IAP, but the Kibana Service controller seems to ignore my annotations if they are added to an existing Kibana resource (@pebrc).

What did you expect to see?
Kibana service respect the annotation added.

What did you see instead? Under which circumstances?
Empty annotation, even by deleting the service. (@pebrc I found that deleting the service does in fact lead to the correct creation of the the annotation)
The only way I have found now to annotate, is to use this command kubectl annotate service kibana-kb-http --overwrite beta.cloud.google.com/backend-config='{"ports": {"5601":"backend-kibana"}}' but if Kibana service is deleted, the annotation is lost.

Environment

• ECK version:

1.0.0

• Kubernetes information:

  GKE

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-gke.12", GitCommit:"188432a69210ca32cafded81b4dd1c063720cac0", GitTreeState:"clean", BuildDate:"2019-11-07T19:27:01Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: "elasticsearch"
  http:
    service:
      metadata:
        annotations:
          beta.cloud.google.com/backend-config: '{"ports": {"5601":"backend-kibana"}}'
      spec:
        type: NodePort
    tls:
      selfSignedCertificate:
        disabled: true

[pebrc]

@xkobal I have not been able to reproduce your issue. I used the same client version and ECK 1.0.0-beta1 and the annotations are set as expected.

[xkobal]

@pebrc The service was already exists without annotation, that was an update made with kubectl apply -f . Can it come from that ?

[xkobal]

Ok @pebrc I have found how delete Kibana. So I can confirm you it works when no previous Kibana service exists. But update doesn't work.

[pebrc]

Yes you are right. There is an issue with the update detection code here that does ignore metadata changes.

cloud-on-k8s/pkg/controller/common/service_control.go

Lines 45 to 81 in e888a37

	func needsUpdate(expected *corev1.Service, reconciled *corev1.Service) bool {
	// ClusterIP might not exist in the expected service,
	// but might have been set after creation by k8s on the actual resource.
	// In such case, we want to use these values for comparison.
	if expected.Spec.ClusterIP == "" {
	expected.Spec.ClusterIP = reconciled.Spec.ClusterIP
	}
	// Type may be defaulted by the api server
	if expected.Spec.Type == "" {
	expected.Spec.Type = reconciled.Spec.Type
	}
	// SessionAffinity may be defaulted by the api server
	if expected.Spec.SessionAffinity == "" {
	expected.Spec.SessionAffinity = reconciled.Spec.SessionAffinity
	}
	// same for the target port and node port
	if len(expected.Spec.Ports) == len(reconciled.Spec.Ports) {
	for i := range expected.Spec.Ports {
	if expected.Spec.Ports[i].TargetPort.IntValue() == 0 {
	expected.Spec.Ports[i].TargetPort = reconciled.Spec.Ports[i].TargetPort
	}
	// check if NodePort makes sense for this service type
	if hasNodePort(expected.Spec.Type) && expected.Spec.Ports[i].NodePort == 0 {
	expected.Spec.Ports[i].NodePort = reconciled.Spec.Ports[i].NodePort
	}
	}
	}
	if expected.Spec.HealthCheckNodePort == 0 {
	expected.Spec.HealthCheckNodePort = reconciled.Spec.HealthCheckNodePort
	}
	return !reflect.DeepEqual(expected.Spec, reconciled.Spec)
	}

Deleting the Kibana service is a workaround to get the annotation applied.

[xkobal]

Thanks for your help. It works for now with Kibana deletion.