es-monitoring-association controller should not update the status when no association #4985
Comments
|
I spent a little time trying to figure out the root cause of this bug but got nothing but I wanted to share what I saw as it is quite confusing. I kept only the EsMonitoring association controller: diff --git a/cmd/manager/main.go b/cmd/manager/main.go
index 6fb174105..d0913476d 100644
--- a/cmd/manager/main.go
+++ b/cmd/manager/main.go
@@ -569,7 +569,7 @@ func startOperator(ctx context.Context) error {
accessReviewer = rbac.NewPermissiveAccessReviewer()
}
- if err := registerControllers(mgr, params, accessReviewer); err != nil {
+ if err := associationctl.AddEsMonitoring(mgr, accessReviewer, params); err != nil {
And I replaced the func (r *Reconciler) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
uuid := uuid2.NewUUID()
associated := r.AssociatedObjTemplate()
if err := r.Client.Get(context.Background(), request.NamespacedName, associated); err != nil {
return reconcile.Result{}, err
}
fmt.Println("------")
fmt.Println("@@ 1 GetResourceVersion", uuid, " - ", associated.GetResourceVersion())
if err := r.Status().Update(ctx, associated); err != nil {
return reconcile.Result{}, err
}
if err := r.Client.Get(context.Background(), request.NamespacedName, associated); err != nil {
return reconcile.Result{}, err
}
fmt.Println("@@ 2 GetResourceVersion", uuid, " - ", associated.GetResourceVersion())
return reconcile.Result{}, nil
}Here is the result: The resource version appears to be updated from 1 in most cases, sometimes from 2 and sometimes from zero, which may indicate that something else has updated the object? Honestly, I don't know what to think. To compare, on GKE for example, the resource version is never incremented and I get only once: |
I think this is the expected behaviour, updating a subresource should not update The way the issue has been fixed (using |
|
OCP 3.11 is not affected, only minishift. |
|
Landed here from https://discuss.elastic.co/t/elastic-operator-not-creating-elasticsearch-in-openshift-3-11/286847/3 This is happening on bare metal k8s and eventually we are unable to install es. Pod remains in "Init:CrashLoopBackOff" state. Is there any way to go around this before the new release? |
Deploy the quickstart, you will see this event:
The es-monitoring association controller updated the Association status because the status changed, from no association to no association. When we compare if the status has changed:
cloud-on-k8s/pkg/controller/association/reconciler.go
Line 412 in 085dfde
We get that the statuses are not equal because:
oldStatusisnil(the default value because never set)newStatusiscommonv1.AssociationStatusMap{}It seems harmless and not to be a problem, though it remains totally useless and pollutes events for nothing.
And now is where it gets interesting. On minishift, this is problematic. I don't know why yet. The es-monitoring association controller enters an infinite loop of reconciliation loop preventing the Elasticsearch resource from being reconciled by the elasticsearch controller.
💥
log of the controllers
In addition to fixing the issue, we should check that wherever we use
reflect.DeepEqualwith variables of typeMap, we are protected from this.The text was updated successfully, but these errors were encountered: