Operator reports that its installed multiple times #431

Numblesix · 2020-08-07T09:49:40Z

Describe the bug
Operator reports installed in a "loop"

Environment
Provide accurate information about the environment to help us reproduce the issue.

Trident version: 20.07
Trident installation flags used: https://github.com/NetApp/trident/blob/stable/v20.07/deploy/crds/tridentprovisioner_cr.yaml
Container runtime: CRI-0
Kubernetes version: v1.18.3+012b3ec
Kubernetes orchestrator: Openshift 4.5.4
OS: RHEL CoreOS
NetApp backend types: Ontap NAS and Ontap SAN

To Reproduce
Steps to reproduce the behavior: Install via Operator as described in DOc

Expected behavior
Trident install should only be reported once

Additional context

user@host ➜  ~  oc describe tprov -n trident trident                   
Name:         trident
Namespace:    trident
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-08-06T18:32:07Z
  Generation:          1
  Managed Fields:
    API Version:  trident.netapp.io/v1
    Operation:    Update
    Time:         2020-08-06T18:32:07Z
    API Version:  trident.netapp.io/v1
    Manager:         trident-operator
    Operation:       Update
    Time:            2020-08-06T18:33:22Z
  Resource Version:  25931
  Self Link:         /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:               e2529796-b8de-45fa-bc96-fa2293b2bb13
Spec:
  Debug:                true
  Silence Autosupport:  true
Status:
  Current Installation Params:
    IPv6:               false
    Autosupport Image:  netapp/trident-autosupport:20.07.0
    Autosupport Proxy:  
    Debug:              true
    Image Pull Secrets:
    Image Registry:       quay.io
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  true
    Trident Image:        netapp/trident:20.07.0
  Message:                Trident installed
  Status:                 Installed
  Version:                v20.07.0
Events:
  Type    Reason     Age                  From                        Message
  ----    ------     ----                 ----                        -------
  Normal  Installed  55s (x182 over 14h)  trident-operator.netapp.io  Trident installed

The text was updated successfully, but these errors were encountered:

balaramesh · 2020-08-07T12:57:03Z

@Numblesix perhaps I'm missing something here. Can you explain what you mean by "Operator reports that it is installed multiple times"? From your tprov describe output it looks like Trident was installed and the status field indicates that

innergy · 2020-08-07T13:01:47Z

It’s the last line of the output. Repeated 182 times over 14 hours.

Numblesix · 2020-08-07T13:19:54Z

Hi sorry ,

yeah i find it quite weird that it shows this line hear:

Events:
  Type    Reason     Age                  From                        Message
  ----    ------     ----                 ----                        -------
  Normal  Installed  55s (x182 over 14h)  trident-operator.netapp.io  Trident installed

Shouldnt that be displayed only once ?

balaramesh · 2020-08-07T13:20:22Z

@innergy 👍
@Numblesix the trident-operator runs a reconciling job every 5 minutes to sync the status of the tprov and make sure Trident is installed as the tprov object dictates. The operator's logs will show the sync job happening every 5 minutes. This is normal and expected

Numblesix · 2020-08-07T13:21:03Z

Thanks a lot :)

Seemed strange to me but then its okay :)

gnarl · 2020-08-11T17:07:14Z

We plan to provide a fix for this by improving the message.

rohit-arora-dev · 2020-08-11T17:13:36Z

As part of each reconcile loop Operator sets status as well as event message in accordance with the current state of Trident.

There is a logic in place to not to update the CR status if it is same as old status (to avoid continuous CR updates). However, event messages are set each and every time thus resulting in the above behaviour and increment of the age counter.

This behaviour is benign but can cause confusion, to avoid this confusion operator should only update the event message if the status is actually getting changed, else not.

loxley · 2020-09-14T09:13:18Z

Hi, we experience almost the same problem as described here but we are running Openshift 4.4.

But in our case new secrets are generated over and over again also. I guess on each restart it generates a new secret leading to this many:

➜  oc describe tprov -n trident trident
Name:         trident
Namespace:    trident
Labels:       app.kubernetes.io/instance=**************.com-trident
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"trident.netapp.io/v1","kind":"TridentProvisioner","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"****...
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-05-14T10:14:44Z
  Generation:          3
  Resource Version:    102031764
  Self Link:           /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:                 38c0978c-2c9b-4c49-a48f-d6bb947599ee
Spec:
  Debug:          true
  Trident Image:  netapp/trident:20.07
Status:
  Current Installation Params:
    IPv6:               false
    Autosupport Image:  netapp/trident-autosupport:20.07.0
    Autosupport Proxy:
    Debug:              true
    Image Pull Secrets:
    Image Registry:       quay.io
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  false
    Trident Image:        netapp/trident:20.07
  Message:                Trident installed
  Status:                 Installed
  Version:                v20.07.0
Events:
  Type    Reason     Age                    From                        Message
  ----    ------     ----                   ----                        -------
  Normal  Installed  2m49s (x1165 over 4d)  trident-operator.netapp.io  Trident installed

➜  oc get secrets -n trident | grep -c trident-csi-token
1168

rohit-arora-dev · 2020-09-14T22:33:07Z

@loxley

The fix for the issue (generation of multiple secrets) has already been checked into the master and is expected to be part of 20.07.1 release.

Link to the fix: 051294a

gnarl · 2020-09-22T22:08:57Z

This is fixed in the Trident 20.07.1 release.

lfstm · 2021-01-15T08:46:10Z

Hi, we are on trident 20.10.1 running on opsh4.6 and still get the following message every x minutes:

Generated from trident-operator.netapp.io
259 times in the last 19 hours
Trident installed

Is this expected/normal behaviour?

Numblesix · 2021-01-15T08:49:33Z

Yes this is expected with you version of trident once you update to 20.07 this is fixed see the comment above

lfstm · 2021-01-15T08:57:07Z

Sorry, my bad, corrected the version in my previous post, we are on 20.10.1

Numblesix · 2021-01-15T09:02:15Z

aah okay :)

Seems we got the issue back im still on 20.10.0 and can confirm that this is there again see below 🔥

➜  ~ oc describe tprov -n trident trident                                                                                (debug/api-osr2-xeop-de:6443/du243)
Name:         trident
Namespace:    trident
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
....
Spec:
  Debug:                true
  Silence Autosupport:  true
Status:
  Current Installation Params:
    IPv6:                       false
    Autosupport Hostname:       
    Autosupport Image:          netapp/trident-autosupport:20.10.0
    Autosupport Proxy:          
    Autosupport Serial Number:  
    Debug:                      true
    Enable Node Prep:           false
    Image Pull Secrets:
    Image Registry:       quay.io
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  true
    Trident Image:        netapp/trident:20.10.0
  Message:                Trident installed
  Status:                 Installed
  Version:                v20.10.0
Events:
  Type    Reason     Age                   From                        Message
  ----    ------     ----                  ----                        -------
  Normal  Installed  102s (x2861 over 9d)  trident-operator.netapp.io  Trident installed

Numblesix · 2021-01-15T09:02:35Z

@gnarl please reopen :) or Shall i open a new Issue and reference :) ?

lfstm · 2021-01-15T09:30:42Z

@Numblesix Thanks for the quick help 🎉

mattikbk · 2021-01-21T13:50:41Z

Please reopen this. I just upgraded from v20.04.0 to v20.10.1 and the operator still reports installs very frequently:

Normal Installed 2m42s (x9 over 26m) trident-operator.netapp.io Trident installed

gnarl · 2021-01-21T17:12:59Z

Reopening this issue as Trident is still reporting installs more than once.

gnarl · 2021-01-25T20:24:25Z

This is fixed with commit a60b7a which is included in the Trident 21.01 release.

Numblesix added the bug label Aug 7, 2020

gnarl added the tracked label Aug 7, 2020

Numblesix closed this as completed Aug 7, 2020

gnarl reopened this Aug 11, 2020

gnarl closed this as completed Sep 22, 2020

gnarl reopened this Jan 21, 2021

netapp-ci closed this as completed in a60b7ac Jan 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Operator reports that its installed multiple times #431

Operator reports that its installed multiple times #431

Numblesix commented Aug 7, 2020 •

edited

Loading

balaramesh commented Aug 7, 2020

innergy commented Aug 7, 2020

Numblesix commented Aug 7, 2020 •

edited

Loading

balaramesh commented Aug 7, 2020

Numblesix commented Aug 7, 2020 •

edited

Loading

gnarl commented Aug 11, 2020

rohit-arora-dev commented Aug 11, 2020

loxley commented Sep 14, 2020 •

edited

Loading

rohit-arora-dev commented Sep 14, 2020

gnarl commented Sep 22, 2020

lfstm commented Jan 15, 2021 •

edited

Loading

Numblesix commented Jan 15, 2021

lfstm commented Jan 15, 2021 •

edited

Loading

Numblesix commented Jan 15, 2021

Numblesix commented Jan 15, 2021

lfstm commented Jan 15, 2021

mattikbk commented Jan 21, 2021 •

edited

Loading

gnarl commented Jan 21, 2021

gnarl commented Jan 25, 2021 •

edited

Loading

Operator reports that its installed multiple times #431

Operator reports that its installed multiple times #431

Comments

Numblesix commented Aug 7, 2020 • edited Loading

balaramesh commented Aug 7, 2020

innergy commented Aug 7, 2020

Numblesix commented Aug 7, 2020 • edited Loading

balaramesh commented Aug 7, 2020

Numblesix commented Aug 7, 2020 • edited Loading

gnarl commented Aug 11, 2020

rohit-arora-dev commented Aug 11, 2020

loxley commented Sep 14, 2020 • edited Loading

rohit-arora-dev commented Sep 14, 2020

gnarl commented Sep 22, 2020

lfstm commented Jan 15, 2021 • edited Loading

Numblesix commented Jan 15, 2021

lfstm commented Jan 15, 2021 • edited Loading

Numblesix commented Jan 15, 2021

Numblesix commented Jan 15, 2021

lfstm commented Jan 15, 2021

mattikbk commented Jan 21, 2021 • edited Loading

gnarl commented Jan 21, 2021

gnarl commented Jan 25, 2021 • edited Loading

Numblesix commented Aug 7, 2020 •

edited

Loading

Numblesix commented Aug 7, 2020 •

edited

Loading

Numblesix commented Aug 7, 2020 •

edited

Loading

loxley commented Sep 14, 2020 •

edited

Loading

lfstm commented Jan 15, 2021 •

edited

Loading

lfstm commented Jan 15, 2021 •

edited

Loading

mattikbk commented Jan 21, 2021 •

edited

Loading

gnarl commented Jan 25, 2021 •

edited

Loading